[==========] Running 30 tests from 4 test cases. [----------] Global test environment set-up. [----------] 14 tests from AdminCliTest [ RUN ] AdminCliTest.TestChangeConfig I0703 01:39:17.633059 6083 test_util.cc:208] Using random seed: 1072422916 I0703 01:39:17.651295 6083 ts_itest-base.cc:113] Starting cluster with: I0703 01:39:17.651341 6083 ts_itest-base.cc:114] -------------- I0703 01:39:17.651360 6083 ts_itest-base.cc:115] 3 tablet servers I0703 01:39:17.651381 6083 ts_itest-base.cc:116] 2 replicas per TS I0703 01:39:17.651397 6083 ts_itest-base.cc:117] -------------- I0703 01:39:17.653419 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:44908 --webserver_interface=127.5.240.254 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --catalog_manager_wait_for_new_tablets_to_elect_leader=false --allow_unsafe_replication_factor=true --catalog_manager_evict_excess_replicas=false --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:39:17.830332 6093 flags.cc:406] Enabled unsafe flag: --allow_unsafe_replication_factor=true W0703 01:39:17.831346 6093 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:39:17.831404 6093 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:39:17.833638 6093 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0703 01:39:17.833681 6093 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:39:17.833706 6093 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0703 01:39:17.833734 6093 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0703 01:39:17.841890 6093 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --allow_unsafe_replication_factor=true --catalog_manager_evict_excess_replicas=false --catalog_manager_wait_for_new_tablets_to_elect_leader=false --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:44908 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.5.240.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:39:17.847882 6093 master_main.cc:80] Initializing master server... I0703 01:39:17.848553 6093 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 132208us I0703 01:39:17.849581 6093 fs_manager.cc:261] Metadata directory not provided I0703 01:39:17.849659 6093 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:39:17.849884 6093 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:39:17.849947 6093 server_base.cc:433] Attempting to create new FS layout instead I0703 01:39:17.854112 6093 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/instance: uuid: "70c5178815714ecdb7563bf7cb45db18" format_stamp: "Formatted at 2018-07-03 08:39:17 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:17.855036 6093 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "70c5178815714ecdb7563bf7cb45db18" format_stamp: "Formatted at 2018-07-03 08:39:17 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:17.995128 6093 fs_manager.cc:501] Time spent creating directory manager: real 0.140s user 0.004s sys 0.004s I0703 01:39:17.996402 6093 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:39:17.996840 6093 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:39:18.000594 6093 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s I0703 01:39:18.000643 6093 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal uuid: "70c5178815714ecdb7563bf7cb45db18" format_stamp: "Formatted at 2018-07-03 08:39:17 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:18.000758 6093 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:39:18.051064 6093 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:39:18.052495 6093 master_main.cc:83] Starting Master server... I0703 01:39:18.089674 6093 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.254:44908 I0703 01:39:18.090088 6093 webserver.cc:175] Starting webserver on 127.5.240.254:0 I0703 01:39:18.090123 6093 webserver.cc:186] Document root disabled I0703 01:39:18.090816 6093 webserver.cc:313] Webserver started. Bound to: http://127.5.240.254:33414/ I0703 01:39:18.091382 6093 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb I0703 01:39:18.095397 6152 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:39:18.095430 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master as pid 6093 I0703 01:39:18.096225 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.5.240.193:0 --local_ip_for_outbound_sockets=127.5.240.193 --webserver_interface=127.5.240.193 --webserver_port=0 --tserver_master_addrs=127.5.240.254:44908 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --enable_leader_failure_detection=false with env {} I0703 01:39:18.106825 6152 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 70c5178815714ecdb7563bf7cb45db18: Bootstrap starting. I0703 01:39:18.109014 6152 tablet_bootstrap.cc:578] T 00000000000000000000000000000000 P 70c5178815714ecdb7563bf7cb45db18: No blocks or log segments found. Creating new log. I0703 01:39:18.109854 6152 log.cc:525] T 00000000000000000000000000000000 P 70c5178815714ecdb7563bf7cb45db18: Log is configured to *not* fsync() on all Append() calls I0703 01:39:18.112149 6152 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 70c5178815714ecdb7563bf7cb45db18: No bootstrap required, opened a new log I0703 01:39:18.118310 6152 raft_consensus.cc:325] T 00000000000000000000000000000000 P 70c5178815714ecdb7563bf7cb45db18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70c5178815714ecdb7563bf7cb45db18" member_type: VOTER } I0703 01:39:18.118507 6152 raft_consensus.cc:351] T 00000000000000000000000000000000 P 70c5178815714ecdb7563bf7cb45db18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:39:18.118583 6152 raft_consensus.cc:628] T 00000000000000000000000000000000 P 70c5178815714ecdb7563bf7cb45db18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70c5178815714ecdb7563bf7cb45db18, State: Initialized, Role: FOLLOWER I0703 01:39:18.119031 6152 consensus_queue.cc:227] T 00000000000000000000000000000000 P 70c5178815714ecdb7563bf7cb45db18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70c5178815714ecdb7563bf7cb45db18" member_type: VOTER } I0703 01:39:18.119210 6152 raft_consensus.cc:365] T 00000000000000000000000000000000 P 70c5178815714ecdb7563bf7cb45db18 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:39:18.119292 6152 raft_consensus.cc:457] T 00000000000000000000000000000000 P 70c5178815714ecdb7563bf7cb45db18 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:39:18.119377 6152 raft_consensus.cc:2730] T 00000000000000000000000000000000 P 70c5178815714ecdb7563bf7cb45db18: Snoozing failure detection for 1.970s (starting election) I0703 01:39:18.119464 6152 raft_consensus.cc:2782] T 00000000000000000000000000000000 P 70c5178815714ecdb7563bf7cb45db18 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:39:18.120841 6152 raft_consensus.cc:479] T 00000000000000000000000000000000 P 70c5178815714ecdb7563bf7cb45db18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70c5178815714ecdb7563bf7cb45db18" member_type: VOTER } I0703 01:39:18.121204 6152 leader_election.cc:270] T 00000000000000000000000000000000 P 70c5178815714ecdb7563bf7cb45db18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:39:18.122148 6155 raft_consensus.cc:2730] T 00000000000000000000000000000000 P 70c5178815714ecdb7563bf7cb45db18: Snoozing failure detection for 2.196s (election complete) I0703 01:39:18.122452 6155 raft_consensus.cc:2537] T 00000000000000000000000000000000 P 70c5178815714ecdb7563bf7cb45db18 [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:39:18.124445 6155 raft_consensus.cc:593] T 00000000000000000000000000000000 P 70c5178815714ecdb7563bf7cb45db18 [term 1 LEADER]: Becoming Leader. State: Replica: 70c5178815714ecdb7563bf7cb45db18, State: Running, Role: LEADER I0703 01:39:18.124842 6155 consensus_queue.cc:204] T 00000000000000000000000000000000 P 70c5178815714ecdb7563bf7cb45db18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70c5178815714ecdb7563bf7cb45db18" member_type: VOTER } I0703 01:39:18.126850 6152 sys_catalog.cc:337] T 00000000000000000000000000000000 P 70c5178815714ecdb7563bf7cb45db18 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "70c5178815714ecdb7563bf7cb45db18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70c5178815714ecdb7563bf7cb45db18" member_type: VOTER } } I0703 01:39:18.126678 6156 sys_catalog.cc:337] T 00000000000000000000000000000000 P 70c5178815714ecdb7563bf7cb45db18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70c5178815714ecdb7563bf7cb45db18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70c5178815714ecdb7563bf7cb45db18" member_type: VOTER } } I0703 01:39:18.127352 6152 sys_catalog.cc:340] T 00000000000000000000000000000000 P 70c5178815714ecdb7563bf7cb45db18 [sys.catalog]: This master's current role is: LEADER I0703 01:39:18.127516 6156 sys_catalog.cc:340] T 00000000000000000000000000000000 P 70c5178815714ecdb7563bf7cb45db18 [sys.catalog]: This master's current role is: LEADER I0703 01:39:18.126850 6157 sys_catalog.cc:337] T 00000000000000000000000000000000 P 70c5178815714ecdb7563bf7cb45db18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70c5178815714ecdb7563bf7cb45db18. Latest consensus state: current_term: 1 leader_uuid: "70c5178815714ecdb7563bf7cb45db18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70c5178815714ecdb7563bf7cb45db18" member_type: VOTER } } I0703 01:39:18.128068 6157 sys_catalog.cc:340] T 00000000000000000000000000000000 P 70c5178815714ecdb7563bf7cb45db18 [sys.catalog]: This master's current role is: LEADER I0703 01:39:18.130736 6152 sys_catalog.cc:424] T 00000000000000000000000000000000 P 70c5178815714ecdb7563bf7cb45db18 [sys.catalog]: configured and running, proceeding with master startup. I0703 01:39:18.130965 6159 catalog_manager.cc:1006] Loading table and tablet metadata into memory... I0703 01:39:18.132436 6093 master_main.cc:86] Master server successfully started. I0703 01:39:18.137758 6159 catalog_manager.cc:1017] Initializing Kudu internal certificate authority... I0703 01:39:18.223114 6159 catalog_manager.cc:904] Generated new certificate authority record I0703 01:39:18.225212 6159 catalog_manager.cc:1026] Loading token signing keys... I0703 01:39:18.238320 6159 catalog_manager.cc:4102] T 00000000000000000000000000000000 P 70c5178815714ecdb7563bf7cb45db18: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:39:18.278316 6153 flags.cc:406] Enabled unsafe flag: --enable_leader_failure_detection=false W0703 01:39:18.279322 6153 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:39:18.279418 6153 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:39:18.281455 6153 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:39:18.281600 6153 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.193 I0703 01:39:18.288997 6153 tablet_server_main.cc:78] Tablet server non-default flags: --enable_leader_failure_detection=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.5.240.193 --webserver_port=0 --tserver_master_addrs=127.5.240.254:44908 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.193 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:39:18.295745 6153 tablet_server_main.cc:85] Initializing tablet server... I0703 01:39:18.297304 6153 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 132708us I0703 01:39:18.297911 6153 fs_manager.cc:261] Metadata directory not provided I0703 01:39:18.297981 6153 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:39:18.298208 6153 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:39:18.298277 6153 server_base.cc:433] Attempting to create new FS layout instead I0703 01:39:18.302508 6153 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "3af92b254d28411dbb9bb12787be9a8a" format_stamp: "Formatted at 2018-07-03 08:39:18 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:18.302997 6153 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "3af92b254d28411dbb9bb12787be9a8a" format_stamp: "Formatted at 2018-07-03 08:39:18 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:18.311008 6153 fs_manager.cc:501] Time spent creating directory manager: real 0.008s user 0.003s sys 0.004s I0703 01:39:18.312239 6153 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:39:18.312650 6153 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:39:18.316247 6153 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0703 01:39:18.316299 6153 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal uuid: "3af92b254d28411dbb9bb12787be9a8a" format_stamp: "Formatted at 2018-07-03 08:39:18 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:18.316431 6153 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:39:18.370658 6153 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:39:18.372236 6153 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:39:18.372634 6153 tablet_server_main.cc:90] Starting tablet server... I0703 01:39:18.467829 6153 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.193:47305 I0703 01:39:18.468322 6153 webserver.cc:175] Starting webserver on 127.5.240.193:0 I0703 01:39:18.468369 6153 webserver.cc:186] Document root disabled I0703 01:39:18.469705 6153 webserver.cc:313] Webserver started. Bound to: http://127.5.240.193:33103/ I0703 01:39:18.470238 6153 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb I0703 01:39:18.471568 6153 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:39:18.476816 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 6153 I0703 01:39:18.477596 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.5.240.194:0 --local_ip_for_outbound_sockets=127.5.240.194 --webserver_interface=127.5.240.194 --webserver_port=0 --tserver_master_addrs=127.5.240.254:44908 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --enable_leader_failure_detection=false with env {} I0703 01:39:18.488430 6282 heartbeater.cc:346] Connected to a master server at 127.5.240.254:44908 I0703 01:39:18.488585 6282 heartbeater.cc:426] Registering TS with master... I0703 01:39:18.489002 6282 heartbeater.cc:475] Master 127.5.240.254:44908 requested a full tablet report, sending... I0703 01:39:18.490494 6119 ts_manager.cc:98] Registered new tserver with Master: 3af92b254d28411dbb9bb12787be9a8a (127.5.240.193:47305) I0703 01:39:18.492337 6119 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.193:54287 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:39:18.636548 6286 flags.cc:406] Enabled unsafe flag: --enable_leader_failure_detection=false W0703 01:39:18.637604 6286 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:39:18.637698 6286 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:39:18.639758 6286 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:39:18.639879 6286 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.194 I0703 01:39:18.646572 6286 tablet_server_main.cc:78] Tablet server non-default flags: --enable_leader_failure_detection=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.194:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.5.240.194 --webserver_port=0 --tserver_master_addrs=127.5.240.254:44908 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.194 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:39:18.653407 6286 tablet_server_main.cc:85] Initializing tablet server... I0703 01:39:18.655011 6286 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 132708us I0703 01:39:18.655598 6286 fs_manager.cc:261] Metadata directory not provided I0703 01:39:18.655666 6286 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:39:18.655906 6286 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:39:18.655977 6286 server_base.cc:433] Attempting to create new FS layout instead I0703 01:39:18.660084 6286 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "28da852a15344a52b350ff3d187764af" format_stamp: "Formatted at 2018-07-03 08:39:18 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:18.660645 6286 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "28da852a15344a52b350ff3d187764af" format_stamp: "Formatted at 2018-07-03 08:39:18 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:18.668006 6286 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.004s sys 0.002s I0703 01:39:18.669193 6286 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:39:18.669600 6286 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:39:18.673200 6286 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s I0703 01:39:18.673251 6286 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal uuid: "28da852a15344a52b350ff3d187764af" format_stamp: "Formatted at 2018-07-03 08:39:18 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:18.673382 6286 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:39:18.699673 6286 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:39:18.701606 6286 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:39:18.701930 6286 tablet_server_main.cc:90] Starting tablet server... I0703 01:39:18.785607 6286 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.194:52370 I0703 01:39:18.786029 6286 webserver.cc:175] Starting webserver on 127.5.240.194:0 I0703 01:39:18.786075 6286 webserver.cc:186] Document root disabled I0703 01:39:18.787231 6286 webserver.cc:313] Webserver started. Bound to: http://127.5.240.194:52860/ I0703 01:39:18.787808 6286 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb I0703 01:39:18.788070 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 6286 I0703 01:39:18.788661 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.5.240.195:0 --local_ip_for_outbound_sockets=127.5.240.195 --webserver_interface=127.5.240.195 --webserver_port=0 --tserver_master_addrs=127.5.240.254:44908 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --enable_leader_failure_detection=false with env {} I0703 01:39:18.789825 6286 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:39:18.804966 6405 heartbeater.cc:346] Connected to a master server at 127.5.240.254:44908 I0703 01:39:18.805107 6405 heartbeater.cc:426] Registering TS with master... I0703 01:39:18.805488 6405 heartbeater.cc:475] Master 127.5.240.254:44908 requested a full tablet report, sending... I0703 01:39:18.806488 6119 ts_manager.cc:98] Registered new tserver with Master: 28da852a15344a52b350ff3d187764af (127.5.240.194:52370) I0703 01:39:18.807318 6119 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.194:39593 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:39:18.950619 6407 flags.cc:406] Enabled unsafe flag: --enable_leader_failure_detection=false W0703 01:39:18.951627 6407 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:39:18.951704 6407 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:39:18.953680 6407 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:39:18.953775 6407 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.195 I0703 01:39:18.960606 6407 tablet_server_main.cc:78] Tablet server non-default flags: --enable_leader_failure_detection=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.195:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.5.240.195 --webserver_port=0 --tserver_master_addrs=127.5.240.254:44908 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.195 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:39:18.967164 6407 tablet_server_main.cc:85] Initializing tablet server... I0703 01:39:18.968709 6407 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 132708us I0703 01:39:18.969243 6407 fs_manager.cc:261] Metadata directory not provided I0703 01:39:18.969300 6407 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:39:18.969511 6407 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:39:18.969589 6407 server_base.cc:433] Attempting to create new FS layout instead I0703 01:39:18.973624 6407 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "b5cc42674ef44a0597b91c5b8fbe0e42" format_stamp: "Formatted at 2018-07-03 08:39:18 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:18.974124 6407 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "b5cc42674ef44a0597b91c5b8fbe0e42" format_stamp: "Formatted at 2018-07-03 08:39:18 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:18.981462 6407 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.003s sys 0.003s I0703 01:39:18.982686 6407 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:39:18.983042 6407 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:39:18.986591 6407 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s I0703 01:39:18.986640 6407 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal uuid: "b5cc42674ef44a0597b91c5b8fbe0e42" format_stamp: "Formatted at 2018-07-03 08:39:18 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:18.986759 6407 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:39:19.052757 6407 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:39:19.054792 6407 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:39:19.055193 6407 tablet_server_main.cc:90] Starting tablet server... I0703 01:39:19.139367 6407 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.195:57810 I0703 01:39:19.139778 6407 webserver.cc:175] Starting webserver on 127.5.240.195:0 I0703 01:39:19.139816 6407 webserver.cc:186] Document root disabled I0703 01:39:19.140957 6407 webserver.cc:313] Webserver started. Bound to: http://127.5.240.195:58392/ I0703 01:39:19.141481 6407 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb I0703 01:39:19.143246 6407 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:39:19.150815 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 6407 I0703 01:39:19.155405 6527 heartbeater.cc:346] Connected to a master server at 127.5.240.254:44908 I0703 01:39:19.155545 6527 heartbeater.cc:426] Registering TS with master... I0703 01:39:19.155995 6527 heartbeater.cc:475] Master 127.5.240.254:44908 requested a full tablet report, sending... I0703 01:39:19.157001 6119 ts_manager.cc:98] Registered new tserver with Master: b5cc42674ef44a0597b91c5b8fbe0e42 (127.5.240.195:57810) I0703 01:39:19.157845 6119 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.195:49740 I0703 01:39:19.169061 6083 external_mini_cluster.cc:490] 3 TS(s) registered with all masters I0703 01:39:19.187646 6119 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.5.240.254:53983: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 2 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0703 01:39:19.215502 6342 tablet_service.cc:759] Processing CreateTablet for tablet 4b4422e1666446a5aa36d8037d02c00d (table=TestTable [id=f479753443a945e98d55195c9a02b230]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:39:19.215754 6464 tablet_service.cc:759] Processing CreateTablet for tablet 4b4422e1666446a5aa36d8037d02c00d (table=TestTable [id=f479753443a945e98d55195c9a02b230]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:39:19.216258 6342 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 4b4422e1666446a5aa36d8037d02c00d. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:39:19.216306 6464 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 4b4422e1666446a5aa36d8037d02c00d. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:39:19.225884 6464 ts_tablet_manager.cc:1173] T 4b4422e1666446a5aa36d8037d02c00d P b5cc42674ef44a0597b91c5b8fbe0e42: Registered tablet (data state: TABLET_DATA_READY) I0703 01:39:19.225883 6342 ts_tablet_manager.cc:1173] T 4b4422e1666446a5aa36d8037d02c00d P 28da852a15344a52b350ff3d187764af: Registered tablet (data state: TABLET_DATA_READY) I0703 01:39:19.226732 6545 ts_tablet_manager.cc:1028] T 4b4422e1666446a5aa36d8037d02c00d P b5cc42674ef44a0597b91c5b8fbe0e42: Bootstrapping tablet I0703 01:39:19.226760 6544 ts_tablet_manager.cc:1028] T 4b4422e1666446a5aa36d8037d02c00d P 28da852a15344a52b350ff3d187764af: Bootstrapping tablet I0703 01:39:19.227329 6544 tablet_bootstrap.cc:427] T 4b4422e1666446a5aa36d8037d02c00d P 28da852a15344a52b350ff3d187764af: Bootstrap starting. I0703 01:39:19.227552 6545 tablet_bootstrap.cc:427] T 4b4422e1666446a5aa36d8037d02c00d P b5cc42674ef44a0597b91c5b8fbe0e42: Bootstrap starting. I0703 01:39:19.229859 6544 tablet_bootstrap.cc:578] T 4b4422e1666446a5aa36d8037d02c00d P 28da852a15344a52b350ff3d187764af: No blocks or log segments found. Creating new log. I0703 01:39:19.230343 6545 tablet_bootstrap.cc:578] T 4b4422e1666446a5aa36d8037d02c00d P b5cc42674ef44a0597b91c5b8fbe0e42: No blocks or log segments found. Creating new log. I0703 01:39:19.230593 6544 log.cc:525] T 4b4422e1666446a5aa36d8037d02c00d P 28da852a15344a52b350ff3d187764af: Log is configured to *not* fsync() on all Append() calls I0703 01:39:19.231011 6545 log.cc:525] T 4b4422e1666446a5aa36d8037d02c00d P b5cc42674ef44a0597b91c5b8fbe0e42: Log is configured to *not* fsync() on all Append() calls I0703 01:39:19.232794 6544 tablet_bootstrap.cc:427] T 4b4422e1666446a5aa36d8037d02c00d P 28da852a15344a52b350ff3d187764af: No bootstrap required, opened a new log I0703 01:39:19.232964 6544 ts_tablet_manager.cc:1045] T 4b4422e1666446a5aa36d8037d02c00d P 28da852a15344a52b350ff3d187764af: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s I0703 01:39:19.233160 6545 tablet_bootstrap.cc:427] T 4b4422e1666446a5aa36d8037d02c00d P b5cc42674ef44a0597b91c5b8fbe0e42: No bootstrap required, opened a new log I0703 01:39:19.233310 6545 ts_tablet_manager.cc:1045] T 4b4422e1666446a5aa36d8037d02c00d P b5cc42674ef44a0597b91c5b8fbe0e42: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s I0703 01:39:19.239109 6544 raft_consensus.cc:325] T 4b4422e1666446a5aa36d8037d02c00d P 28da852a15344a52b350ff3d187764af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28da852a15344a52b350ff3d187764af" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 52370 } } peers { permanent_uuid: "b5cc42674ef44a0597b91c5b8fbe0e42" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 57810 } } I0703 01:39:19.239389 6544 raft_consensus.cc:628] T 4b4422e1666446a5aa36d8037d02c00d P 28da852a15344a52b350ff3d187764af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28da852a15344a52b350ff3d187764af, State: Initialized, Role: FOLLOWER I0703 01:39:19.239539 6545 raft_consensus.cc:325] T 4b4422e1666446a5aa36d8037d02c00d P b5cc42674ef44a0597b91c5b8fbe0e42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28da852a15344a52b350ff3d187764af" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 52370 } } peers { permanent_uuid: "b5cc42674ef44a0597b91c5b8fbe0e42" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 57810 } } I0703 01:39:19.239673 6544 consensus_queue.cc:227] T 4b4422e1666446a5aa36d8037d02c00d P 28da852a15344a52b350ff3d187764af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28da852a15344a52b350ff3d187764af" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 52370 } } peers { permanent_uuid: "b5cc42674ef44a0597b91c5b8fbe0e42" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 57810 } } I0703 01:39:19.239784 6545 raft_consensus.cc:628] T 4b4422e1666446a5aa36d8037d02c00d P b5cc42674ef44a0597b91c5b8fbe0e42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5cc42674ef44a0597b91c5b8fbe0e42, State: Initialized, Role: FOLLOWER I0703 01:39:19.240021 6545 consensus_queue.cc:227] T 4b4422e1666446a5aa36d8037d02c00d P b5cc42674ef44a0597b91c5b8fbe0e42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28da852a15344a52b350ff3d187764af" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 52370 } } peers { permanent_uuid: "b5cc42674ef44a0597b91c5b8fbe0e42" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 57810 } } I0703 01:39:19.241114 6405 heartbeater.cc:467] Master 127.5.240.254:44908 was elected leader, sending a full tablet report... I0703 01:39:19.241484 6544 ts_tablet_manager.cc:1073] T 4b4422e1666446a5aa36d8037d02c00d P 28da852a15344a52b350ff3d187764af: Time spent starting tablet: real 0.008s user 0.006s sys 0.002s I0703 01:39:19.242179 6527 heartbeater.cc:467] Master 127.5.240.254:44908 was elected leader, sending a full tablet report... I0703 01:39:19.242444 6545 ts_tablet_manager.cc:1073] T 4b4422e1666446a5aa36d8037d02c00d P b5cc42674ef44a0597b91c5b8fbe0e42: Time spent starting tablet: real 0.009s user 0.004s sys 0.005s I0703 01:39:19.256959 6083 external_mini_cluster.cc:490] 3 TS(s) registered with all masters I0703 01:39:19.258286 6083 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 3af92b254d28411dbb9bb12787be9a8a to finish bootstrapping I0703 01:39:19.264190 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 28da852a15344a52b350ff3d187764af to finish bootstrapping I0703 01:39:19.269085 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver b5cc42674ef44a0597b91c5b8fbe0e42 to finish bootstrapping I0703 01:39:19.278193 6484 tablet_service.cc:1112] Received Run Leader Election RPC: tablet_id: "4b4422e1666446a5aa36d8037d02c00d" dest_uuid: "b5cc42674ef44a0597b91c5b8fbe0e42" from {username='jenkins'} at 127.5.240.195:34903 I0703 01:39:19.278410 6484 raft_consensus.cc:457] T 4b4422e1666446a5aa36d8037d02c00d P b5cc42674ef44a0597b91c5b8fbe0e42 [term 0 FOLLOWER]: Starting forced leader election (received explicit request) I0703 01:39:19.278520 6484 raft_consensus.cc:2782] T 4b4422e1666446a5aa36d8037d02c00d P b5cc42674ef44a0597b91c5b8fbe0e42 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:39:19.280014 6484 raft_consensus.cc:479] T 4b4422e1666446a5aa36d8037d02c00d P b5cc42674ef44a0597b91c5b8fbe0e42 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28da852a15344a52b350ff3d187764af" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 52370 } } peers { permanent_uuid: "b5cc42674ef44a0597b91c5b8fbe0e42" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 57810 } } I0703 01:39:19.280719 6484 leader_election.cc:243] T 4b4422e1666446a5aa36d8037d02c00d P b5cc42674ef44a0597b91c5b8fbe0e42 [CANDIDATE]: Term 1 election: Requesting vote from peer 28da852a15344a52b350ff3d187764af (127.5.240.194:52370) I0703 01:39:19.286309 6362 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "4b4422e1666446a5aa36d8037d02c00d" candidate_uuid: "b5cc42674ef44a0597b91c5b8fbe0e42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "28da852a15344a52b350ff3d187764af" I0703 01:39:19.287163 6362 raft_consensus.cc:2782] T 4b4422e1666446a5aa36d8037d02c00d P 28da852a15344a52b350ff3d187764af [term 0 FOLLOWER]: Advancing to term 1 I0703 01:39:19.289221 6362 raft_consensus.cc:2243] T 4b4422e1666446a5aa36d8037d02c00d P 28da852a15344a52b350ff3d187764af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5cc42674ef44a0597b91c5b8fbe0e42 in term 1. I0703 01:39:19.289891 6415 leader_election.cc:379] T 4b4422e1666446a5aa36d8037d02c00d P b5cc42674ef44a0597b91c5b8fbe0e42 [CANDIDATE]: Term 1 election: Vote granted by peer 28da852a15344a52b350ff3d187764af (127.5.240.194:52370) I0703 01:39:19.289975 6415 leader_election.cc:270] T 4b4422e1666446a5aa36d8037d02c00d P b5cc42674ef44a0597b91c5b8fbe0e42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:39:19.290352 6549 raft_consensus.cc:2537] T 4b4422e1666446a5aa36d8037d02c00d P b5cc42674ef44a0597b91c5b8fbe0e42 [term 1 FOLLOWER]: Leader election won for term 1 W0703 01:39:19.290436 6406 tablet.cc:1678] T 4b4422e1666446a5aa36d8037d02c00d P 28da852a15344a52b350ff3d187764af: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:39:19.291133 6549 raft_consensus.cc:593] T 4b4422e1666446a5aa36d8037d02c00d P b5cc42674ef44a0597b91c5b8fbe0e42 [term 1 LEADER]: Becoming Leader. State: Replica: b5cc42674ef44a0597b91c5b8fbe0e42, State: Running, Role: LEADER I0703 01:39:19.291565 6549 consensus_queue.cc:204] T 4b4422e1666446a5aa36d8037d02c00d P b5cc42674ef44a0597b91c5b8fbe0e42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28da852a15344a52b350ff3d187764af" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 52370 } } peers { permanent_uuid: "b5cc42674ef44a0597b91c5b8fbe0e42" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 57810 } } I0703 01:39:19.295029 6083 cluster_itest_util.cc:248] Not converged past 1 yet: 0.0 1.1 I0703 01:39:19.295346 6118 catalog_manager.cc:3798] T 4b4422e1666446a5aa36d8037d02c00d P b5cc42674ef44a0597b91c5b8fbe0e42 reported cstate change: term changed from 0 to 1, leader changed from to b5cc42674ef44a0597b91c5b8fbe0e42 (127.5.240.195). New cstate: current_term: 1 leader_uuid: "b5cc42674ef44a0597b91c5b8fbe0e42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28da852a15344a52b350ff3d187764af" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 52370 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5cc42674ef44a0597b91c5b8fbe0e42" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 57810 } health_report { overall_health: HEALTHY } } } W0703 01:39:19.393450 6528 tablet.cc:1678] T 4b4422e1666446a5aa36d8037d02c00d P b5cc42674ef44a0597b91c5b8fbe0e42: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:39:19.397040 6083 cluster_itest_util.cc:248] Not converged past 1 yet: 0.0 1.1 I0703 01:39:19.494593 6282 heartbeater.cc:467] Master 127.5.240.254:44908 was elected leader, sending a full tablet report... I0703 01:39:19.599058 6083 cluster_itest_util.cc:248] Not converged past 1 yet: 0.0 1.1 I0703 01:39:19.763020 6361 raft_consensus.cc:1081] T 4b4422e1666446a5aa36d8037d02c00d P 28da852a15344a52b350ff3d187764af [term 1 FOLLOWER]: Refusing update from remote peer b5cc42674ef44a0597b91c5b8fbe0e42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0703 01:39:19.763816 6549 consensus_queue.cc:962] T 4b4422e1666446a5aa36d8037d02c00d P b5cc42674ef44a0597b91c5b8fbe0e42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28da852a15344a52b350ff3d187764af" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 52370 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0703 01:39:19.901041 6083 test_util.cc:208] Using random seed: 1074690878 I0703 01:39:19.911171 6083 test_workload.cc:306] TestWorkload: Skipping table creation because table TestTable already exists I0703 01:39:19.914611 6083 kudu-admin-test.cc:174] Adding replica at tserver with UUID 3af92b254d28411dbb9bb12787be9a8a as VOTER... W0703 01:39:20.150032 6564 flags.cc:406] Enabled unsafe flag: --never_fsync=true I0703 01:39:20.215445 6484 consensus_peers.cc:455] T 4b4422e1666446a5aa36d8037d02c00d P b5cc42674ef44a0597b91c5b8fbe0e42 -> Peer 28da852a15344a52b350ff3d187764af (127.5.240.194:52370): Closing peer: 28da852a15344a52b350ff3d187764af I0703 01:39:20.215718 6484 consensus_queue.cc:204] T 4b4422e1666446a5aa36d8037d02c00d P b5cc42674ef44a0597b91c5b8fbe0e42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 59, Committed index: 59, Last appended: 1.60, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "28da852a15344a52b350ff3d187764af" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 52370 } } peers { permanent_uuid: "b5cc42674ef44a0597b91c5b8fbe0e42" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 57810 } } peers { permanent_uuid: "3af92b254d28411dbb9bb12787be9a8a" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 47305 } } I0703 01:39:20.218181 6361 raft_consensus.cc:1081] T 4b4422e1666446a5aa36d8037d02c00d P 28da852a15344a52b350ff3d187764af [term 1 FOLLOWER]: Refusing update from remote peer b5cc42674ef44a0597b91c5b8fbe0e42: Log matching property violated. Preceding OpId in replica: term: 1 index: 60. Preceding OpId from leader: term: 1 index: 61. (index mismatch) I0703 01:39:20.218739 6553 consensus_queue.cc:962] T 4b4422e1666446a5aa36d8037d02c00d P b5cc42674ef44a0597b91c5b8fbe0e42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28da852a15344a52b350ff3d187764af" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 52370 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 61, Last known committed idx: 59, Time since last communication: 0.000s I0703 01:39:20.221407 6549 raft_consensus.cc:2684] T 4b4422e1666446a5aa36d8037d02c00d P b5cc42674ef44a0597b91c5b8fbe0e42 [term 1 LEADER]: Committing config change with OpId 1.61: config changed from index -1 to 61, VOTER 3af92b254d28411dbb9bb12787be9a8a (127.5.240.193) added. New config: { opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "28da852a15344a52b350ff3d187764af" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 52370 } } peers { permanent_uuid: "b5cc42674ef44a0597b91c5b8fbe0e42" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 57810 } } peers { permanent_uuid: "3af92b254d28411dbb9bb12787be9a8a" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 47305 } } } W0703 01:39:20.223795 6415 consensus_peers.cc:433] T 4b4422e1666446a5aa36d8037d02c00d P b5cc42674ef44a0597b91c5b8fbe0e42 -> Peer 3af92b254d28411dbb9bb12787be9a8a (127.5.240.193:47305): Couldn't send request to peer 3af92b254d28411dbb9bb12787be9a8a for tablet 4b4422e1666446a5aa36d8037d02c00d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4b4422e1666446a5aa36d8037d02c00d. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:39:20.223891 6361 raft_consensus.cc:2684] T 4b4422e1666446a5aa36d8037d02c00d P 28da852a15344a52b350ff3d187764af [term 1 FOLLOWER]: Committing config change with OpId 1.61: config changed from index -1 to 61, VOTER 3af92b254d28411dbb9bb12787be9a8a (127.5.240.193) added. New config: { opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "28da852a15344a52b350ff3d187764af" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 52370 } } peers { permanent_uuid: "b5cc42674ef44a0597b91c5b8fbe0e42" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 57810 } } peers { permanent_uuid: "3af92b254d28411dbb9bb12787be9a8a" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 47305 } } } I0703 01:39:20.227094 6118 catalog_manager.cc:3798] T 4b4422e1666446a5aa36d8037d02c00d P b5cc42674ef44a0597b91c5b8fbe0e42 reported cstate change: config changed from index -1 to 61, VOTER 3af92b254d28411dbb9bb12787be9a8a (127.5.240.193) added. New cstate: current_term: 1 leader_uuid: "b5cc42674ef44a0597b91c5b8fbe0e42" committed_config { opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "28da852a15344a52b350ff3d187764af" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 52370 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5cc42674ef44a0597b91c5b8fbe0e42" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 57810 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3af92b254d28411dbb9bb12787be9a8a" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 47305 } health_report { overall_health: UNKNOWN } } } I0703 01:39:20.287933 6083 kudu-admin-test.cc:194] Waiting for replicas to agree... I0703 01:39:20.293943 6083 cluster_itest_util.cc:248] Not converged past 74 yet: 1.74 1.74 I0703 01:39:20.396559 6083 cluster_itest_util.cc:248] Not converged past 74 yet: 1.74 1.74 I0703 01:39:20.599213 6083 cluster_itest_util.cc:248] Not converged past 74 yet: 1.74 1.74 I0703 01:39:20.712137 6590 ts_tablet_manager.cc:669] T 4b4422e1666446a5aa36d8037d02c00d P 3af92b254d28411dbb9bb12787be9a8a: Initiating tablet copy from peer b5cc42674ef44a0597b91c5b8fbe0e42 (127.5.240.195:57810) I0703 01:39:20.712412 6590 tablet_copy_client.cc:239] T 4b4422e1666446a5aa36d8037d02c00d P 3af92b254d28411dbb9bb12787be9a8a: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.195:57810 I0703 01:39:20.718677 6504 tablet_copy_service.cc:135] P b5cc42674ef44a0597b91c5b8fbe0e42: Received BeginTabletCopySession request for tablet 4b4422e1666446a5aa36d8037d02c00d from peer 3af92b254d28411dbb9bb12787be9a8a ({username='jenkins'} at 127.5.240.193:57529) I0703 01:39:20.718813 6504 tablet_copy_service.cc:156] P b5cc42674ef44a0597b91c5b8fbe0e42: Beginning new tablet copy session on tablet 4b4422e1666446a5aa36d8037d02c00d from peer 3af92b254d28411dbb9bb12787be9a8a at {username='jenkins'} at 127.5.240.193:57529: session id = 3af92b254d28411dbb9bb12787be9a8a-4b4422e1666446a5aa36d8037d02c00d I0703 01:39:20.720726 6504 tablet_copy_source_session.cc:202] T 4b4422e1666446a5aa36d8037d02c00d P b5cc42674ef44a0597b91c5b8fbe0e42: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:39:20.723528 6590 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 4b4422e1666446a5aa36d8037d02c00d. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:39:20.733117 6590 ts_tablet_manager.cc:1173] T 4b4422e1666446a5aa36d8037d02c00d P 3af92b254d28411dbb9bb12787be9a8a: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:39:20.733361 6590 tablet_copy_client.cc:566] T 4b4422e1666446a5aa36d8037d02c00d P 3af92b254d28411dbb9bb12787be9a8a: tablet copy: Starting download of 0 data blocks... I0703 01:39:20.733533 6590 tablet_copy_client.cc:529] T 4b4422e1666446a5aa36d8037d02c00d P 3af92b254d28411dbb9bb12787be9a8a: tablet copy: Starting download of 1 WAL segments... I0703 01:39:20.735999 6590 tablet_copy_client.cc:405] T 4b4422e1666446a5aa36d8037d02c00d P 3af92b254d28411dbb9bb12787be9a8a: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:39:20.737656 6590 ts_tablet_manager.cc:1028] T 4b4422e1666446a5aa36d8037d02c00d P 3af92b254d28411dbb9bb12787be9a8a: Bootstrapping tablet I0703 01:39:20.737869 6590 tablet_bootstrap.cc:427] T 4b4422e1666446a5aa36d8037d02c00d P 3af92b254d28411dbb9bb12787be9a8a: Bootstrap starting. I0703 01:39:20.749068 6590 log.cc:525] T 4b4422e1666446a5aa36d8037d02c00d P 3af92b254d28411dbb9bb12787be9a8a: Log is configured to *not* fsync() on all Append() calls I0703 01:39:20.800390 6590 tablet_bootstrap.cc:427] T 4b4422e1666446a5aa36d8037d02c00d P 3af92b254d28411dbb9bb12787be9a8a: Bootstrap replayed 1/1 log segments. Stats: ops{read=74 overwritten=0 applied=74 ignored=0} inserts{seen=72 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:39:20.800807 6590 tablet_bootstrap.cc:427] T 4b4422e1666446a5aa36d8037d02c00d P 3af92b254d28411dbb9bb12787be9a8a: Bootstrap complete. I0703 01:39:20.801074 6590 ts_tablet_manager.cc:1045] T 4b4422e1666446a5aa36d8037d02c00d P 3af92b254d28411dbb9bb12787be9a8a: Time spent bootstrapping tablet: real 0.063s user 0.059s sys 0.003s I0703 01:39:20.805135 6590 raft_consensus.cc:325] T 4b4422e1666446a5aa36d8037d02c00d P 3af92b254d28411dbb9bb12787be9a8a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "28da852a15344a52b350ff3d187764af" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 52370 } } peers { permanent_uuid: "b5cc42674ef44a0597b91c5b8fbe0e42" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 57810 } } peers { permanent_uuid: "3af92b254d28411dbb9bb12787be9a8a" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 47305 } } I0703 01:39:20.805424 6590 raft_consensus.cc:628] T 4b4422e1666446a5aa36d8037d02c00d P 3af92b254d28411dbb9bb12787be9a8a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3af92b254d28411dbb9bb12787be9a8a, State: Initialized, Role: FOLLOWER I0703 01:39:20.805719 6590 consensus_queue.cc:227] T 4b4422e1666446a5aa36d8037d02c00d P 3af92b254d28411dbb9bb12787be9a8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 74, Last appended: 1.74, Last appended by leader: 74, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 61 OBSOLETE_local: false peers { permanent_uuid: "28da852a15344a52b350ff3d187764af" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 52370 } } peers { permanent_uuid: "b5cc42674ef44a0597b91c5b8fbe0e42" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 57810 } } peers { permanent_uuid: "3af92b254d28411dbb9bb12787be9a8a" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 47305 } } I0703 01:39:20.807009 6590 ts_tablet_manager.cc:1073] T 4b4422e1666446a5aa36d8037d02c00d P 3af92b254d28411dbb9bb12787be9a8a: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s I0703 01:39:20.807881 6504 tablet_copy_service.cc:325] P b5cc42674ef44a0597b91c5b8fbe0e42: Request end of tablet copy session 3af92b254d28411dbb9bb12787be9a8a-4b4422e1666446a5aa36d8037d02c00d received from {username='jenkins'} at 127.5.240.193:57529 I0703 01:39:20.807971 6504 tablet_copy_service.cc:413] P b5cc42674ef44a0597b91c5b8fbe0e42: ending tablet copy session 3af92b254d28411dbb9bb12787be9a8a-4b4422e1666446a5aa36d8037d02c00d on tablet 4b4422e1666446a5aa36d8037d02c00d with peer 3af92b254d28411dbb9bb12787be9a8a I0703 01:39:20.901887 6083 kudu-admin-test.cc:205] Number of rows inserted: 72 Master Summary UUID | Address | Status ----------------------------------+---------------------+--------- 70c5178815714ecdb7563bf7cb45db18 | 127.5.240.254:44908 | HEALTHY Flag | Value | Tags | Master ------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+------------------------- allow_unsafe_replication_factor | true | unsafe | all 1 server(s) checked catalog_manager_evict_excess_replicas | false | runtime,hidden | all 1 server(s) checked catalog_manager_wait_for_new_tablets_to_elect_leader | false | hidden | all 1 server(s) checked ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status ----------------------------------+---------------------+--------- 28da852a15344a52b350ff3d187764af | 127.5.240.194:52370 | HEALTHY 3af92b254d28411dbb9bb12787be9a8a | 127.5.240.193:47305 | HEALTHY b5cc42674ef44a0597b91c5b8fbe0e42 | 127.5.240.195:57810 | HEALTHY Flag | Value | Tags | Tablet Server ----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+------------------------- enable_leader_failure_detection | false | unsafe | all 3 server(s) checked ipki_server_key_size | 1024 | experimental | all 3 server(s) checked local_ip_for_outbound_sockets | 127.5.240.193 | experimental | 127.5.240.193:47305 local_ip_for_outbound_sockets | 127.5.240.194 | experimental | 127.5.240.194:52370 local_ip_for_outbound_sockets | 127.5.240.195 | experimental | 127.5.240.195:57810 never_fsync | true | unsafe,advanced | all 3 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked server_dump_info_format | pb | hidden | all 3 server(s) checked server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.5.240.193:47305 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.240.194:52370 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.240.195:57810 Version Summary Version | Servers ----------------+------------------------- 1.8.0-SNAPSHOT | all 4 server(s) checked Tablet 4b4422e1666446a5aa36d8037d02c00d of table 'TestTable' is conflicted: Tablet 4b4422e1666446a5aa36d8037d02c00d of table 'TestTable' replicas' active configs disagree with the master's 28da852a15344a52b350ff3d187764af (127.5.240.194:52370): RUNNING b5cc42674ef44a0597b91c5b8fbe0e42 (127.5.240.195:57810): RUNNING [LEADER] 3af92b254d28411dbb9bb12787be9a8a (127.5.240.193:47305): RUNNING All reported replicas are: A = 28da852a15344a52b350ff3d187764af B = b5cc42674ef44a0597b91c5b8fbe0e42 C = 3af92b254d28411dbb9bb12787be9a8a The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+--------------+--------------+--------------+------------ master | A B* C | | | Yes A | A B* C | 1 | 61 | Yes B | A B* C | 1 | 61 | Yes C | A B C | 1 | 61 | Yes Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable -----------+----+--------------------+---------------+---------+------------+------------------+------------- TestTable | 2 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1 | Total Count ----------------+------------- Masters | 1 Tablet Servers | 3 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0703 01:39:20.976728 6083 cluster_verifier.cc:82] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors Master Summary UUID | Address | Status ----------------------------------+---------------------+--------- 70c5178815714ecdb7563bf7cb45db18 | 127.5.240.254:44908 | HEALTHY Flag | Value | Tags | Master ------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+------------------------- allow_unsafe_replication_factor | true | unsafe | all 1 server(s) checked catalog_manager_evict_excess_replicas | false | runtime,hidden | all 1 server(s) checked catalog_manager_wait_for_new_tablets_to_elect_leader | false | hidden | all 1 server(s) checked ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status ----------------------------------+---------------------+--------- 28da852a15344a52b350ff3d187764af | 127.5.240.194:52370 | HEALTHY 3af92b254d28411dbb9bb12787be9a8a | 127.5.240.193:47305 | HEALTHY b5cc42674ef44a0597b91c5b8fbe0e42 | 127.5.240.195:57810 | HEALTHY Flag | Value | Tags | Tablet Server ----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+------------------------- enable_leader_failure_detection | false | unsafe | all 3 server(s) checked ipki_server_key_size | 1024 | experimental | all 3 server(s) checked local_ip_for_outbound_sockets | 127.5.240.193 | experimental | 127.5.240.193:47305 local_ip_for_outbound_sockets | 127.5.240.194 | experimental | 127.5.240.194:52370 local_ip_for_outbound_sockets | 127.5.240.195 | experimental | 127.5.240.195:57810 never_fsync | true | unsafe,advanced | all 3 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked server_dump_info_format | pb | hidden | all 3 server(s) checked server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.5.240.193:47305 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.240.194:52370 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.240.195:57810 Version Summary Version | Servers ----------------+------------------------- 1.8.0-SNAPSHOT | all 4 server(s) checked Tablet 4b4422e1666446a5aa36d8037d02c00d of table 'TestTable' is conflicted: Tablet 4b4422e1666446a5aa36d8037d02c00d of table 'TestTable' replicas' active configs disagree with the master's 28da852a15344a52b350ff3d187764af (127.5.240.194:52370): RUNNING b5cc42674ef44a0597b91c5b8fbe0e42 (127.5.240.195:57810): RUNNING [LEADER] 3af92b254d28411dbb9bb12787be9a8a (127.5.240.193:47305): RUNNING All reported replicas are: A = 28da852a15344a52b350ff3d187764af B = b5cc42674ef44a0597b91c5b8fbe0e42 C = 3af92b254d28411dbb9bb12787be9a8a The consensus matrix is: Config source | Replicas | Current term | Config index | Committed? ---------------+--------------+--------------+--------------+------------ master | A B* C | | | Yes A | A B* C | 1 | 61 | Yes B | A B* C | 1 | 61 | Yes C | A B C | 1 | 61 | Yes Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable -----------+----+--------------------+---------------+---------+------------+------------------+------------- TestTable | 2 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1 | Total Count ----------------+------------- Masters | 1 Tablet Servers | 3 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set ================== Errors: ================== Corruption: table consistency check error: 1 out of 1 table(s) are not healthy FAILED I0703 01:39:21.176039 6083 cluster_verifier.cc:82] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors I0703 01:39:21.308312 6239 raft_consensus.cc:1026] T 4b4422e1666446a5aa36d8037d02c00d P 3af92b254d28411dbb9bb12787be9a8a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.60->[1.61-1.74] Dedup: 1.74->[] Master Summary UUID | Address | Status ----------------------------------+---------------------+--------- 70c5178815714ecdb7563bf7cb45db18 | 127.5.240.254:44908 | HEALTHY Flag | Value | Tags | Master ------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+------------------------- allow_unsafe_replication_factor | true | unsafe | all 1 server(s) checked catalog_manager_evict_excess_replicas | false | runtime,hidden | all 1 server(s) checked catalog_manager_wait_for_new_tablets_to_elect_leader | false | hidden | all 1 server(s) checked ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status ----------------------------------+---------------------+--------- 28da852a15344a52b350ff3d187764af | 127.5.240.194:52370 | HEALTHY 3af92b254d28411dbb9bb12787be9a8a | 127.5.240.193:47305 | HEALTHY b5cc42674ef44a0597b91c5b8fbe0e42 | 127.5.240.195:57810 | HEALTHY Flag | Value | Tags | Tablet Server ----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+------------------------- enable_leader_failure_detection | false | unsafe | all 3 server(s) checked ipki_server_key_size | 1024 | experimental | all 3 server(s) checked local_ip_for_outbound_sockets | 127.5.240.193 | experimental | 127.5.240.193:47305 local_ip_for_outbound_sockets | 127.5.240.194 | experimental | 127.5.240.194:52370 local_ip_for_outbound_sockets | 127.5.240.195 | experimental | 127.5.240.195:57810 never_fsync | true | unsafe,advanced | all 3 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked server_dump_info_format | pb | hidden | all 3 server(s) checked server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.5.240.193:47305 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.240.194:52370 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.240.195:57810 Version Summary Version | Servers ----------------+------------------------- 1.8.0-SNAPSHOT | all 4 server(s) checked Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable -----------+----+---------+---------------+---------+------------+------------------+------------- TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0 | Total Count ----------------+------------- Masters | 1 Tablet Servers | 3 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set OK I0703 01:39:21.455008 6083 log_verifier.cc:121] Checking tablet 4b4422e1666446a5aa36d8037d02c00d I0703 01:39:21.518335 6083 log_verifier.cc:172] Verified matching terms for 74 ops in tablet 4b4422e1666446a5aa36d8037d02c00d I0703 01:39:21.821092 6083 kudu-admin-test.cc:212] Removing replica at tserver with UUID 3af92b254d28411dbb9bb12787be9a8a from the config... W0703 01:39:22.033143 6666 flags.cc:406] Enabled unsafe flag: --never_fsync=true I0703 01:39:22.096086 6484 consensus_peers.cc:455] T 4b4422e1666446a5aa36d8037d02c00d P b5cc42674ef44a0597b91c5b8fbe0e42 -> Peer 3af92b254d28411dbb9bb12787be9a8a (127.5.240.193:47305): Closing peer: 3af92b254d28411dbb9bb12787be9a8a I0703 01:39:22.096163 6484 consensus_peers.cc:455] T 4b4422e1666446a5aa36d8037d02c00d P b5cc42674ef44a0597b91c5b8fbe0e42 -> Peer 28da852a15344a52b350ff3d187764af (127.5.240.194:52370): Closing peer: 28da852a15344a52b350ff3d187764af I0703 01:39:22.096410 6484 consensus_queue.cc:204] T 4b4422e1666446a5aa36d8037d02c00d P b5cc42674ef44a0597b91c5b8fbe0e42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 74, Committed index: 74, Last appended: 1.74, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "28da852a15344a52b350ff3d187764af" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 52370 } } peers { permanent_uuid: "b5cc42674ef44a0597b91c5b8fbe0e42" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 57810 } } I0703 01:39:22.098225 6361 raft_consensus.cc:1081] T 4b4422e1666446a5aa36d8037d02c00d P 28da852a15344a52b350ff3d187764af [term 1 FOLLOWER]: Refusing update from remote peer b5cc42674ef44a0597b91c5b8fbe0e42: Log matching property violated. Preceding OpId in replica: term: 1 index: 74. Preceding OpId from leader: term: 1 index: 75. (index mismatch) I0703 01:39:22.098843 6637 consensus_queue.cc:962] T 4b4422e1666446a5aa36d8037d02c00d P b5cc42674ef44a0597b91c5b8fbe0e42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28da852a15344a52b350ff3d187764af" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 52370 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 75, Last known committed idx: 74, Time since last communication: 0.000s I0703 01:39:22.101893 6361 raft_consensus.cc:2684] T 4b4422e1666446a5aa36d8037d02c00d P 28da852a15344a52b350ff3d187764af [term 1 FOLLOWER]: Committing config change with OpId 1.75: config changed from index 61 to 75, VOTER 3af92b254d28411dbb9bb12787be9a8a (127.5.240.193) evicted. New config: { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "28da852a15344a52b350ff3d187764af" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 52370 } } peers { permanent_uuid: "b5cc42674ef44a0597b91c5b8fbe0e42" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 57810 } } } I0703 01:39:22.101279 6682 raft_consensus.cc:2684] T 4b4422e1666446a5aa36d8037d02c00d P b5cc42674ef44a0597b91c5b8fbe0e42 [term 1 LEADER]: Committing config change with OpId 1.75: config changed from index 61 to 75, VOTER 3af92b254d28411dbb9bb12787be9a8a (127.5.240.193) evicted. New config: { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "28da852a15344a52b350ff3d187764af" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 52370 } } peers { permanent_uuid: "b5cc42674ef44a0597b91c5b8fbe0e42" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 57810 } } } I0703 01:39:22.105217 6119 catalog_manager.cc:3798] T 4b4422e1666446a5aa36d8037d02c00d P 28da852a15344a52b350ff3d187764af reported cstate change: config changed from index 61 to 75, VOTER 3af92b254d28411dbb9bb12787be9a8a (127.5.240.193) evicted. New cstate: current_term: 1 leader_uuid: "b5cc42674ef44a0597b91c5b8fbe0e42" committed_config { opid_index: 75 OBSOLETE_local: false peers { permanent_uuid: "28da852a15344a52b350ff3d187764af" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 52370 } } peers { permanent_uuid: "b5cc42674ef44a0597b91c5b8fbe0e42" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 57810 } } } I0703 01:39:22.109616 6119 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 4b4422e1666446a5aa36d8037d02c00d on 3af92b254d28411dbb9bb12787be9a8a (127.5.240.193:47305) (TS 3af92b254d28411dbb9bb12787be9a8a not found in new config with opid_index 75) I0703 01:39:22.116457 6219 tablet_service.cc:800] Processing DeleteTablet for tablet 4b4422e1666446a5aa36d8037d02c00d with delete_type TABLET_DATA_TOMBSTONED (TS 3af92b254d28411dbb9bb12787be9a8a not found in new config with opid_index 75) from {username='jenkins'} at 127.5.240.193:45440 I0703 01:39:22.117516 6690 tablet_replica.cc:261] T 4b4422e1666446a5aa36d8037d02c00d P 3af92b254d28411dbb9bb12787be9a8a: stopping tablet replica I0703 01:39:22.117727 6690 maintenance_manager.cc:237] P 3af92b254d28411dbb9bb12787be9a8a: Unregistered op CompactRowSetsOp(4b4422e1666446a5aa36d8037d02c00d) I0703 01:39:22.117820 6690 maintenance_manager.cc:237] P 3af92b254d28411dbb9bb12787be9a8a: Unregistered op MinorDeltaCompactionOp(4b4422e1666446a5aa36d8037d02c00d) I0703 01:39:22.117869 6690 maintenance_manager.cc:237] P 3af92b254d28411dbb9bb12787be9a8a: Unregistered op MajorDeltaCompactionOp(4b4422e1666446a5aa36d8037d02c00d) I0703 01:39:22.117924 6690 maintenance_manager.cc:237] P 3af92b254d28411dbb9bb12787be9a8a: Unregistered op UndoDeltaBlockGCOp(4b4422e1666446a5aa36d8037d02c00d) I0703 01:39:22.118001 6690 maintenance_manager.cc:237] P 3af92b254d28411dbb9bb12787be9a8a: Unregistered op FlushMRSOp(4b4422e1666446a5aa36d8037d02c00d) I0703 01:39:22.118052 6690 maintenance_manager.cc:237] P 3af92b254d28411dbb9bb12787be9a8a: Unregistered op FlushDeltaMemStoresOp(4b4422e1666446a5aa36d8037d02c00d) I0703 01:39:22.118093 6690 maintenance_manager.cc:237] P 3af92b254d28411dbb9bb12787be9a8a: Unregistered op LogGCOp(4b4422e1666446a5aa36d8037d02c00d) I0703 01:39:22.118156 6690 raft_consensus.cc:2040] T 4b4422e1666446a5aa36d8037d02c00d P 3af92b254d28411dbb9bb12787be9a8a [term 1 FOLLOWER]: Raft consensus shutting down. I0703 01:39:22.118366 6690 raft_consensus.cc:2067] T 4b4422e1666446a5aa36d8037d02c00d P 3af92b254d28411dbb9bb12787be9a8a [term 1 FOLLOWER]: Raft consensus is shut down! I0703 01:39:22.120052 6690 ts_tablet_manager.cc:1369] T 4b4422e1666446a5aa36d8037d02c00d P 3af92b254d28411dbb9bb12787be9a8a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:39:22.122961 6690 ts_tablet_manager.cc:1382] T 4b4422e1666446a5aa36d8037d02c00d P 3af92b254d28411dbb9bb12787be9a8a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.74 I0703 01:39:22.123055 6690 log.cc:986] T 4b4422e1666446a5aa36d8037d02c00d P 3af92b254d28411dbb9bb12787be9a8a: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestChangeConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal/wals/4b4422e1666446a5aa36d8037d02c00d I0703 01:39:22.123733 6100 catalog_manager.cc:3103] TS 3af92b254d28411dbb9bb12787be9a8a (127.5.240.193:47305): tablet 4b4422e1666446a5aa36d8037d02c00d (table TestTable [id=f479753443a945e98d55195c9a02b230]) successfully deleted I0703 01:39:22.162243 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 6153 I0703 01:39:22.163978 6279 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:39:22.229446 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 6153 W0703 01:39:22.241586 6417 connection.cc:512] server connection from 127.5.240.193:57529 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:57529): Connection reset by peer (error 104) W0703 01:39:22.241542 6102 connection.cc:512] server connection from 127.5.240.193:54287 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:54287): Connection reset by peer (error 104) W0703 01:39:22.242069 6415 connection.cc:512] client connection to 127.5.240.193:47305 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:47305): Connection reset by peer (error 104) I0703 01:39:22.242198 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 6286 I0703 01:39:22.242885 6402 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:39:22.290153 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 6286 W0703 01:39:22.298888 6101 connection.cc:512] server connection from 127.5.240.194:39593 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:39593): Connection reset by peer (error 104) W0703 01:39:22.298895 6415 connection.cc:512] client connection to 127.5.240.194:52370 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:52370): Connection reset by peer (error 104) I0703 01:39:22.299232 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 6407 I0703 01:39:22.299954 6524 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:39:22.349592 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 6407 W0703 01:39:22.358371 6101 connection.cc:512] server connection from 127.5.240.195:49740 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.195:49740): Connection reset by peer (error 104) I0703 01:39:22.358505 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master pid 6093 I0703 01:39:22.359304 6149 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:39:22.424603 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master with pid 6093 [ OK ] AdminCliTest.TestChangeConfig (4819 ms) [ RUN ] AdminCliTest.TestUnsafeChangeConfigOnSingleFollower I0703 01:39:22.447715 6083 test_util.cc:208] Using random seed: 1077237584 I0703 01:39:22.449419 6083 ts_itest-base.cc:113] Starting cluster with: I0703 01:39:22.449458 6083 ts_itest-base.cc:114] -------------- I0703 01:39:22.449487 6083 ts_itest-base.cc:115] 5 tablet servers I0703 01:39:22.449514 6083 ts_itest-base.cc:116] 3 replicas per TS I0703 01:39:22.449553 6083 ts_itest-base.cc:117] -------------- I0703 01:39:22.450727 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:54399 --webserver_interface=127.5.240.254 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:39:22.631860 6701 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:39:22.632095 6701 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:39:22.634332 6701 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0703 01:39:22.634382 6701 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:39:22.634433 6701 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0703 01:39:22.634477 6701 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0703 01:39:22.641746 6701 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:54399 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.5.240.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:39:22.647426 6701 master_main.cc:80] Initializing master server... I0703 01:39:22.648526 6701 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 134708us I0703 01:39:22.649643 6701 fs_manager.cc:261] Metadata directory not provided I0703 01:39:22.649708 6701 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:39:22.649950 6701 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:39:22.650012 6701 server_base.cc:433] Attempting to create new FS layout instead I0703 01:39:22.654163 6701 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/instance: uuid: "95bbdac381bb4ab690642dd43f1992e6" format_stamp: "Formatted at 2018-07-03 08:39:22 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:22.654717 6701 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "95bbdac381bb4ab690642dd43f1992e6" format_stamp: "Formatted at 2018-07-03 08:39:22 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:22.663293 6701 fs_manager.cc:501] Time spent creating directory manager: real 0.008s user 0.002s sys 0.005s I0703 01:39:22.664502 6701 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:39:22.664913 6701 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:39:22.668112 6701 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0703 01:39:22.668170 6701 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal uuid: "95bbdac381bb4ab690642dd43f1992e6" format_stamp: "Formatted at 2018-07-03 08:39:22 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:22.668279 6701 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:39:22.720042 6701 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:39:22.721041 6701 master_main.cc:83] Starting Master server... I0703 01:39:22.757719 6701 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.254:54399 I0703 01:39:22.758173 6701 webserver.cc:175] Starting webserver on 127.5.240.254:0 I0703 01:39:22.758219 6701 webserver.cc:186] Document root disabled I0703 01:39:22.759392 6701 webserver.cc:313] Webserver started. Bound to: http://127.5.240.254:45037/ I0703 01:39:22.759994 6701 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb I0703 01:39:22.760323 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master as pid 6701 I0703 01:39:22.761162 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.5.240.193:0 --local_ip_for_outbound_sockets=127.5.240.193 --webserver_interface=127.5.240.193 --webserver_port=0 --tserver_master_addrs=127.5.240.254:54399 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0703 01:39:22.763773 6760 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:39:22.775266 6760 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 95bbdac381bb4ab690642dd43f1992e6: Bootstrap starting. I0703 01:39:22.777422 6760 tablet_bootstrap.cc:578] T 00000000000000000000000000000000 P 95bbdac381bb4ab690642dd43f1992e6: No blocks or log segments found. Creating new log. I0703 01:39:22.778156 6760 log.cc:525] T 00000000000000000000000000000000 P 95bbdac381bb4ab690642dd43f1992e6: Log is configured to *not* fsync() on all Append() calls I0703 01:39:22.780460 6760 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 95bbdac381bb4ab690642dd43f1992e6: No bootstrap required, opened a new log I0703 01:39:22.786711 6760 raft_consensus.cc:325] T 00000000000000000000000000000000 P 95bbdac381bb4ab690642dd43f1992e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95bbdac381bb4ab690642dd43f1992e6" member_type: VOTER } I0703 01:39:22.786904 6760 raft_consensus.cc:351] T 00000000000000000000000000000000 P 95bbdac381bb4ab690642dd43f1992e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:39:22.786953 6760 raft_consensus.cc:628] T 00000000000000000000000000000000 P 95bbdac381bb4ab690642dd43f1992e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95bbdac381bb4ab690642dd43f1992e6, State: Initialized, Role: FOLLOWER I0703 01:39:22.787382 6760 consensus_queue.cc:227] T 00000000000000000000000000000000 P 95bbdac381bb4ab690642dd43f1992e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95bbdac381bb4ab690642dd43f1992e6" member_type: VOTER } I0703 01:39:22.787523 6760 raft_consensus.cc:365] T 00000000000000000000000000000000 P 95bbdac381bb4ab690642dd43f1992e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:39:22.787587 6760 raft_consensus.cc:457] T 00000000000000000000000000000000 P 95bbdac381bb4ab690642dd43f1992e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:39:22.787679 6760 raft_consensus.cc:2730] T 00000000000000000000000000000000 P 95bbdac381bb4ab690642dd43f1992e6: Snoozing failure detection for 2.065s (starting election) I0703 01:39:22.787755 6760 raft_consensus.cc:2782] T 00000000000000000000000000000000 P 95bbdac381bb4ab690642dd43f1992e6 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:39:22.789132 6760 raft_consensus.cc:479] T 00000000000000000000000000000000 P 95bbdac381bb4ab690642dd43f1992e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95bbdac381bb4ab690642dd43f1992e6" member_type: VOTER } I0703 01:39:22.789484 6760 leader_election.cc:270] T 00000000000000000000000000000000 P 95bbdac381bb4ab690642dd43f1992e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:39:22.790959 6763 raft_consensus.cc:2730] T 00000000000000000000000000000000 P 95bbdac381bb4ab690642dd43f1992e6: Snoozing failure detection for 2.142s (election complete) I0703 01:39:22.791435 6763 raft_consensus.cc:2537] T 00000000000000000000000000000000 P 95bbdac381bb4ab690642dd43f1992e6 [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:39:22.792857 6763 raft_consensus.cc:593] T 00000000000000000000000000000000 P 95bbdac381bb4ab690642dd43f1992e6 [term 1 LEADER]: Becoming Leader. State: Replica: 95bbdac381bb4ab690642dd43f1992e6, State: Running, Role: LEADER I0703 01:39:22.793233 6763 consensus_queue.cc:204] T 00000000000000000000000000000000 P 95bbdac381bb4ab690642dd43f1992e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95bbdac381bb4ab690642dd43f1992e6" member_type: VOTER } I0703 01:39:22.795684 6760 sys_catalog.cc:337] T 00000000000000000000000000000000 P 95bbdac381bb4ab690642dd43f1992e6 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "95bbdac381bb4ab690642dd43f1992e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95bbdac381bb4ab690642dd43f1992e6" member_type: VOTER } } I0703 01:39:22.796020 6760 sys_catalog.cc:340] T 00000000000000000000000000000000 P 95bbdac381bb4ab690642dd43f1992e6 [sys.catalog]: This master's current role is: LEADER I0703 01:39:22.796016 6764 sys_catalog.cc:337] T 00000000000000000000000000000000 P 95bbdac381bb4ab690642dd43f1992e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95bbdac381bb4ab690642dd43f1992e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95bbdac381bb4ab690642dd43f1992e6" member_type: VOTER } } I0703 01:39:22.796017 6765 sys_catalog.cc:337] T 00000000000000000000000000000000 P 95bbdac381bb4ab690642dd43f1992e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95bbdac381bb4ab690642dd43f1992e6. Latest consensus state: current_term: 1 leader_uuid: "95bbdac381bb4ab690642dd43f1992e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95bbdac381bb4ab690642dd43f1992e6" member_type: VOTER } } I0703 01:39:22.797189 6764 sys_catalog.cc:340] T 00000000000000000000000000000000 P 95bbdac381bb4ab690642dd43f1992e6 [sys.catalog]: This master's current role is: LEADER I0703 01:39:22.797195 6765 sys_catalog.cc:340] T 00000000000000000000000000000000 P 95bbdac381bb4ab690642dd43f1992e6 [sys.catalog]: This master's current role is: LEADER I0703 01:39:22.798946 6760 sys_catalog.cc:424] T 00000000000000000000000000000000 P 95bbdac381bb4ab690642dd43f1992e6 [sys.catalog]: configured and running, proceeding with master startup. I0703 01:39:22.799262 6767 catalog_manager.cc:1006] Loading table and tablet metadata into memory... I0703 01:39:22.800863 6701 master_main.cc:86] Master server successfully started. I0703 01:39:22.806340 6767 catalog_manager.cc:1017] Initializing Kudu internal certificate authority... I0703 01:39:22.872084 6767 catalog_manager.cc:904] Generated new certificate authority record I0703 01:39:22.874145 6767 catalog_manager.cc:1026] Loading token signing keys... I0703 01:39:22.893966 6767 catalog_manager.cc:4102] T 00000000000000000000000000000000 P 95bbdac381bb4ab690642dd43f1992e6: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:39:22.937897 6761 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:39:22.938136 6761 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:39:22.940212 6761 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:39:22.940316 6761 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.193 I0703 01:39:22.947801 6761 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.5.240.193 --webserver_port=0 --tserver_master_addrs=127.5.240.254:54399 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.193 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:39:22.954416 6761 tablet_server_main.cc:85] Initializing tablet server... I0703 01:39:22.955628 6761 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 134708us I0703 01:39:22.956578 6761 fs_manager.cc:261] Metadata directory not provided I0703 01:39:22.956655 6761 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:39:22.956897 6761 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:39:22.956960 6761 server_base.cc:433] Attempting to create new FS layout instead I0703 01:39:22.961134 6761 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "d7071ff1216d42808d7cfb3ba8e4ee0b" format_stamp: "Formatted at 2018-07-03 08:39:22 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:22.961637 6761 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "d7071ff1216d42808d7cfb3ba8e4ee0b" format_stamp: "Formatted at 2018-07-03 08:39:22 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:22.969594 6761 fs_manager.cc:501] Time spent creating directory manager: real 0.008s user 0.003s sys 0.003s I0703 01:39:22.970811 6761 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:39:22.971209 6761 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:39:22.975230 6761 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0703 01:39:22.975289 6761 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal uuid: "d7071ff1216d42808d7cfb3ba8e4ee0b" format_stamp: "Formatted at 2018-07-03 08:39:22 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:22.975441 6761 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:39:23.016325 6761 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:39:23.018846 6761 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:39:23.019310 6761 tablet_server_main.cc:90] Starting tablet server... I0703 01:39:23.118031 6761 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.193:42085 I0703 01:39:23.118505 6761 webserver.cc:175] Starting webserver on 127.5.240.193:0 I0703 01:39:23.118543 6761 webserver.cc:186] Document root disabled I0703 01:39:23.119252 6761 webserver.cc:313] Webserver started. Bound to: http://127.5.240.193:58558/ I0703 01:39:23.119822 6761 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb I0703 01:39:23.122573 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 6761 I0703 01:39:23.123245 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.5.240.194:0 --local_ip_for_outbound_sockets=127.5.240.194 --webserver_interface=127.5.240.194 --webserver_port=0 --tserver_master_addrs=127.5.240.254:54399 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0703 01:39:23.131382 6761 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:39:23.144860 6890 heartbeater.cc:346] Connected to a master server at 127.5.240.254:54399 I0703 01:39:23.145009 6890 heartbeater.cc:426] Registering TS with master... I0703 01:39:23.145807 6890 heartbeater.cc:475] Master 127.5.240.254:54399 requested a full tablet report, sending... I0703 01:39:23.147903 6727 ts_manager.cc:98] Registered new tserver with Master: d7071ff1216d42808d7cfb3ba8e4ee0b (127.5.240.193:42085) I0703 01:39:23.151192 6727 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.193:48209 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:39:23.288283 6893 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:39:23.288496 6893 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:39:23.290534 6893 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:39:23.290659 6893 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.194 I0703 01:39:23.297236 6893 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.194:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.5.240.194 --webserver_port=0 --tserver_master_addrs=127.5.240.254:54399 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.194 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:39:23.303854 6893 tablet_server_main.cc:85] Initializing tablet server... I0703 01:39:23.304975 6893 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 135208us I0703 01:39:23.305928 6893 fs_manager.cc:261] Metadata directory not provided I0703 01:39:23.305984 6893 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:39:23.306200 6893 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:39:23.306257 6893 server_base.cc:433] Attempting to create new FS layout instead I0703 01:39:23.310305 6893 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "29c07deff3f54b10b7ad9a60bc14edeb" format_stamp: "Formatted at 2018-07-03 08:39:23 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:23.310829 6893 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "29c07deff3f54b10b7ad9a60bc14edeb" format_stamp: "Formatted at 2018-07-03 08:39:23 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:23.318083 6893 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.004s sys 0.002s I0703 01:39:23.319254 6893 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:39:23.319669 6893 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:39:23.323218 6893 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0703 01:39:23.323271 6893 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal uuid: "29c07deff3f54b10b7ad9a60bc14edeb" format_stamp: "Formatted at 2018-07-03 08:39:23 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:23.323382 6893 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:39:23.346065 6893 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:39:23.348338 6893 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:39:23.348712 6893 tablet_server_main.cc:90] Starting tablet server... I0703 01:39:23.431682 6893 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.194:48424 I0703 01:39:23.432163 6893 webserver.cc:175] Starting webserver on 127.5.240.194:0 I0703 01:39:23.432195 6893 webserver.cc:186] Document root disabled I0703 01:39:23.433351 6893 webserver.cc:313] Webserver started. Bound to: http://127.5.240.194:45499/ I0703 01:39:23.433885 6893 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb I0703 01:39:23.435869 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 6893 I0703 01:39:23.436488 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.5.240.195:0 --local_ip_for_outbound_sockets=127.5.240.195 --webserver_interface=127.5.240.195 --webserver_port=0 --tserver_master_addrs=127.5.240.254:54399 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0703 01:39:23.439489 6893 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:39:23.451998 7013 heartbeater.cc:346] Connected to a master server at 127.5.240.254:54399 I0703 01:39:23.452178 7013 heartbeater.cc:426] Registering TS with master... I0703 01:39:23.453109 7013 heartbeater.cc:475] Master 127.5.240.254:54399 requested a full tablet report, sending... I0703 01:39:23.454219 6727 ts_manager.cc:98] Registered new tserver with Master: 29c07deff3f54b10b7ad9a60bc14edeb (127.5.240.194:48424) I0703 01:39:23.455505 6727 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.194:43004 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:39:23.599697 7016 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:39:23.599907 7016 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:39:23.601869 7016 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:39:23.601958 7016 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.195 I0703 01:39:23.608817 7016 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.195:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.5.240.195 --webserver_port=0 --tserver_master_addrs=127.5.240.254:54399 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.195 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:39:23.615293 7016 tablet_server_main.cc:85] Initializing tablet server... I0703 01:39:23.616422 7016 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 135208us I0703 01:39:23.617358 7016 fs_manager.cc:261] Metadata directory not provided I0703 01:39:23.617413 7016 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:39:23.617645 7016 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:39:23.617703 7016 server_base.cc:433] Attempting to create new FS layout instead I0703 01:39:23.621726 7016 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "d45639fe421a4c0c81abaf8a04d04fa9" format_stamp: "Formatted at 2018-07-03 08:39:23 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:23.622227 7016 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "d45639fe421a4c0c81abaf8a04d04fa9" format_stamp: "Formatted at 2018-07-03 08:39:23 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:23.680614 7016 fs_manager.cc:501] Time spent creating directory manager: real 0.058s user 0.006s sys 0.000s I0703 01:39:23.681821 7016 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:39:23.682258 7016 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:39:23.685959 7016 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0703 01:39:23.686017 7016 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal uuid: "d45639fe421a4c0c81abaf8a04d04fa9" format_stamp: "Formatted at 2018-07-03 08:39:23 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:23.686146 7016 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:39:23.735090 7016 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:39:23.736901 7016 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:39:23.737193 7016 tablet_server_main.cc:90] Starting tablet server... I0703 01:39:23.821065 7016 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.195:33176 I0703 01:39:23.821492 7016 webserver.cc:175] Starting webserver on 127.5.240.195:0 I0703 01:39:23.821545 7016 webserver.cc:186] Document root disabled I0703 01:39:23.822712 7016 webserver.cc:313] Webserver started. Bound to: http://127.5.240.195:41447/ I0703 01:39:23.823235 7016 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb I0703 01:39:23.825090 7016 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:39:23.828464 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 7016 I0703 01:39:23.829397 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.5.240.196:0 --local_ip_for_outbound_sockets=127.5.240.196 --webserver_interface=127.5.240.196 --webserver_port=0 --tserver_master_addrs=127.5.240.254:54399 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0703 01:39:23.838443 7135 heartbeater.cc:346] Connected to a master server at 127.5.240.254:54399 I0703 01:39:23.838641 7135 heartbeater.cc:426] Registering TS with master... I0703 01:39:23.839610 7135 heartbeater.cc:475] Master 127.5.240.254:54399 requested a full tablet report, sending... I0703 01:39:23.840817 6727 ts_manager.cc:98] Registered new tserver with Master: d45639fe421a4c0c81abaf8a04d04fa9 (127.5.240.195:33176) I0703 01:39:23.842111 6727 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.195:39594 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:39:23.988924 7138 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:39:23.989137 7138 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:39:23.991158 7138 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:39:23.991250 7138 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.196 I0703 01:39:23.998234 7138 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.196:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb --webserver_interface=127.5.240.196 --webserver_port=0 --tserver_master_addrs=127.5.240.254:54399 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.196 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:39:24.004837 7138 tablet_server_main.cc:85] Initializing tablet server... I0703 01:39:24.005990 7138 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 135708us I0703 01:39:24.006932 7138 fs_manager.cc:261] Metadata directory not provided I0703 01:39:24.006985 7138 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:39:24.007201 7138 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:39:24.007258 7138 server_base.cc:433] Attempting to create new FS layout instead I0703 01:39:24.011298 7138 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/instance: uuid: "50f7f59cd4bf4ccdbb7fef595e615f5c" format_stamp: "Formatted at 2018-07-03 08:39:24 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:24.011829 7138 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal/instance: uuid: "50f7f59cd4bf4ccdbb7fef595e615f5c" format_stamp: "Formatted at 2018-07-03 08:39:24 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:24.019884 7138 fs_manager.cc:501] Time spent creating directory manager: real 0.008s user 0.003s sys 0.004s I0703 01:39:24.021054 7138 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:39:24.021435 7138 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:39:24.025110 7138 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0703 01:39:24.025168 7138 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal uuid: "50f7f59cd4bf4ccdbb7fef595e615f5c" format_stamp: "Formatted at 2018-07-03 08:39:24 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:24.025296 7138 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:39:24.079464 7138 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:39:24.081275 7138 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:39:24.081631 7138 tablet_server_main.cc:90] Starting tablet server... I0703 01:39:24.153719 6890 heartbeater.cc:467] Master 127.5.240.254:54399 was elected leader, sending a full tablet report... I0703 01:39:24.166633 7138 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.196:59711 I0703 01:39:24.167064 7138 webserver.cc:175] Starting webserver on 127.5.240.196:0 I0703 01:39:24.167098 7138 webserver.cc:186] Document root disabled I0703 01:39:24.168257 7138 webserver.cc:313] Webserver started. Bound to: http://127.5.240.196:52428/ I0703 01:39:24.168815 7138 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb I0703 01:39:24.170713 7138 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:39:24.172081 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 7138 I0703 01:39:24.172936 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data --block_manager=log --rpc_bind_addresses=127.5.240.197:0 --local_ip_for_outbound_sockets=127.5.240.197 --webserver_interface=127.5.240.197 --webserver_port=0 --tserver_master_addrs=127.5.240.254:54399 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0703 01:39:24.182265 7257 heartbeater.cc:346] Connected to a master server at 127.5.240.254:54399 I0703 01:39:24.182410 7257 heartbeater.cc:426] Registering TS with master... I0703 01:39:24.183233 7257 heartbeater.cc:475] Master 127.5.240.254:54399 requested a full tablet report, sending... I0703 01:39:24.184392 6727 ts_manager.cc:98] Registered new tserver with Master: 50f7f59cd4bf4ccdbb7fef595e615f5c (127.5.240.196:59711) I0703 01:39:24.185684 6727 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.196:53434 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:39:24.331883 7260 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:39:24.332145 7260 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:39:24.334211 7260 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:39:24.334321 7260 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.197 I0703 01:39:24.340826 7260 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.197:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb --webserver_interface=127.5.240.197 --webserver_port=0 --tserver_master_addrs=127.5.240.254:54399 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.197 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:39:24.348145 7260 tablet_server_main.cc:85] Initializing tablet server... I0703 01:39:24.349653 7260 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 135708us I0703 01:39:24.350606 7260 fs_manager.cc:261] Metadata directory not provided I0703 01:39:24.350670 7260 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:39:24.350903 7260 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:39:24.350965 7260 server_base.cc:433] Attempting to create new FS layout instead I0703 01:39:24.355067 7260 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/instance: uuid: "a5fbe0bff97843f89529ce5722817fad" format_stamp: "Formatted at 2018-07-03 08:39:24 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:24.355613 7260 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal/instance: uuid: "a5fbe0bff97843f89529ce5722817fad" format_stamp: "Formatted at 2018-07-03 08:39:24 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:24.363373 7260 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s I0703 01:39:24.364570 7260 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:39:24.364969 7260 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:39:24.368480 7260 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s I0703 01:39:24.368540 7260 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal uuid: "a5fbe0bff97843f89529ce5722817fad" format_stamp: "Formatted at 2018-07-03 08:39:24 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:24.368679 7260 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:39:24.398167 7260 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:39:24.400413 7260 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:39:24.400789 7260 tablet_server_main.cc:90] Starting tablet server... I0703 01:39:24.457809 7013 heartbeater.cc:467] Master 127.5.240.254:54399 was elected leader, sending a full tablet report... I0703 01:39:24.484164 7260 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.197:60719 I0703 01:39:24.484671 7260 webserver.cc:175] Starting webserver on 127.5.240.197:0 I0703 01:39:24.484709 7260 webserver.cc:186] Document root disabled I0703 01:39:24.485877 7260 webserver.cc:313] Webserver started. Bound to: http://127.5.240.197:32905/ I0703 01:39:24.486399 7260 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb I0703 01:39:24.488123 7260 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:39:24.494753 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 7260 I0703 01:39:24.501294 7379 heartbeater.cc:346] Connected to a master server at 127.5.240.254:54399 I0703 01:39:24.501426 7379 heartbeater.cc:426] Registering TS with master... I0703 01:39:24.502259 7379 heartbeater.cc:475] Master 127.5.240.254:54399 requested a full tablet report, sending... I0703 01:39:24.503238 6727 ts_manager.cc:98] Registered new tserver with Master: a5fbe0bff97843f89529ce5722817fad (127.5.240.197:60719) I0703 01:39:24.504101 6083 external_mini_cluster.cc:490] 5 TS(s) registered with all masters I0703 01:39:24.504400 6727 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.197:56268 I0703 01:39:24.513365 6726 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.5.240.254:51180: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0703 01:39:24.542752 6827 tablet_service.cc:759] Processing CreateTablet for tablet a61eb64bf7cc417684a8446570d07208 (table=TestTable [id=24e11f69554b41ef994802e369af7c41]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:39:24.543036 7316 tablet_service.cc:759] Processing CreateTablet for tablet a61eb64bf7cc417684a8446570d07208 (table=TestTable [id=24e11f69554b41ef994802e369af7c41]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:39:24.543673 6827 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet a61eb64bf7cc417684a8446570d07208. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:39:24.543763 7316 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet a61eb64bf7cc417684a8446570d07208. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:39:24.543720 6950 tablet_service.cc:759] Processing CreateTablet for tablet a61eb64bf7cc417684a8446570d07208 (table=TestTable [id=24e11f69554b41ef994802e369af7c41]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:39:24.544293 6950 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet a61eb64bf7cc417684a8446570d07208. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:39:24.554097 6950 ts_tablet_manager.cc:1173] T a61eb64bf7cc417684a8446570d07208 P 29c07deff3f54b10b7ad9a60bc14edeb: Registered tablet (data state: TABLET_DATA_READY) I0703 01:39:24.554879 7399 ts_tablet_manager.cc:1028] T a61eb64bf7cc417684a8446570d07208 P 29c07deff3f54b10b7ad9a60bc14edeb: Bootstrapping tablet I0703 01:39:24.555444 7399 tablet_bootstrap.cc:427] T a61eb64bf7cc417684a8446570d07208 P 29c07deff3f54b10b7ad9a60bc14edeb: Bootstrap starting. I0703 01:39:24.557639 7399 tablet_bootstrap.cc:578] T a61eb64bf7cc417684a8446570d07208 P 29c07deff3f54b10b7ad9a60bc14edeb: No blocks or log segments found. Creating new log. I0703 01:39:24.558354 7399 log.cc:525] T a61eb64bf7cc417684a8446570d07208 P 29c07deff3f54b10b7ad9a60bc14edeb: Log is configured to *not* fsync() on all Append() calls I0703 01:39:24.558732 6827 ts_tablet_manager.cc:1173] T a61eb64bf7cc417684a8446570d07208 P d7071ff1216d42808d7cfb3ba8e4ee0b: Registered tablet (data state: TABLET_DATA_READY) I0703 01:39:24.559011 7316 ts_tablet_manager.cc:1173] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad: Registered tablet (data state: TABLET_DATA_READY) I0703 01:39:24.561219 7399 tablet_bootstrap.cc:427] T a61eb64bf7cc417684a8446570d07208 P 29c07deff3f54b10b7ad9a60bc14edeb: No bootstrap required, opened a new log I0703 01:39:24.561393 7399 ts_tablet_manager.cc:1045] T a61eb64bf7cc417684a8446570d07208 P 29c07deff3f54b10b7ad9a60bc14edeb: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.002s I0703 01:39:24.562415 7402 ts_tablet_manager.cc:1028] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad: Bootstrapping tablet I0703 01:39:24.562436 7401 ts_tablet_manager.cc:1028] T a61eb64bf7cc417684a8446570d07208 P d7071ff1216d42808d7cfb3ba8e4ee0b: Bootstrapping tablet I0703 01:39:24.562903 7402 tablet_bootstrap.cc:427] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad: Bootstrap starting. I0703 01:39:24.562976 7401 tablet_bootstrap.cc:427] T a61eb64bf7cc417684a8446570d07208 P d7071ff1216d42808d7cfb3ba8e4ee0b: Bootstrap starting. I0703 01:39:24.565013 7401 tablet_bootstrap.cc:578] T a61eb64bf7cc417684a8446570d07208 P d7071ff1216d42808d7cfb3ba8e4ee0b: No blocks or log segments found. Creating new log. I0703 01:39:24.565743 7401 log.cc:525] T a61eb64bf7cc417684a8446570d07208 P d7071ff1216d42808d7cfb3ba8e4ee0b: Log is configured to *not* fsync() on all Append() calls I0703 01:39:24.565927 7402 tablet_bootstrap.cc:578] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad: No blocks or log segments found. Creating new log. I0703 01:39:24.566838 7402 log.cc:525] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad: Log is configured to *not* fsync() on all Append() calls I0703 01:39:24.567862 7401 tablet_bootstrap.cc:427] T a61eb64bf7cc417684a8446570d07208 P d7071ff1216d42808d7cfb3ba8e4ee0b: No bootstrap required, opened a new log I0703 01:39:24.568033 7401 ts_tablet_manager.cc:1045] T a61eb64bf7cc417684a8446570d07208 P d7071ff1216d42808d7cfb3ba8e4ee0b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s I0703 01:39:24.570106 7399 raft_consensus.cc:325] T a61eb64bf7cc417684a8446570d07208 P 29c07deff3f54b10b7ad9a60bc14edeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fbe0bff97843f89529ce5722817fad" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 60719 } } peers { permanent_uuid: "29c07deff3f54b10b7ad9a60bc14edeb" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 48424 } } peers { permanent_uuid: "d7071ff1216d42808d7cfb3ba8e4ee0b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42085 } } I0703 01:39:24.570469 7399 raft_consensus.cc:351] T a61eb64bf7cc417684a8446570d07208 P 29c07deff3f54b10b7ad9a60bc14edeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:39:24.570549 7399 raft_consensus.cc:628] T a61eb64bf7cc417684a8446570d07208 P 29c07deff3f54b10b7ad9a60bc14edeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29c07deff3f54b10b7ad9a60bc14edeb, State: Initialized, Role: FOLLOWER I0703 01:39:24.571074 7399 consensus_queue.cc:227] T a61eb64bf7cc417684a8446570d07208 P 29c07deff3f54b10b7ad9a60bc14edeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fbe0bff97843f89529ce5722817fad" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 60719 } } peers { permanent_uuid: "29c07deff3f54b10b7ad9a60bc14edeb" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 48424 } } peers { permanent_uuid: "d7071ff1216d42808d7cfb3ba8e4ee0b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42085 } } I0703 01:39:24.574173 7402 tablet_bootstrap.cc:427] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad: No bootstrap required, opened a new log I0703 01:39:24.574362 7402 ts_tablet_manager.cc:1045] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.006s I0703 01:39:24.574718 7399 ts_tablet_manager.cc:1073] T a61eb64bf7cc417684a8446570d07208 P 29c07deff3f54b10b7ad9a60bc14edeb: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s I0703 01:39:24.576539 7401 raft_consensus.cc:325] T a61eb64bf7cc417684a8446570d07208 P d7071ff1216d42808d7cfb3ba8e4ee0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fbe0bff97843f89529ce5722817fad" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 60719 } } peers { permanent_uuid: "29c07deff3f54b10b7ad9a60bc14edeb" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 48424 } } peers { permanent_uuid: "d7071ff1216d42808d7cfb3ba8e4ee0b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42085 } } I0703 01:39:24.576966 7401 raft_consensus.cc:351] T a61eb64bf7cc417684a8446570d07208 P d7071ff1216d42808d7cfb3ba8e4ee0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:39:24.577052 7401 raft_consensus.cc:628] T a61eb64bf7cc417684a8446570d07208 P d7071ff1216d42808d7cfb3ba8e4ee0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7071ff1216d42808d7cfb3ba8e4ee0b, State: Initialized, Role: FOLLOWER I0703 01:39:24.577430 7401 consensus_queue.cc:227] T a61eb64bf7cc417684a8446570d07208 P d7071ff1216d42808d7cfb3ba8e4ee0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fbe0bff97843f89529ce5722817fad" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 60719 } } peers { permanent_uuid: "29c07deff3f54b10b7ad9a60bc14edeb" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 48424 } } peers { permanent_uuid: "d7071ff1216d42808d7cfb3ba8e4ee0b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42085 } } I0703 01:39:24.578649 7401 ts_tablet_manager.cc:1073] T a61eb64bf7cc417684a8446570d07208 P d7071ff1216d42808d7cfb3ba8e4ee0b: Time spent starting tablet: real 0.011s user 0.007s sys 0.003s I0703 01:39:24.581719 7402 raft_consensus.cc:325] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fbe0bff97843f89529ce5722817fad" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 60719 } } peers { permanent_uuid: "29c07deff3f54b10b7ad9a60bc14edeb" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 48424 } } peers { permanent_uuid: "d7071ff1216d42808d7cfb3ba8e4ee0b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42085 } } I0703 01:39:24.581989 7402 raft_consensus.cc:351] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:39:24.582037 7402 raft_consensus.cc:628] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5fbe0bff97843f89529ce5722817fad, State: Initialized, Role: FOLLOWER I0703 01:39:24.582399 7402 consensus_queue.cc:227] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fbe0bff97843f89529ce5722817fad" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 60719 } } peers { permanent_uuid: "29c07deff3f54b10b7ad9a60bc14edeb" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 48424 } } peers { permanent_uuid: "d7071ff1216d42808d7cfb3ba8e4ee0b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42085 } } I0703 01:39:24.583114 7379 heartbeater.cc:467] Master 127.5.240.254:54399 was elected leader, sending a full tablet report... I0703 01:39:24.583423 7402 ts_tablet_manager.cc:1073] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad: Time spent starting tablet: real 0.009s user 0.007s sys 0.002s I0703 01:39:24.600389 7405 raft_consensus.cc:457] T a61eb64bf7cc417684a8446570d07208 P 29c07deff3f54b10b7ad9a60bc14edeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0703 01:39:24.600505 7405 raft_consensus.cc:2730] T a61eb64bf7cc417684a8446570d07208 P 29c07deff3f54b10b7ad9a60bc14edeb: Snoozing failure detection for 2.165s (starting election) I0703 01:39:24.600633 7405 raft_consensus.cc:479] T a61eb64bf7cc417684a8446570d07208 P 29c07deff3f54b10b7ad9a60bc14edeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fbe0bff97843f89529ce5722817fad" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 60719 } } peers { permanent_uuid: "29c07deff3f54b10b7ad9a60bc14edeb" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 48424 } } peers { permanent_uuid: "d7071ff1216d42808d7cfb3ba8e4ee0b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42085 } } I0703 01:39:24.601778 7405 leader_election.cc:243] T a61eb64bf7cc417684a8446570d07208 P 29c07deff3f54b10b7ad9a60bc14edeb [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer a5fbe0bff97843f89529ce5722817fad (127.5.240.197:60719) I0703 01:39:24.602092 7405 leader_election.cc:243] T a61eb64bf7cc417684a8446570d07208 P 29c07deff3f54b10b7ad9a60bc14edeb [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer d7071ff1216d42808d7cfb3ba8e4ee0b (127.5.240.193:42085) I0703 01:39:24.611013 6847 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "a61eb64bf7cc417684a8446570d07208" candidate_uuid: "29c07deff3f54b10b7ad9a60bc14edeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7071ff1216d42808d7cfb3ba8e4ee0b" is_pre_election: true I0703 01:39:24.611948 6847 raft_consensus.cc:2730] T a61eb64bf7cc417684a8446570d07208 P d7071ff1216d42808d7cfb3ba8e4ee0b: Snoozing failure detection for 1.883s (vote granted) I0703 01:39:24.612083 6847 raft_consensus.cc:2243] T a61eb64bf7cc417684a8446570d07208 P d7071ff1216d42808d7cfb3ba8e4ee0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29c07deff3f54b10b7ad9a60bc14edeb in term 0. I0703 01:39:24.611865 7336 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "a61eb64bf7cc417684a8446570d07208" candidate_uuid: "29c07deff3f54b10b7ad9a60bc14edeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5fbe0bff97843f89529ce5722817fad" is_pre_election: true I0703 01:39:24.612563 7336 raft_consensus.cc:2730] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad: Snoozing failure detection for 1.981s (vote granted) I0703 01:39:24.612696 7336 raft_consensus.cc:2243] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29c07deff3f54b10b7ad9a60bc14edeb in term 0. I0703 01:39:24.612975 6903 leader_election.cc:379] T a61eb64bf7cc417684a8446570d07208 P 29c07deff3f54b10b7ad9a60bc14edeb [CANDIDATE]: Term 1 pre-election: Vote granted by peer d7071ff1216d42808d7cfb3ba8e4ee0b (127.5.240.193:42085) I0703 01:39:24.613056 6903 leader_election.cc:270] T a61eb64bf7cc417684a8446570d07208 P 29c07deff3f54b10b7ad9a60bc14edeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0703 01:39:24.613351 7405 raft_consensus.cc:2730] T a61eb64bf7cc417684a8446570d07208 P 29c07deff3f54b10b7ad9a60bc14edeb: Snoozing failure detection for 1.878s (election complete) I0703 01:39:24.613461 7405 raft_consensus.cc:2537] T a61eb64bf7cc417684a8446570d07208 P 29c07deff3f54b10b7ad9a60bc14edeb [term 0 FOLLOWER]: Leader pre-election won for term 1 I0703 01:39:24.613572 6903 leader_election.cc:379] T a61eb64bf7cc417684a8446570d07208 P 29c07deff3f54b10b7ad9a60bc14edeb [CANDIDATE]: Term 1 pre-election: Vote granted by peer a5fbe0bff97843f89529ce5722817fad (127.5.240.197:60719) I0703 01:39:24.613574 7405 raft_consensus.cc:457] T a61eb64bf7cc417684a8446570d07208 P 29c07deff3f54b10b7ad9a60bc14edeb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0703 01:39:24.613698 7405 raft_consensus.cc:2730] T a61eb64bf7cc417684a8446570d07208 P 29c07deff3f54b10b7ad9a60bc14edeb: Snoozing failure detection for 1.833s (starting election) I0703 01:39:24.613790 7405 raft_consensus.cc:2782] T a61eb64bf7cc417684a8446570d07208 P 29c07deff3f54b10b7ad9a60bc14edeb [term 0 FOLLOWER]: Advancing to term 1 I0703 01:39:24.615296 7405 raft_consensus.cc:479] T a61eb64bf7cc417684a8446570d07208 P 29c07deff3f54b10b7ad9a60bc14edeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fbe0bff97843f89529ce5722817fad" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 60719 } } peers { permanent_uuid: "29c07deff3f54b10b7ad9a60bc14edeb" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 48424 } } peers { permanent_uuid: "d7071ff1216d42808d7cfb3ba8e4ee0b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42085 } } I0703 01:39:24.615670 7405 leader_election.cc:243] T a61eb64bf7cc417684a8446570d07208 P 29c07deff3f54b10b7ad9a60bc14edeb [CANDIDATE]: Term 1 election: Requesting vote from peer a5fbe0bff97843f89529ce5722817fad (127.5.240.197:60719) I0703 01:39:24.615830 7405 leader_election.cc:243] T a61eb64bf7cc417684a8446570d07208 P 29c07deff3f54b10b7ad9a60bc14edeb [CANDIDATE]: Term 1 election: Requesting vote from peer d7071ff1216d42808d7cfb3ba8e4ee0b (127.5.240.193:42085) I0703 01:39:24.616336 7336 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "a61eb64bf7cc417684a8446570d07208" candidate_uuid: "29c07deff3f54b10b7ad9a60bc14edeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5fbe0bff97843f89529ce5722817fad" I0703 01:39:24.616453 7336 raft_consensus.cc:2782] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad [term 0 FOLLOWER]: Advancing to term 1 I0703 01:39:24.616467 6847 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "a61eb64bf7cc417684a8446570d07208" candidate_uuid: "29c07deff3f54b10b7ad9a60bc14edeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7071ff1216d42808d7cfb3ba8e4ee0b" I0703 01:39:24.616546 7336 raft_consensus.cc:2730] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad: Snoozing failure detection for 1.629s (vote granted) I0703 01:39:24.616611 6847 raft_consensus.cc:2782] T a61eb64bf7cc417684a8446570d07208 P d7071ff1216d42808d7cfb3ba8e4ee0b [term 0 FOLLOWER]: Advancing to term 1 I0703 01:39:24.616699 6847 raft_consensus.cc:2730] T a61eb64bf7cc417684a8446570d07208 P d7071ff1216d42808d7cfb3ba8e4ee0b: Snoozing failure detection for 1.693s (vote granted) I0703 01:39:24.617894 7336 raft_consensus.cc:2243] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29c07deff3f54b10b7ad9a60bc14edeb in term 1. I0703 01:39:24.618067 6847 raft_consensus.cc:2243] T a61eb64bf7cc417684a8446570d07208 P d7071ff1216d42808d7cfb3ba8e4ee0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29c07deff3f54b10b7ad9a60bc14edeb in term 1. I0703 01:39:24.618268 6903 leader_election.cc:379] T a61eb64bf7cc417684a8446570d07208 P 29c07deff3f54b10b7ad9a60bc14edeb [CANDIDATE]: Term 1 election: Vote granted by peer a5fbe0bff97843f89529ce5722817fad (127.5.240.197:60719) I0703 01:39:24.618329 6903 leader_election.cc:270] T a61eb64bf7cc417684a8446570d07208 P 29c07deff3f54b10b7ad9a60bc14edeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:39:24.618479 7405 raft_consensus.cc:2730] T a61eb64bf7cc417684a8446570d07208 P 29c07deff3f54b10b7ad9a60bc14edeb: Snoozing failure detection for 2.165s (election complete) I0703 01:39:24.618599 7405 raft_consensus.cc:2537] T a61eb64bf7cc417684a8446570d07208 P 29c07deff3f54b10b7ad9a60bc14edeb [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:39:24.618602 6903 leader_election.cc:379] T a61eb64bf7cc417684a8446570d07208 P 29c07deff3f54b10b7ad9a60bc14edeb [CANDIDATE]: Term 1 election: Vote granted by peer d7071ff1216d42808d7cfb3ba8e4ee0b (127.5.240.193:42085) I0703 01:39:24.619457 7405 raft_consensus.cc:593] T a61eb64bf7cc417684a8446570d07208 P 29c07deff3f54b10b7ad9a60bc14edeb [term 1 LEADER]: Becoming Leader. State: Replica: 29c07deff3f54b10b7ad9a60bc14edeb, State: Running, Role: LEADER I0703 01:39:24.619928 7405 consensus_queue.cc:204] T a61eb64bf7cc417684a8446570d07208 P 29c07deff3f54b10b7ad9a60bc14edeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fbe0bff97843f89529ce5722817fad" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 60719 } } peers { permanent_uuid: "29c07deff3f54b10b7ad9a60bc14edeb" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 48424 } } peers { permanent_uuid: "d7071ff1216d42808d7cfb3ba8e4ee0b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42085 } } W0703 01:39:24.623479 6891 tablet.cc:1678] T a61eb64bf7cc417684a8446570d07208 P d7071ff1216d42808d7cfb3ba8e4ee0b: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:39:24.623638 6727 catalog_manager.cc:3798] T a61eb64bf7cc417684a8446570d07208 P 29c07deff3f54b10b7ad9a60bc14edeb reported cstate change: term changed from 0 to 1, leader changed from to 29c07deff3f54b10b7ad9a60bc14edeb (127.5.240.194). New cstate: current_term: 1 leader_uuid: "29c07deff3f54b10b7ad9a60bc14edeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fbe0bff97843f89529ce5722817fad" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 60719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29c07deff3f54b10b7ad9a60bc14edeb" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 48424 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7071ff1216d42808d7cfb3ba8e4ee0b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42085 } health_report { overall_health: UNKNOWN } } } I0703 01:39:24.629549 6083 external_mini_cluster.cc:490] 5 TS(s) registered with all masters I0703 01:39:24.630592 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver d7071ff1216d42808d7cfb3ba8e4ee0b to finish bootstrapping I0703 01:39:24.636104 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 29c07deff3f54b10b7ad9a60bc14edeb to finish bootstrapping I0703 01:39:24.640205 6083 ts_itest-base.cc:233] Waiting for 0 tablets on tserver d45639fe421a4c0c81abaf8a04d04fa9 to finish bootstrapping I0703 01:39:24.646205 6083 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 50f7f59cd4bf4ccdbb7fef595e615f5c to finish bootstrapping I0703 01:39:24.651876 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver a5fbe0bff97843f89529ce5722817fad to finish bootstrapping I0703 01:39:24.656749 6083 kudu-admin-test.cc:386] Finding tablet leader and waiting for things to start... I0703 01:39:24.656803 6083 kudu-admin-test.cc:397] Waiting for Master to see the current replicas... I0703 01:39:24.658077 6083 kudu-admin-test.cc:405] Tablet locations: tablet_id: "a61eb64bf7cc417684a8446570d07208" replicas { ts_info { permanent_uuid: "a5fbe0bff97843f89529ce5722817fad" rpc_addresses { host: "127.5.240.197" port: 60719 } } role: FOLLOWER } replicas { ts_info { permanent_uuid: "29c07deff3f54b10b7ad9a60bc14edeb" rpc_addresses { host: "127.5.240.194" port: 48424 } } role: LEADER } replicas { ts_info { permanent_uuid: "d7071ff1216d42808d7cfb3ba8e4ee0b" rpc_addresses { host: "127.5.240.193" port: 42085 } } role: FOLLOWER } DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } W0703 01:39:24.686646 7014 tablet.cc:1678] T a61eb64bf7cc417684a8446570d07208 P 29c07deff3f54b10b7ad9a60bc14edeb: Can't schedule compaction. Clean time has not been advanced past its initial value. W0703 01:39:24.738281 7380 tablet.cc:1678] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:39:24.844347 7135 heartbeater.cc:467] Master 127.5.240.254:54399 was elected leader, sending a full tablet report... I0703 01:39:25.066840 6847 raft_consensus.cc:1081] T a61eb64bf7cc417684a8446570d07208 P d7071ff1216d42808d7cfb3ba8e4ee0b [term 1 FOLLOWER]: Refusing update from remote peer 29c07deff3f54b10b7ad9a60bc14edeb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0703 01:39:25.068112 7405 consensus_queue.cc:962] T a61eb64bf7cc417684a8446570d07208 P 29c07deff3f54b10b7ad9a60bc14edeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7071ff1216d42808d7cfb3ba8e4ee0b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0703 01:39:25.073930 7336 raft_consensus.cc:1081] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad [term 1 FOLLOWER]: Refusing update from remote peer 29c07deff3f54b10b7ad9a60bc14edeb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0703 01:39:25.075723 7405 consensus_queue.cc:962] T a61eb64bf7cc417684a8446570d07208 P 29c07deff3f54b10b7ad9a60bc14edeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5fbe0bff97843f89529ce5722817fad" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 60719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0703 01:39:25.077278 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master pid 6701 I0703 01:39:25.077895 6757 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:39:25.145140 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master with pid 6701 W0703 01:39:25.157835 6904 connection.cc:512] client connection to 127.5.240.254:54399 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.254:54399): Connection reset by peer (error 104) W0703 01:39:25.157835 6781 connection.cc:512] client connection to 127.5.240.254:54399 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.254:54399): Connection reset by peer (error 104) W0703 01:39:25.158565 7026 connection.cc:512] client connection to 127.5.240.254:54399 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.254:54399): Connection reset by peer (error 104) W0703 01:39:25.159090 7148 connection.cc:512] client connection to 127.5.240.254:54399 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.254:54399): Connection reset by peer (error 104) W0703 01:39:25.159096 7270 connection.cc:512] client connection to 127.5.240.254:54399 recv error: Network error: failed to read from TLS socket (remote: 0.0.0.0:0): Connection reset by peer (error 104) W0703 01:39:25.159384 7379 heartbeater.cc:587] Failed to heartbeat to 127.5.240.254:54399: Network error: Failed to send heartbeat to master: failed to read from TLS socket (remote: 0.0.0.0:0): Connection reset by peer (error 104) W0703 01:39:25.159493 7270 connection.cc:185] Error closing socket: Network error: TlsSocket::Close: Broken pipe I0703 01:39:25.159552 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 6893 I0703 01:39:25.160328 7010 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:39:25.187999 7257 heartbeater.cc:467] Master 127.5.240.254:54399 was elected leader, sending a full tablet report... W0703 01:39:25.189256 7257 heartbeater.cc:587] Failed to heartbeat to 127.5.240.254:54399: Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.240.254:54399: connect: Connection refused (error 111) I0703 01:39:25.204661 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 6893 W0703 01:39:25.213152 7269 connection.cc:512] server connection from 127.5.240.194:57090 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:57090): Connection reset by peer (error 104) W0703 01:39:25.213155 6781 connection.cc:512] server connection from 127.5.240.194:43701 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:43701): Connection reset by peer (error 104) I0703 01:39:25.213732 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 6761 I0703 01:39:25.214494 6887 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:39:25.256563 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 6761 I0703 01:39:25.264916 6083 kudu-admin-test.cc:423] Forcing unsafe config change on remaining follower a5fbe0bff97843f89529ce5722817fad W0703 01:39:25.485536 7422 flags.cc:406] Enabled unsafe flag: --never_fsync=true I0703 01:39:25.533838 7336 tablet_service.cc:1077] Received UnsafeChangeConfig RPC: dest_uuid: "a5fbe0bff97843f89529ce5722817fad" tablet_id: "a61eb64bf7cc417684a8446570d07208" caller_id: "kudu-tools" new_config { peers { permanent_uuid: "a5fbe0bff97843f89529ce5722817fad" } } from {username='jenkins'} at 127.5.240.197:45180 W0703 01:39:25.534229 7336 raft_consensus.cc:2017] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad [term 1 FOLLOWER]: PROCEEDING WITH UNSAFE CONFIG CHANGE ON THIS SERVER, COMMITTED CONFIG: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fbe0bff97843f89529ce5722817fad" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 60719 } } peers { permanent_uuid: "29c07deff3f54b10b7ad9a60bc14edeb" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 48424 } } peers { permanent_uuid: "d7071ff1216d42808d7cfb3ba8e4ee0b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42085 } }NEW CONFIG: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a5fbe0bff97843f89529ce5722817fad" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 60719 } } unsafe_config_change: true I0703 01:39:25.534466 7336 raft_consensus.cc:2782] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad [term 1 FOLLOWER]: Advancing to term 2 W0703 01:39:25.537663 7379 heartbeater.cc:587] Failed to heartbeat to 127.5.240.254:54399: Network error: Failed to ping master at 127.5.240.254:54399: Client connection negotiation failed: client connection to 127.5.240.254:54399: connect: Connection refused (error 111) W0703 01:39:25.847239 7135 heartbeater.cc:587] Failed to heartbeat to 127.5.240.254:54399: Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.240.254:54399: connect: Connection refused (error 111) W0703 01:39:26.190598 7257 heartbeater.cc:587] Failed to heartbeat to 127.5.240.254:54399: Network error: Failed to ping master at 127.5.240.254:54399: Client connection negotiation failed: client connection to 127.5.240.254:54399: connect: Connection refused (error 111) W0703 01:39:26.538889 7379 heartbeater.cc:587] Failed to heartbeat to 127.5.240.254:54399: Network error: Failed to ping master at 127.5.240.254:54399: Client connection negotiation failed: client connection to 127.5.240.254:54399: connect: Connection refused (error 111) W0703 01:39:26.848533 7135 heartbeater.cc:587] Failed to heartbeat to 127.5.240.254:54399: Network error: Failed to ping master at 127.5.240.254:54399: Client connection negotiation failed: client connection to 127.5.240.254:54399: connect: Connection refused (error 111) I0703 01:39:27.037371 7441 raft_consensus.cc:457] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad [term 2 FOLLOWER]: Starting pre-election (detected failure of leader kudu-tools) I0703 01:39:27.037457 7441 raft_consensus.cc:2730] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad: Snoozing failure detection for 2.114s (starting election) I0703 01:39:27.037560 7441 raft_consensus.cc:479] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a5fbe0bff97843f89529ce5722817fad" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 60719 } } unsafe_config_change: true I0703 01:39:27.037953 7441 leader_election.cc:270] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. I0703 01:39:27.038415 7442 raft_consensus.cc:2730] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad: Snoozing failure detection for 1.526s (election complete) I0703 01:39:27.038480 7442 raft_consensus.cc:2537] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad [term 2 FOLLOWER]: Leader pre-election won for term 3 I0703 01:39:27.038525 7442 raft_consensus.cc:457] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad [term 2 FOLLOWER]: Starting leader election (detected failure of leader kudu-tools) I0703 01:39:27.038563 7442 raft_consensus.cc:2730] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad: Snoozing failure detection for 1.882s (starting election) I0703 01:39:27.038609 7442 raft_consensus.cc:2782] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad [term 2 FOLLOWER]: Advancing to term 3 I0703 01:39:27.040278 7442 raft_consensus.cc:479] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad [term 3 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a5fbe0bff97843f89529ce5722817fad" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 60719 } } unsafe_config_change: true I0703 01:39:27.040426 7442 leader_election.cc:270] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. I0703 01:39:27.040602 7441 raft_consensus.cc:2730] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad: Snoozing failure detection for 2.207s (election complete) I0703 01:39:27.040663 7441 raft_consensus.cc:2537] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad [term 3 FOLLOWER]: Leader election won for term 3 I0703 01:39:27.040787 7441 raft_consensus.cc:593] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad [term 3 LEADER]: Becoming Leader. State: Replica: a5fbe0bff97843f89529ce5722817fad, State: Running, Role: LEADER I0703 01:39:27.041118 7441 consensus_queue.cc:204] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 0, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a5fbe0bff97843f89529ce5722817fad" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 60719 } } unsafe_config_change: true W0703 01:39:27.042093 7379 heartbeater.cc:587] Failed to heartbeat to 127.5.240.254:54399: Network error: Failed to ping master at 127.5.240.254:54399: Client connection negotiation failed: client connection to 127.5.240.254:54399: connect: Connection refused (error 111) I0703 01:39:27.042711 7441 raft_consensus.cc:2684] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad [term 3 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, VOTER 29c07deff3f54b10b7ad9a60bc14edeb (127.5.240.194) evicted, VOTER d7071ff1216d42808d7cfb3ba8e4ee0b (127.5.240.193) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a5fbe0bff97843f89529ce5722817fad" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 60719 } } unsafe_config_change: true } W0703 01:39:27.045028 7379 heartbeater.cc:587] Failed to heartbeat to 127.5.240.254:54399: Network error: Failed to ping master at 127.5.240.254:54399: Client connection negotiation failed: client connection to 127.5.240.254:54399: connect: Connection refused (error 111) I0703 01:39:27.138739 6083 kudu-admin-test.cc:430] Restarting master... I0703 01:39:27.139135 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:54399 --webserver_interface=127.5.240.254 --webserver_port=45037 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --rpc_reuseport=true with env {} W0703 01:39:27.191996 7257 heartbeater.cc:587] Failed to heartbeat to 127.5.240.254:54399: Network error: Failed to ping master at 127.5.240.254:54399: Client connection negotiation failed: client connection to 127.5.240.254:54399: connect: Connection refused (error 111) WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:39:27.319838 7445 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:39:27.320071 7445 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:39:27.322290 7445 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0703 01:39:27.322336 7445 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:39:27.322394 7445 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0703 01:39:27.322446 7445 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0703 01:39:27.330564 7445 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:54399 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.5.240.254 --webserver_port=45037 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:39:27.336401 7445 master_main.cc:80] Initializing master server... I0703 01:39:27.337497 7445 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 137208us I0703 01:39:27.338625 7445 fs_manager.cc:261] Metadata directory not provided I0703 01:39:27.338692 7445 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:39:27.346078 7445 fs_manager.cc:397] Time spent opening directory manager: real 0.005s user 0.003s sys 0.002s I0703 01:39:27.346920 7445 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:39:27.347347 7445 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:39:27.351007 7445 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0703 01:39:27.351081 7445 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal uuid: "95bbdac381bb4ab690642dd43f1992e6" format_stamp: "Formatted at 2018-07-03 08:39:22 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:27.351959 7445 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:39:27.411818 7445 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:39:27.413246 7445 master_main.cc:83] Starting Master server... I0703 01:39:27.450932 7445 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.254:54399 I0703 01:39:27.451385 7445 webserver.cc:175] Starting webserver on 127.5.240.254:45037 I0703 01:39:27.451431 7445 webserver.cc:186] Document root disabled I0703 01:39:27.452157 7445 webserver.cc:313] Webserver started. Bound to: http://127.5.240.254:45037/ I0703 01:39:27.452847 7445 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb I0703 01:39:27.460274 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master as pid 7445 I0703 01:39:27.462059 7505 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 95bbdac381bb4ab690642dd43f1992e6: Bootstrap starting. I0703 01:39:27.475497 7505 log.cc:525] T 00000000000000000000000000000000 P 95bbdac381bb4ab690642dd43f1992e6: Log is configured to *not* fsync() on all Append() calls I0703 01:39:27.484859 7505 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 95bbdac381bb4ab690642dd43f1992e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=1} inserts{seen=4 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:39:27.485266 7505 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 95bbdac381bb4ab690642dd43f1992e6: Bootstrap complete. I0703 01:39:27.494884 7505 raft_consensus.cc:325] T 00000000000000000000000000000000 P 95bbdac381bb4ab690642dd43f1992e6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95bbdac381bb4ab690642dd43f1992e6" member_type: VOTER } I0703 01:39:27.495597 7505 raft_consensus.cc:628] T 00000000000000000000000000000000 P 95bbdac381bb4ab690642dd43f1992e6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95bbdac381bb4ab690642dd43f1992e6, State: Initialized, Role: FOLLOWER I0703 01:39:27.496047 7505 consensus_queue.cc:227] T 00000000000000000000000000000000 P 95bbdac381bb4ab690642dd43f1992e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95bbdac381bb4ab690642dd43f1992e6" member_type: VOTER } I0703 01:39:27.496170 7505 raft_consensus.cc:365] T 00000000000000000000000000000000 P 95bbdac381bb4ab690642dd43f1992e6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:39:27.496233 7505 raft_consensus.cc:457] T 00000000000000000000000000000000 P 95bbdac381bb4ab690642dd43f1992e6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:39:27.496301 7505 raft_consensus.cc:2730] T 00000000000000000000000000000000 P 95bbdac381bb4ab690642dd43f1992e6: Snoozing failure detection for 1.964s (starting election) I0703 01:39:27.496373 7505 raft_consensus.cc:2782] T 00000000000000000000000000000000 P 95bbdac381bb4ab690642dd43f1992e6 [term 1 FOLLOWER]: Advancing to term 2 I0703 01:39:27.500147 7505 raft_consensus.cc:479] T 00000000000000000000000000000000 P 95bbdac381bb4ab690642dd43f1992e6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95bbdac381bb4ab690642dd43f1992e6" member_type: VOTER } I0703 01:39:27.500485 7505 leader_election.cc:270] T 00000000000000000000000000000000 P 95bbdac381bb4ab690642dd43f1992e6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0703 01:39:27.501807 7507 raft_consensus.cc:2730] T 00000000000000000000000000000000 P 95bbdac381bb4ab690642dd43f1992e6: Snoozing failure detection for 2.063s (election complete) I0703 01:39:27.502257 7507 raft_consensus.cc:2537] T 00000000000000000000000000000000 P 95bbdac381bb4ab690642dd43f1992e6 [term 2 FOLLOWER]: Leader election won for term 2 I0703 01:39:27.503603 7507 raft_consensus.cc:593] T 00000000000000000000000000000000 P 95bbdac381bb4ab690642dd43f1992e6 [term 2 LEADER]: Becoming Leader. State: Replica: 95bbdac381bb4ab690642dd43f1992e6, State: Running, Role: LEADER I0703 01:39:27.503985 7507 consensus_queue.cc:204] T 00000000000000000000000000000000 P 95bbdac381bb4ab690642dd43f1992e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95bbdac381bb4ab690642dd43f1992e6" member_type: VOTER } I0703 01:39:27.505493 7509 sys_catalog.cc:337] T 00000000000000000000000000000000 P 95bbdac381bb4ab690642dd43f1992e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95bbdac381bb4ab690642dd43f1992e6. Latest consensus state: current_term: 2 leader_uuid: "95bbdac381bb4ab690642dd43f1992e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95bbdac381bb4ab690642dd43f1992e6" member_type: VOTER } } I0703 01:39:27.506284 7509 sys_catalog.cc:340] T 00000000000000000000000000000000 P 95bbdac381bb4ab690642dd43f1992e6 [sys.catalog]: This master's current role is: LEADER I0703 01:39:27.506901 7505 sys_catalog.cc:337] T 00000000000000000000000000000000 P 95bbdac381bb4ab690642dd43f1992e6 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 2 leader_uuid: "95bbdac381bb4ab690642dd43f1992e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95bbdac381bb4ab690642dd43f1992e6" member_type: VOTER } } I0703 01:39:27.507776 7505 sys_catalog.cc:340] T 00000000000000000000000000000000 P 95bbdac381bb4ab690642dd43f1992e6 [sys.catalog]: This master's current role is: LEADER I0703 01:39:27.506893 7508 sys_catalog.cc:337] T 00000000000000000000000000000000 P 95bbdac381bb4ab690642dd43f1992e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "95bbdac381bb4ab690642dd43f1992e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95bbdac381bb4ab690642dd43f1992e6" member_type: VOTER } } I0703 01:39:27.507931 7508 sys_catalog.cc:340] T 00000000000000000000000000000000 P 95bbdac381bb4ab690642dd43f1992e6 [sys.catalog]: This master's current role is: LEADER I0703 01:39:27.508790 7505 sys_catalog.cc:424] T 00000000000000000000000000000000 P 95bbdac381bb4ab690642dd43f1992e6 [sys.catalog]: configured and running, proceeding with master startup. I0703 01:39:27.508946 7511 catalog_manager.cc:1006] Loading table and tablet metadata into memory... I0703 01:39:27.510277 7445 master_main.cc:86] Master server successfully started. I0703 01:39:27.514683 7511 catalog_manager.cc:323] Loaded metadata for table TestTable [id=24e11f69554b41ef994802e369af7c41] I0703 01:39:27.516253 7511 catalog_manager.cc:377] Loaded metadata for tablet a61eb64bf7cc417684a8446570d07208 (table TestTable [id=24e11f69554b41ef994802e369af7c41]) I0703 01:39:27.516454 7511 catalog_manager.cc:1017] Initializing Kudu internal certificate authority... I0703 01:39:27.519407 7511 catalog_manager.cc:1026] Loading token signing keys... I0703 01:39:27.520478 7511 catalog_manager.cc:4113] T 00000000000000000000000000000000 P 95bbdac381bb4ab690642dd43f1992e6: Loaded TSK: 0 I0703 01:39:27.857759 7135 heartbeater.cc:346] Connected to a master server at 127.5.240.254:54399 I0703 01:39:27.858685 7472 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "d45639fe421a4c0c81abaf8a04d04fa9" instance_seqno: 1530607163804367) as {username='jenkins'} at 127.5.240.195:34511; Asking this server to re-register. I0703 01:39:27.859236 7135 heartbeater.cc:426] Registering TS with master... I0703 01:39:27.859364 7135 heartbeater.cc:475] Master 127.5.240.254:54399 requested a full tablet report, sending... I0703 01:39:27.860276 7472 ts_manager.cc:98] Registered new tserver with Master: d45639fe421a4c0c81abaf8a04d04fa9 (127.5.240.195:33176) I0703 01:39:28.050185 7379 heartbeater.cc:346] Connected to a master server at 127.5.240.254:54399 I0703 01:39:28.051300 7472 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "a5fbe0bff97843f89529ce5722817fad" instance_seqno: 1530607164467064) as {username='jenkins'} at 127.5.240.197:49117; Asking this server to re-register. I0703 01:39:28.051869 7379 heartbeater.cc:426] Registering TS with master... I0703 01:39:28.051976 7379 heartbeater.cc:475] Master 127.5.240.254:54399 requested a full tablet report, sending... I0703 01:39:28.052717 7472 ts_manager.cc:98] Registered new tserver with Master: a5fbe0bff97843f89529ce5722817fad (127.5.240.197:60719) I0703 01:39:28.053169 7472 catalog_manager.cc:3798] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad reported cstate change: config changed from index -1 to 2, term changed from 1 to 3, leader changed from 29c07deff3f54b10b7ad9a60bc14edeb (127.5.240.194) to a5fbe0bff97843f89529ce5722817fad (127.5.240.197), VOTER 29c07deff3f54b10b7ad9a60bc14edeb (127.5.240.194) evicted, VOTER d7071ff1216d42808d7cfb3ba8e4ee0b (127.5.240.193) evicted. New cstate: current_term: 3 leader_uuid: "a5fbe0bff97843f89529ce5722817fad" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a5fbe0bff97843f89529ce5722817fad" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 60719 } health_report { overall_health: HEALTHY } } unsafe_config_change: true } I0703 01:39:28.060657 7472 catalog_manager.cc:2903] Scheduling retry of a61eb64bf7cc417684a8446570d07208 Delete Tablet RPC for TS=29c07deff3f54b10b7ad9a60bc14edeb with a delay of 26 ms (attempt = 1) W0703 01:39:28.060783 7472 catalog_manager.cc:3957] Failed to send a61eb64bf7cc417684a8446570d07208 Delete Tablet RPC for TS=29c07deff3f54b10b7ad9a60bc14edeb: Not found: failed to reset TS proxy: Could not find TS for UUID 29c07deff3f54b10b7ad9a60bc14edeb I0703 01:39:28.060904 7472 catalog_manager.cc:2903] Scheduling retry of a61eb64bf7cc417684a8446570d07208 Delete Tablet RPC for TS=d7071ff1216d42808d7cfb3ba8e4ee0b with a delay of 21 ms (attempt = 1) W0703 01:39:28.060989 7472 catalog_manager.cc:3957] Failed to send a61eb64bf7cc417684a8446570d07208 Delete Tablet RPC for TS=d7071ff1216d42808d7cfb3ba8e4ee0b: Not found: failed to reset TS proxy: Could not find TS for UUID d7071ff1216d42808d7cfb3ba8e4ee0b I0703 01:39:28.061352 7472 catalog_manager.cc:3437] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet a61eb64bf7cc417684a8446570d07208 (attempt 1) I0703 01:39:28.067270 7336 consensus_queue.cc:204] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 0, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a5fbe0bff97843f89529ce5722817fad" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 60719 } } peers { permanent_uuid: "d45639fe421a4c0c81abaf8a04d04fa9" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 33176 } attrs { promote: true } } unsafe_config_change: true I0703 01:39:28.069409 7525 raft_consensus.cc:2684] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index 2 to 4, NON_VOTER d45639fe421a4c0c81abaf8a04d04fa9 (127.5.240.195) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a5fbe0bff97843f89529ce5722817fad" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 60719 } } peers { permanent_uuid: "d45639fe421a4c0c81abaf8a04d04fa9" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 33176 } attrs { promote: true } } unsafe_config_change: true } I0703 01:39:28.072342 7455 catalog_manager.cc:3352] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a61eb64bf7cc417684a8446570d07208 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0703 01:39:28.073161 7472 catalog_manager.cc:3798] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad reported cstate change: config changed from index 2 to 4, NON_VOTER d45639fe421a4c0c81abaf8a04d04fa9 (127.5.240.195) added. New cstate: current_term: 3 leader_uuid: "a5fbe0bff97843f89529ce5722817fad" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a5fbe0bff97843f89529ce5722817fad" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 60719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d45639fe421a4c0c81abaf8a04d04fa9" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 33176 } attrs { promote: true } health_report { overall_health: UNKNOWN } } unsafe_config_change: true } I0703 01:39:28.075434 7472 catalog_manager.cc:3437] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet a61eb64bf7cc417684a8446570d07208 (attempt 1) W0703 01:39:28.075639 7472 catalog_manager.cc:3486] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a61eb64bf7cc417684a8446570d07208 with cas_config_opid_index 4: no candidate replacement replica found for tablet a61eb64bf7cc417684a8446570d07208 (table TestTable [id=24e11f69554b41ef994802e369af7c41]); the total number of registered tablet servers (2) does not allow for replacement of the failed replica: at least 4 tablet servers are required I0703 01:39:28.075700 7472 catalog_manager.cc:2903] Scheduling retry of ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a61eb64bf7cc417684a8446570d07208 with cas_config_opid_index 4 with a delay of 34 ms (attempt = 1) W0703 01:39:28.077025 7267 consensus_peers.cc:433] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad -> Peer d45639fe421a4c0c81abaf8a04d04fa9 (127.5.240.195:33176): Couldn't send request to peer d45639fe421a4c0c81abaf8a04d04fa9 for tablet a61eb64bf7cc417684a8446570d07208. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a61eb64bf7cc417684a8446570d07208. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:39:28.082792 7455 catalog_manager.cc:2903] Scheduling retry of a61eb64bf7cc417684a8446570d07208 Delete Tablet RPC for TS=d7071ff1216d42808d7cfb3ba8e4ee0b with a delay of 74 ms (attempt = 2) W0703 01:39:28.082854 7455 catalog_manager.cc:2922] Async tablet task a61eb64bf7cc417684a8446570d07208 Delete Tablet RPC for TS=d7071ff1216d42808d7cfb3ba8e4ee0b failed: Not found: failed to reset TS proxy: Could not find TS for UUID d7071ff1216d42808d7cfb3ba8e4ee0b I0703 01:39:28.087124 7453 catalog_manager.cc:2903] Scheduling retry of a61eb64bf7cc417684a8446570d07208 Delete Tablet RPC for TS=29c07deff3f54b10b7ad9a60bc14edeb with a delay of 64 ms (attempt = 2) W0703 01:39:28.087184 7453 catalog_manager.cc:2922] Async tablet task a61eb64bf7cc417684a8446570d07208 Delete Tablet RPC for TS=29c07deff3f54b10b7ad9a60bc14edeb failed: Not found: failed to reset TS proxy: Could not find TS for UUID 29c07deff3f54b10b7ad9a60bc14edeb I0703 01:39:28.110461 7456 catalog_manager.cc:3437] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet a61eb64bf7cc417684a8446570d07208 (attempt 2) I0703 01:39:28.110630 7456 catalog_manager.cc:2903] Scheduling retry of ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a61eb64bf7cc417684a8446570d07208 with cas_config_opid_index 4 with a delay of 78 ms (attempt = 2) I0703 01:39:28.151582 7453 catalog_manager.cc:2903] Scheduling retry of a61eb64bf7cc417684a8446570d07208 Delete Tablet RPC for TS=29c07deff3f54b10b7ad9a60bc14edeb with a delay of 93 ms (attempt = 3) W0703 01:39:28.151662 7453 catalog_manager.cc:2922] Async tablet task a61eb64bf7cc417684a8446570d07208 Delete Tablet RPC for TS=29c07deff3f54b10b7ad9a60bc14edeb failed: Not found: failed to reset TS proxy: Could not find TS for UUID 29c07deff3f54b10b7ad9a60bc14edeb I0703 01:39:28.157248 7455 catalog_manager.cc:2903] Scheduling retry of a61eb64bf7cc417684a8446570d07208 Delete Tablet RPC for TS=d7071ff1216d42808d7cfb3ba8e4ee0b with a delay of 85 ms (attempt = 3) W0703 01:39:28.157301 7455 catalog_manager.cc:2922] Async tablet task a61eb64bf7cc417684a8446570d07208 Delete Tablet RPC for TS=d7071ff1216d42808d7cfb3ba8e4ee0b failed: Not found: failed to reset TS proxy: Could not find TS for UUID d7071ff1216d42808d7cfb3ba8e4ee0b I0703 01:39:28.189303 7456 catalog_manager.cc:3437] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet a61eb64bf7cc417684a8446570d07208 (attempt 3) I0703 01:39:28.189427 7456 catalog_manager.cc:2903] Scheduling retry of ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a61eb64bf7cc417684a8446570d07208 with cas_config_opid_index 4 with a delay of 84 ms (attempt = 3) I0703 01:39:28.197126 7257 heartbeater.cc:346] Connected to a master server at 127.5.240.254:54399 I0703 01:39:28.197233 7257 heartbeater.cc:467] Master 127.5.240.254:54399 was elected leader, sending a full tablet report... I0703 01:39:28.197957 7472 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "50f7f59cd4bf4ccdbb7fef595e615f5c" instance_seqno: 1530607164149160) as {username='jenkins'} at 127.5.240.196:33426; Asking this server to re-register. I0703 01:39:28.198501 7257 heartbeater.cc:426] Registering TS with master... I0703 01:39:28.198631 7257 heartbeater.cc:475] Master 127.5.240.254:54399 requested a full tablet report, sending... I0703 01:39:28.199298 7472 ts_manager.cc:98] Registered new tserver with Master: 50f7f59cd4bf4ccdbb7fef595e615f5c (127.5.240.196:59711) I0703 01:39:28.242986 7455 catalog_manager.cc:2903] Scheduling retry of a61eb64bf7cc417684a8446570d07208 Delete Tablet RPC for TS=d7071ff1216d42808d7cfb3ba8e4ee0b with a delay of 162 ms (attempt = 4) W0703 01:39:28.243046 7455 catalog_manager.cc:2922] Async tablet task a61eb64bf7cc417684a8446570d07208 Delete Tablet RPC for TS=d7071ff1216d42808d7cfb3ba8e4ee0b failed: Not found: failed to reset TS proxy: Could not find TS for UUID d7071ff1216d42808d7cfb3ba8e4ee0b I0703 01:39:28.245349 7453 catalog_manager.cc:2903] Scheduling retry of a61eb64bf7cc417684a8446570d07208 Delete Tablet RPC for TS=29c07deff3f54b10b7ad9a60bc14edeb with a delay of 150 ms (attempt = 4) W0703 01:39:28.245396 7453 catalog_manager.cc:2922] Async tablet task a61eb64bf7cc417684a8446570d07208 Delete Tablet RPC for TS=29c07deff3f54b10b7ad9a60bc14edeb failed: Not found: failed to reset TS proxy: Could not find TS for UUID 29c07deff3f54b10b7ad9a60bc14edeb I0703 01:39:28.274008 7456 catalog_manager.cc:3437] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet a61eb64bf7cc417684a8446570d07208 (attempt 4) I0703 01:39:28.274855 7336 consensus_peers.cc:455] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad -> Peer d45639fe421a4c0c81abaf8a04d04fa9 (127.5.240.195:33176): Closing peer: d45639fe421a4c0c81abaf8a04d04fa9 I0703 01:39:28.275118 7336 consensus_queue.cc:204] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 0, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5fbe0bff97843f89529ce5722817fad" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 60719 } } peers { permanent_uuid: "d45639fe421a4c0c81abaf8a04d04fa9" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 33176 } attrs { promote: true } } peers { permanent_uuid: "50f7f59cd4bf4ccdbb7fef595e615f5c" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 59711 } attrs { promote: true } } unsafe_config_change: true W0703 01:39:28.278313 7267 consensus_peers.cc:433] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad -> Peer d45639fe421a4c0c81abaf8a04d04fa9 (127.5.240.195:33176): Couldn't send request to peer d45639fe421a4c0c81abaf8a04d04fa9 for tablet a61eb64bf7cc417684a8446570d07208. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a61eb64bf7cc417684a8446570d07208. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:39:28.277485 7525 raft_consensus.cc:2684] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, NON_VOTER 50f7f59cd4bf4ccdbb7fef595e615f5c (127.5.240.196) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5fbe0bff97843f89529ce5722817fad" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 60719 } } peers { permanent_uuid: "d45639fe421a4c0c81abaf8a04d04fa9" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 33176 } attrs { promote: true } } peers { permanent_uuid: "50f7f59cd4bf4ccdbb7fef595e615f5c" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 59711 } attrs { promote: true } } unsafe_config_change: true } I0703 01:39:28.280478 7455 catalog_manager.cc:3352] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a61eb64bf7cc417684a8446570d07208 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 4) I0703 01:39:28.281435 7472 catalog_manager.cc:3798] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad reported cstate change: config changed from index 4 to 5, NON_VOTER 50f7f59cd4bf4ccdbb7fef595e615f5c (127.5.240.196) added. New cstate: current_term: 3 leader_uuid: "a5fbe0bff97843f89529ce5722817fad" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5fbe0bff97843f89529ce5722817fad" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 60719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d45639fe421a4c0c81abaf8a04d04fa9" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 33176 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50f7f59cd4bf4ccdbb7fef595e615f5c" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 59711 } attrs { promote: true } health_report { overall_health: UNKNOWN } } unsafe_config_change: true } W0703 01:39:28.284034 7270 consensus_peers.cc:433] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad -> Peer 50f7f59cd4bf4ccdbb7fef595e615f5c (127.5.240.196:59711): Couldn't send request to peer 50f7f59cd4bf4ccdbb7fef595e615f5c for tablet a61eb64bf7cc417684a8446570d07208. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a61eb64bf7cc417684a8446570d07208. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:39:28.396353 7453 catalog_manager.cc:2903] Scheduling retry of a61eb64bf7cc417684a8446570d07208 Delete Tablet RPC for TS=29c07deff3f54b10b7ad9a60bc14edeb with a delay of 283 ms (attempt = 5) W0703 01:39:28.396422 7453 catalog_manager.cc:2922] Async tablet task a61eb64bf7cc417684a8446570d07208 Delete Tablet RPC for TS=29c07deff3f54b10b7ad9a60bc14edeb failed: Not found: failed to reset TS proxy: Could not find TS for UUID 29c07deff3f54b10b7ad9a60bc14edeb I0703 01:39:28.406204 7455 catalog_manager.cc:2903] Scheduling retry of a61eb64bf7cc417684a8446570d07208 Delete Tablet RPC for TS=d7071ff1216d42808d7cfb3ba8e4ee0b with a delay of 290 ms (attempt = 5) W0703 01:39:28.406257 7455 catalog_manager.cc:2922] Async tablet task a61eb64bf7cc417684a8446570d07208 Delete Tablet RPC for TS=d7071ff1216d42808d7cfb3ba8e4ee0b failed: Not found: failed to reset TS proxy: Could not find TS for UUID d7071ff1216d42808d7cfb3ba8e4ee0b I0703 01:39:28.680670 7453 catalog_manager.cc:2903] Scheduling retry of a61eb64bf7cc417684a8446570d07208 Delete Tablet RPC for TS=29c07deff3f54b10b7ad9a60bc14edeb with a delay of 552 ms (attempt = 6) W0703 01:39:28.680740 7453 catalog_manager.cc:2922] Async tablet task a61eb64bf7cc417684a8446570d07208 Delete Tablet RPC for TS=29c07deff3f54b10b7ad9a60bc14edeb failed: Not found: failed to reset TS proxy: Could not find TS for UUID 29c07deff3f54b10b7ad9a60bc14edeb I0703 01:39:28.683854 7531 ts_tablet_manager.cc:669] T a61eb64bf7cc417684a8446570d07208 P 50f7f59cd4bf4ccdbb7fef595e615f5c: Initiating tablet copy from peer a5fbe0bff97843f89529ce5722817fad (127.5.240.197:60719) I0703 01:39:28.684105 7531 tablet_copy_client.cc:239] T a61eb64bf7cc417684a8446570d07208 P 50f7f59cd4bf4ccdbb7fef595e615f5c: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.197:60719 I0703 01:39:28.688690 7356 tablet_copy_service.cc:135] P a5fbe0bff97843f89529ce5722817fad: Received BeginTabletCopySession request for tablet a61eb64bf7cc417684a8446570d07208 from peer 50f7f59cd4bf4ccdbb7fef595e615f5c ({username='jenkins'} at 127.5.240.196:51256) I0703 01:39:28.688827 7356 tablet_copy_service.cc:156] P a5fbe0bff97843f89529ce5722817fad: Beginning new tablet copy session on tablet a61eb64bf7cc417684a8446570d07208 from peer 50f7f59cd4bf4ccdbb7fef595e615f5c at {username='jenkins'} at 127.5.240.196:51256: session id = 50f7f59cd4bf4ccdbb7fef595e615f5c-a61eb64bf7cc417684a8446570d07208 I0703 01:39:28.691239 7356 tablet_copy_source_session.cc:202] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:39:28.694036 7531 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet a61eb64bf7cc417684a8446570d07208. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:39:28.697597 7455 catalog_manager.cc:2903] Scheduling retry of a61eb64bf7cc417684a8446570d07208 Delete Tablet RPC for TS=d7071ff1216d42808d7cfb3ba8e4ee0b with a delay of 545 ms (attempt = 6) W0703 01:39:28.697679 7455 catalog_manager.cc:2922] Async tablet task a61eb64bf7cc417684a8446570d07208 Delete Tablet RPC for TS=d7071ff1216d42808d7cfb3ba8e4ee0b failed: Not found: failed to reset TS proxy: Could not find TS for UUID d7071ff1216d42808d7cfb3ba8e4ee0b I0703 01:39:28.704208 7531 ts_tablet_manager.cc:1173] T a61eb64bf7cc417684a8446570d07208 P 50f7f59cd4bf4ccdbb7fef595e615f5c: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:39:28.704469 7531 tablet_copy_client.cc:566] T a61eb64bf7cc417684a8446570d07208 P 50f7f59cd4bf4ccdbb7fef595e615f5c: tablet copy: Starting download of 0 data blocks... I0703 01:39:28.704627 7531 tablet_copy_client.cc:529] T a61eb64bf7cc417684a8446570d07208 P 50f7f59cd4bf4ccdbb7fef595e615f5c: tablet copy: Starting download of 1 WAL segments... I0703 01:39:28.706279 7531 tablet_copy_client.cc:405] T a61eb64bf7cc417684a8446570d07208 P 50f7f59cd4bf4ccdbb7fef595e615f5c: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:39:28.707985 7531 ts_tablet_manager.cc:1028] T a61eb64bf7cc417684a8446570d07208 P 50f7f59cd4bf4ccdbb7fef595e615f5c: Bootstrapping tablet I0703 01:39:28.708202 7531 tablet_bootstrap.cc:427] T a61eb64bf7cc417684a8446570d07208 P 50f7f59cd4bf4ccdbb7fef595e615f5c: Bootstrap starting. I0703 01:39:28.712996 7531 log.cc:525] T a61eb64bf7cc417684a8446570d07208 P 50f7f59cd4bf4ccdbb7fef595e615f5c: Log is configured to *not* fsync() on all Append() calls I0703 01:39:28.717267 7531 tablet_bootstrap.cc:427] T a61eb64bf7cc417684a8446570d07208 P 50f7f59cd4bf4ccdbb7fef595e615f5c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:39:28.717595 7531 tablet_bootstrap.cc:427] T a61eb64bf7cc417684a8446570d07208 P 50f7f59cd4bf4ccdbb7fef595e615f5c: Bootstrap complete. I0703 01:39:28.717830 7531 ts_tablet_manager.cc:1045] T a61eb64bf7cc417684a8446570d07208 P 50f7f59cd4bf4ccdbb7fef595e615f5c: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.004s I0703 01:39:28.722533 7531 raft_consensus.cc:325] T a61eb64bf7cc417684a8446570d07208 P 50f7f59cd4bf4ccdbb7fef595e615f5c [term 3 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5fbe0bff97843f89529ce5722817fad" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 60719 } } peers { permanent_uuid: "d45639fe421a4c0c81abaf8a04d04fa9" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 33176 } attrs { promote: true } } peers { permanent_uuid: "50f7f59cd4bf4ccdbb7fef595e615f5c" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 59711 } attrs { promote: true } } unsafe_config_change: true I0703 01:39:28.722836 7531 raft_consensus.cc:628] T a61eb64bf7cc417684a8446570d07208 P 50f7f59cd4bf4ccdbb7fef595e615f5c [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 50f7f59cd4bf4ccdbb7fef595e615f5c, State: Initialized, Role: LEARNER I0703 01:39:28.723083 7531 consensus_queue.cc:227] T a61eb64bf7cc417684a8446570d07208 P 50f7f59cd4bf4ccdbb7fef595e615f5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5fbe0bff97843f89529ce5722817fad" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 60719 } } peers { permanent_uuid: "d45639fe421a4c0c81abaf8a04d04fa9" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 33176 } attrs { promote: true } } peers { permanent_uuid: "50f7f59cd4bf4ccdbb7fef595e615f5c" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 59711 } attrs { promote: true } } unsafe_config_change: true I0703 01:39:28.724970 7531 ts_tablet_manager.cc:1073] T a61eb64bf7cc417684a8446570d07208 P 50f7f59cd4bf4ccdbb7fef595e615f5c: Time spent starting tablet: real 0.007s user 0.004s sys 0.003s I0703 01:39:28.726022 7356 tablet_copy_service.cc:325] P a5fbe0bff97843f89529ce5722817fad: Request end of tablet copy session 50f7f59cd4bf4ccdbb7fef595e615f5c-a61eb64bf7cc417684a8446570d07208 received from {username='jenkins'} at 127.5.240.196:51256 I0703 01:39:28.726095 7356 tablet_copy_service.cc:413] P a5fbe0bff97843f89529ce5722817fad: ending tablet copy session 50f7f59cd4bf4ccdbb7fef595e615f5c-a61eb64bf7cc417684a8446570d07208 on tablet a61eb64bf7cc417684a8446570d07208 with peer 50f7f59cd4bf4ccdbb7fef595e615f5c I0703 01:39:28.843660 7535 ts_tablet_manager.cc:669] T a61eb64bf7cc417684a8446570d07208 P d45639fe421a4c0c81abaf8a04d04fa9: Initiating tablet copy from peer a5fbe0bff97843f89529ce5722817fad (127.5.240.197:60719) I0703 01:39:28.843937 7535 tablet_copy_client.cc:239] T a61eb64bf7cc417684a8446570d07208 P d45639fe421a4c0c81abaf8a04d04fa9: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.197:60719 I0703 01:39:28.848161 7356 tablet_copy_service.cc:135] P a5fbe0bff97843f89529ce5722817fad: Received BeginTabletCopySession request for tablet a61eb64bf7cc417684a8446570d07208 from peer d45639fe421a4c0c81abaf8a04d04fa9 ({username='jenkins'} at 127.5.240.195:53995) I0703 01:39:28.848268 7356 tablet_copy_service.cc:156] P a5fbe0bff97843f89529ce5722817fad: Beginning new tablet copy session on tablet a61eb64bf7cc417684a8446570d07208 from peer d45639fe421a4c0c81abaf8a04d04fa9 at {username='jenkins'} at 127.5.240.195:53995: session id = d45639fe421a4c0c81abaf8a04d04fa9-a61eb64bf7cc417684a8446570d07208 I0703 01:39:28.849946 7356 tablet_copy_source_session.cc:202] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:39:28.852422 7535 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet a61eb64bf7cc417684a8446570d07208. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:39:28.861810 7535 ts_tablet_manager.cc:1173] T a61eb64bf7cc417684a8446570d07208 P d45639fe421a4c0c81abaf8a04d04fa9: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:39:28.862078 7535 tablet_copy_client.cc:566] T a61eb64bf7cc417684a8446570d07208 P d45639fe421a4c0c81abaf8a04d04fa9: tablet copy: Starting download of 0 data blocks... I0703 01:39:28.862287 7535 tablet_copy_client.cc:529] T a61eb64bf7cc417684a8446570d07208 P d45639fe421a4c0c81abaf8a04d04fa9: tablet copy: Starting download of 1 WAL segments... I0703 01:39:28.864253 7535 tablet_copy_client.cc:405] T a61eb64bf7cc417684a8446570d07208 P d45639fe421a4c0c81abaf8a04d04fa9: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:39:28.865905 7535 ts_tablet_manager.cc:1028] T a61eb64bf7cc417684a8446570d07208 P d45639fe421a4c0c81abaf8a04d04fa9: Bootstrapping tablet I0703 01:39:28.866122 7535 tablet_bootstrap.cc:427] T a61eb64bf7cc417684a8446570d07208 P d45639fe421a4c0c81abaf8a04d04fa9: Bootstrap starting. I0703 01:39:28.870048 7535 log.cc:525] T a61eb64bf7cc417684a8446570d07208 P d45639fe421a4c0c81abaf8a04d04fa9: Log is configured to *not* fsync() on all Append() calls I0703 01:39:28.873836 7535 tablet_bootstrap.cc:427] T a61eb64bf7cc417684a8446570d07208 P d45639fe421a4c0c81abaf8a04d04fa9: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:39:28.874158 7535 tablet_bootstrap.cc:427] T a61eb64bf7cc417684a8446570d07208 P d45639fe421a4c0c81abaf8a04d04fa9: Bootstrap complete. I0703 01:39:28.874378 7535 ts_tablet_manager.cc:1045] T a61eb64bf7cc417684a8446570d07208 P d45639fe421a4c0c81abaf8a04d04fa9: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s I0703 01:39:28.879411 7535 raft_consensus.cc:325] T a61eb64bf7cc417684a8446570d07208 P d45639fe421a4c0c81abaf8a04d04fa9 [term 3 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5fbe0bff97843f89529ce5722817fad" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 60719 } } peers { permanent_uuid: "d45639fe421a4c0c81abaf8a04d04fa9" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 33176 } attrs { promote: true } } peers { permanent_uuid: "50f7f59cd4bf4ccdbb7fef595e615f5c" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 59711 } attrs { promote: true } } unsafe_config_change: true I0703 01:39:28.879751 7535 raft_consensus.cc:628] T a61eb64bf7cc417684a8446570d07208 P d45639fe421a4c0c81abaf8a04d04fa9 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: d45639fe421a4c0c81abaf8a04d04fa9, State: Initialized, Role: LEARNER I0703 01:39:28.880004 7535 consensus_queue.cc:227] T a61eb64bf7cc417684a8446570d07208 P d45639fe421a4c0c81abaf8a04d04fa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a5fbe0bff97843f89529ce5722817fad" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 60719 } } peers { permanent_uuid: "d45639fe421a4c0c81abaf8a04d04fa9" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 33176 } attrs { promote: true } } peers { permanent_uuid: "50f7f59cd4bf4ccdbb7fef595e615f5c" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 59711 } attrs { promote: true } } unsafe_config_change: true I0703 01:39:28.881916 7535 ts_tablet_manager.cc:1073] T a61eb64bf7cc417684a8446570d07208 P d45639fe421a4c0c81abaf8a04d04fa9: Time spent starting tablet: real 0.007s user 0.005s sys 0.003s I0703 01:39:28.882580 7356 tablet_copy_service.cc:325] P a5fbe0bff97843f89529ce5722817fad: Request end of tablet copy session d45639fe421a4c0c81abaf8a04d04fa9-a61eb64bf7cc417684a8446570d07208 received from {username='jenkins'} at 127.5.240.195:53995 I0703 01:39:28.882664 7356 tablet_copy_service.cc:413] P a5fbe0bff97843f89529ce5722817fad: ending tablet copy session d45639fe421a4c0c81abaf8a04d04fa9-a61eb64bf7cc417684a8446570d07208 on tablet a61eb64bf7cc417684a8446570d07208 with peer d45639fe421a4c0c81abaf8a04d04fa9 W0703 01:39:28.923945 7258 tablet.cc:1678] T a61eb64bf7cc417684a8446570d07208 P 50f7f59cd4bf4ccdbb7fef595e615f5c: Can't schedule compaction. Clean time has not been advanced past its initial value. W0703 01:39:29.078598 7136 tablet.cc:1678] T a61eb64bf7cc417684a8446570d07208 P d45639fe421a4c0c81abaf8a04d04fa9: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:39:29.218006 7214 raft_consensus.cc:1026] T a61eb64bf7cc417684a8446570d07208 P 50f7f59cd4bf4ccdbb7fef595e615f5c [term 3 LEARNER]: Deduplicated request from leader. Original: 3.4->[3.5-3.5] Dedup: 3.5->[] I0703 01:39:29.233639 7453 catalog_manager.cc:2903] Scheduling retry of a61eb64bf7cc417684a8446570d07208 Delete Tablet RPC for TS=29c07deff3f54b10b7ad9a60bc14edeb with a delay of 1046 ms (attempt = 7) W0703 01:39:29.233706 7453 catalog_manager.cc:2922] Async tablet task a61eb64bf7cc417684a8446570d07208 Delete Tablet RPC for TS=29c07deff3f54b10b7ad9a60bc14edeb failed: Not found: failed to reset TS proxy: Could not find TS for UUID 29c07deff3f54b10b7ad9a60bc14edeb I0703 01:39:29.243530 7455 catalog_manager.cc:2903] Scheduling retry of a61eb64bf7cc417684a8446570d07208 Delete Tablet RPC for TS=d7071ff1216d42808d7cfb3ba8e4ee0b with a delay of 1072 ms (attempt = 7) W0703 01:39:29.243590 7455 catalog_manager.cc:2922] Async tablet task a61eb64bf7cc417684a8446570d07208 Delete Tablet RPC for TS=d7071ff1216d42808d7cfb3ba8e4ee0b failed: Not found: failed to reset TS proxy: Could not find TS for UUID d7071ff1216d42808d7cfb3ba8e4ee0b I0703 01:39:29.260854 7092 raft_consensus.cc:1026] T a61eb64bf7cc417684a8446570d07208 P d45639fe421a4c0c81abaf8a04d04fa9 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.4->[3.5-3.5] Dedup: 3.5->[] I0703 01:39:29.607287 7540 raft_consensus.cc:890] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad: attempting to promote NON_VOTER 50f7f59cd4bf4ccdbb7fef595e615f5c to VOTER I0703 01:39:29.607600 7540 consensus_peers.cc:455] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad -> Peer 50f7f59cd4bf4ccdbb7fef595e615f5c (127.5.240.196:59711): Closing peer: 50f7f59cd4bf4ccdbb7fef595e615f5c I0703 01:39:29.607666 7540 consensus_peers.cc:455] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad -> Peer d45639fe421a4c0c81abaf8a04d04fa9 (127.5.240.195:33176): Closing peer: d45639fe421a4c0c81abaf8a04d04fa9 I0703 01:39:29.607866 7540 consensus_queue.cc:204] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 0, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a5fbe0bff97843f89529ce5722817fad" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 60719 } } peers { permanent_uuid: "d45639fe421a4c0c81abaf8a04d04fa9" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 33176 } attrs { promote: true } } peers { permanent_uuid: "50f7f59cd4bf4ccdbb7fef595e615f5c" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 59711 } attrs { promote: false } } unsafe_config_change: true I0703 01:39:29.609915 7214 raft_consensus.cc:1081] T a61eb64bf7cc417684a8446570d07208 P 50f7f59cd4bf4ccdbb7fef595e615f5c [term 3 LEARNER]: Refusing update from remote peer a5fbe0bff97843f89529ce5722817fad: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch) I0703 01:39:29.610072 7092 raft_consensus.cc:1081] T a61eb64bf7cc417684a8446570d07208 P d45639fe421a4c0c81abaf8a04d04fa9 [term 3 LEARNER]: Refusing update from remote peer a5fbe0bff97843f89529ce5722817fad: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch) I0703 01:39:29.610530 7539 consensus_queue.cc:962] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad [LEADER]: Connected to new peer: Peer: permanent_uuid: "50f7f59cd4bf4ccdbb7fef595e615f5c" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 59711 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I0703 01:39:29.610855 7530 consensus_queue.cc:962] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad [LEADER]: Connected to new peer: Peer: permanent_uuid: "d45639fe421a4c0c81abaf8a04d04fa9" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 33176 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I0703 01:39:29.613909 7530 raft_consensus.cc:853] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad [term 3 LEADER]: attempt to promote peer d45639fe421a4c0c81abaf8a04d04fa9: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing. I0703 01:39:29.614070 7530 raft_consensus.cc:2684] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad [term 3 LEADER]: Committing config change with OpId 3.6: config changed from index 5 to 6, 50f7f59cd4bf4ccdbb7fef595e615f5c (127.5.240.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a5fbe0bff97843f89529ce5722817fad" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 60719 } } peers { permanent_uuid: "d45639fe421a4c0c81abaf8a04d04fa9" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 33176 } attrs { promote: true } } peers { permanent_uuid: "50f7f59cd4bf4ccdbb7fef595e615f5c" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 59711 } attrs { promote: false } } unsafe_config_change: true } I0703 01:39:29.614883 7214 raft_consensus.cc:2684] T a61eb64bf7cc417684a8446570d07208 P 50f7f59cd4bf4ccdbb7fef595e615f5c [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, 50f7f59cd4bf4ccdbb7fef595e615f5c (127.5.240.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a5fbe0bff97843f89529ce5722817fad" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 60719 } } peers { permanent_uuid: "d45639fe421a4c0c81abaf8a04d04fa9" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 33176 } attrs { promote: true } } peers { permanent_uuid: "50f7f59cd4bf4ccdbb7fef595e615f5c" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 59711 } attrs { promote: false } } unsafe_config_change: true } I0703 01:39:29.618176 7092 raft_consensus.cc:2684] T a61eb64bf7cc417684a8446570d07208 P d45639fe421a4c0c81abaf8a04d04fa9 [term 3 LEARNER]: Committing config change with OpId 3.6: config changed from index 5 to 6, 50f7f59cd4bf4ccdbb7fef595e615f5c (127.5.240.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a5fbe0bff97843f89529ce5722817fad" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 60719 } } peers { permanent_uuid: "d45639fe421a4c0c81abaf8a04d04fa9" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 33176 } attrs { promote: true } } peers { permanent_uuid: "50f7f59cd4bf4ccdbb7fef595e615f5c" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 59711 } attrs { promote: false } } unsafe_config_change: true } I0703 01:39:29.618715 7472 catalog_manager.cc:3798] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad reported cstate change: config changed from index 5 to 6, 50f7f59cd4bf4ccdbb7fef595e615f5c (127.5.240.196) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "a5fbe0bff97843f89529ce5722817fad" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a5fbe0bff97843f89529ce5722817fad" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 60719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d45639fe421a4c0c81abaf8a04d04fa9" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 33176 } attrs { promote: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50f7f59cd4bf4ccdbb7fef595e615f5c" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 59711 } attrs { promote: false } health_report { overall_health: HEALTHY } } unsafe_config_change: true } I0703 01:39:29.621367 7539 raft_consensus.cc:890] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad: attempting to promote NON_VOTER d45639fe421a4c0c81abaf8a04d04fa9 to VOTER I0703 01:39:29.621703 7539 consensus_peers.cc:455] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad -> Peer 50f7f59cd4bf4ccdbb7fef595e615f5c (127.5.240.196:59711): Closing peer: 50f7f59cd4bf4ccdbb7fef595e615f5c I0703 01:39:29.621759 7539 consensus_peers.cc:455] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad -> Peer d45639fe421a4c0c81abaf8a04d04fa9 (127.5.240.195:33176): Closing peer: d45639fe421a4c0c81abaf8a04d04fa9 I0703 01:39:29.621995 7539 consensus_queue.cc:204] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 0, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a5fbe0bff97843f89529ce5722817fad" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 60719 } } peers { permanent_uuid: "d45639fe421a4c0c81abaf8a04d04fa9" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 33176 } attrs { promote: false } } peers { permanent_uuid: "50f7f59cd4bf4ccdbb7fef595e615f5c" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 59711 } attrs { promote: false } } unsafe_config_change: true I0703 01:39:29.623960 7214 raft_consensus.cc:1081] T a61eb64bf7cc417684a8446570d07208 P 50f7f59cd4bf4ccdbb7fef595e615f5c [term 3 FOLLOWER]: Refusing update from remote peer a5fbe0bff97843f89529ce5722817fad: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch) I0703 01:39:29.624168 7092 raft_consensus.cc:1081] T a61eb64bf7cc417684a8446570d07208 P d45639fe421a4c0c81abaf8a04d04fa9 [term 3 LEARNER]: Refusing update from remote peer a5fbe0bff97843f89529ce5722817fad: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch) I0703 01:39:29.624775 7530 consensus_queue.cc:962] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad [LEADER]: Connected to new peer: Peer: permanent_uuid: "50f7f59cd4bf4ccdbb7fef595e615f5c" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 59711 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I0703 01:39:29.625085 7540 consensus_queue.cc:962] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad [LEADER]: Connected to new peer: Peer: permanent_uuid: "d45639fe421a4c0c81abaf8a04d04fa9" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 33176 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I0703 01:39:29.626983 7530 raft_consensus.cc:2684] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad [term 3 LEADER]: Committing config change with OpId 3.7: config changed from index 6 to 7, d45639fe421a4c0c81abaf8a04d04fa9 (127.5.240.195) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a5fbe0bff97843f89529ce5722817fad" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 60719 } } peers { permanent_uuid: "d45639fe421a4c0c81abaf8a04d04fa9" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 33176 } attrs { promote: false } } peers { permanent_uuid: "50f7f59cd4bf4ccdbb7fef595e615f5c" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 59711 } attrs { promote: false } } unsafe_config_change: true } I0703 01:39:29.627939 7092 raft_consensus.cc:2684] T a61eb64bf7cc417684a8446570d07208 P d45639fe421a4c0c81abaf8a04d04fa9 [term 3 FOLLOWER]: Committing config change with OpId 3.7: config changed from index 6 to 7, d45639fe421a4c0c81abaf8a04d04fa9 (127.5.240.195) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a5fbe0bff97843f89529ce5722817fad" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 60719 } } peers { permanent_uuid: "d45639fe421a4c0c81abaf8a04d04fa9" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 33176 } attrs { promote: false } } peers { permanent_uuid: "50f7f59cd4bf4ccdbb7fef595e615f5c" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 59711 } attrs { promote: false } } unsafe_config_change: true } I0703 01:39:29.627907 7214 raft_consensus.cc:2684] T a61eb64bf7cc417684a8446570d07208 P 50f7f59cd4bf4ccdbb7fef595e615f5c [term 3 FOLLOWER]: Committing config change with OpId 3.7: config changed from index 6 to 7, d45639fe421a4c0c81abaf8a04d04fa9 (127.5.240.195) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a5fbe0bff97843f89529ce5722817fad" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 60719 } } peers { permanent_uuid: "d45639fe421a4c0c81abaf8a04d04fa9" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 33176 } attrs { promote: false } } peers { permanent_uuid: "50f7f59cd4bf4ccdbb7fef595e615f5c" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 59711 } attrs { promote: false } } unsafe_config_change: true } I0703 01:39:29.631268 7470 catalog_manager.cc:3798] T a61eb64bf7cc417684a8446570d07208 P d45639fe421a4c0c81abaf8a04d04fa9 reported cstate change: config changed from index 6 to 7, d45639fe421a4c0c81abaf8a04d04fa9 (127.5.240.195) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "a5fbe0bff97843f89529ce5722817fad" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a5fbe0bff97843f89529ce5722817fad" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 60719 } } peers { permanent_uuid: "d45639fe421a4c0c81abaf8a04d04fa9" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 33176 } attrs { promote: false } } peers { permanent_uuid: "50f7f59cd4bf4ccdbb7fef595e615f5c" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 59711 } attrs { promote: false } } unsafe_config_change: true } I0703 01:39:29.679023 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.5.240.194:48424 --local_ip_for_outbound_sockets=127.5.240.194 --webserver_port=45499 --webserver_interface=127.5.240.194 --tserver_master_addrs=127.5.240.254:54399 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:39:29.835458 7553 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:39:29.835719 7553 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:39:29.837795 7553 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:39:29.837908 7553 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.194 I0703 01:39:29.843996 7553 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.194:48424 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.5.240.194 --webserver_port=45499 --tserver_master_addrs=127.5.240.254:54399 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.194 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:39:29.850623 7553 tablet_server_main.cc:85] Initializing tablet server... I0703 01:39:29.851785 7553 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 138208us I0703 01:39:29.852743 7553 fs_manager.cc:261] Metadata directory not provided I0703 01:39:29.852814 7553 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:39:29.859185 7553 fs_manager.cc:397] Time spent opening directory manager: real 0.005s user 0.001s sys 0.003s I0703 01:39:29.860077 7553 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:39:29.860491 7553 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:39:29.864162 7553 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s I0703 01:39:29.864243 7553 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal uuid: "29c07deff3f54b10b7ad9a60bc14edeb" format_stamp: "Formatted at 2018-07-03 08:39:23 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:29.865967 7553 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:39:29.897667 7553 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:39:29.899118 7553 ts_tablet_manager.cc:344] Loading tablet metadata (0/1 complete) I0703 01:39:29.899173 7553 ts_tablet_manager.cc:1005] T a61eb64bf7cc417684a8446570d07208 P 29c07deff3f54b10b7ad9a60bc14edeb: Loading tablet metadata I0703 01:39:29.903594 7553 ts_tablet_manager.cc:356] Loaded tablet metadata (1 live tablets) I0703 01:39:29.905498 7553 ts_tablet_manager.cc:1173] T a61eb64bf7cc417684a8446570d07208 P 29c07deff3f54b10b7ad9a60bc14edeb: Registered tablet (data state: TABLET_DATA_READY) I0703 01:39:29.906762 7568 ts_tablet_manager.cc:1028] T a61eb64bf7cc417684a8446570d07208 P 29c07deff3f54b10b7ad9a60bc14edeb: Bootstrapping tablet I0703 01:39:29.907183 7568 tablet_bootstrap.cc:427] T a61eb64bf7cc417684a8446570d07208 P 29c07deff3f54b10b7ad9a60bc14edeb: Bootstrap starting. I0703 01:39:29.907879 7553 tablet_server_main.cc:90] Starting tablet server... I0703 01:39:29.931249 7568 log.cc:525] T a61eb64bf7cc417684a8446570d07208 P 29c07deff3f54b10b7ad9a60bc14edeb: Log is configured to *not* fsync() on all Append() calls I0703 01:39:29.941148 7568 tablet_bootstrap.cc:427] T a61eb64bf7cc417684a8446570d07208 P 29c07deff3f54b10b7ad9a60bc14edeb: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:39:29.942426 7568 tablet_bootstrap.cc:427] T a61eb64bf7cc417684a8446570d07208 P 29c07deff3f54b10b7ad9a60bc14edeb: Bootstrap complete. I0703 01:39:29.944255 7568 ts_tablet_manager.cc:1045] T a61eb64bf7cc417684a8446570d07208 P 29c07deff3f54b10b7ad9a60bc14edeb: Time spent bootstrapping tablet: real 0.037s user 0.009s sys 0.016s I0703 01:39:29.980015 7568 raft_consensus.cc:325] T a61eb64bf7cc417684a8446570d07208 P 29c07deff3f54b10b7ad9a60bc14edeb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fbe0bff97843f89529ce5722817fad" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 60719 } } peers { permanent_uuid: "29c07deff3f54b10b7ad9a60bc14edeb" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 48424 } } peers { permanent_uuid: "d7071ff1216d42808d7cfb3ba8e4ee0b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42085 } } I0703 01:39:29.989778 7568 raft_consensus.cc:628] T a61eb64bf7cc417684a8446570d07208 P 29c07deff3f54b10b7ad9a60bc14edeb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29c07deff3f54b10b7ad9a60bc14edeb, State: Initialized, Role: FOLLOWER I0703 01:39:29.991188 7568 consensus_queue.cc:227] T a61eb64bf7cc417684a8446570d07208 P 29c07deff3f54b10b7ad9a60bc14edeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fbe0bff97843f89529ce5722817fad" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 60719 } } peers { permanent_uuid: "29c07deff3f54b10b7ad9a60bc14edeb" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 48424 } } peers { permanent_uuid: "d7071ff1216d42808d7cfb3ba8e4ee0b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42085 } } I0703 01:39:29.993922 7568 ts_tablet_manager.cc:1073] T a61eb64bf7cc417684a8446570d07208 P 29c07deff3f54b10b7ad9a60bc14edeb: Time spent starting tablet: real 0.050s user 0.016s sys 0.020s I0703 01:39:30.024593 7553 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.194:48424 I0703 01:39:30.025010 7553 webserver.cc:175] Starting webserver on 127.5.240.194:45499 I0703 01:39:30.025054 7553 webserver.cc:186] Document root disabled I0703 01:39:30.026190 7553 webserver.cc:313] Webserver started. Bound to: http://127.5.240.194:45499/ I0703 01:39:30.026743 7553 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb I0703 01:39:30.028479 7553 tablet_server_main.cc:93] Tablet server successfully started. W0703 01:39:30.028579 7676 tablet.cc:1678] T a61eb64bf7cc417684a8446570d07208 P 29c07deff3f54b10b7ad9a60bc14edeb: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:39:30.030899 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 7553 I0703 01:39:30.031461 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.5.240.193:42085 --local_ip_for_outbound_sockets=127.5.240.193 --webserver_port=58558 --webserver_interface=127.5.240.193 --tserver_master_addrs=127.5.240.254:54399 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0703 01:39:30.039136 7675 heartbeater.cc:346] Connected to a master server at 127.5.240.254:54399 I0703 01:39:30.039275 7675 heartbeater.cc:426] Registering TS with master... I0703 01:39:30.040074 7675 heartbeater.cc:475] Master 127.5.240.254:54399 requested a full tablet report, sending... I0703 01:39:30.041479 7472 ts_manager.cc:98] Registered new tserver with Master: 29c07deff3f54b10b7ad9a60bc14edeb (127.5.240.194:48424) I0703 01:39:30.042152 7472 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet a61eb64bf7cc417684a8446570d07208 on 29c07deff3f54b10b7ad9a60bc14edeb (127.5.240.194:48424) (Replica has no consensus available (current committed config index is 7)) I0703 01:39:30.044142 7472 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.194:52130 I0703 01:39:30.060590 7611 tablet_service.cc:800] Processing DeleteTablet for tablet a61eb64bf7cc417684a8446570d07208 with delete_type TABLET_DATA_TOMBSTONED (Replica has no consensus available (current committed config index is 7)) from {username='jenkins'} at 127.5.240.194:55025 I0703 01:39:30.061357 7681 tablet_replica.cc:261] T a61eb64bf7cc417684a8446570d07208 P 29c07deff3f54b10b7ad9a60bc14edeb: stopping tablet replica I0703 01:39:30.061487 7681 maintenance_manager.cc:237] P 29c07deff3f54b10b7ad9a60bc14edeb: Unregistered op CompactRowSetsOp(a61eb64bf7cc417684a8446570d07208) I0703 01:39:30.061545 7681 maintenance_manager.cc:237] P 29c07deff3f54b10b7ad9a60bc14edeb: Unregistered op MinorDeltaCompactionOp(a61eb64bf7cc417684a8446570d07208) I0703 01:39:30.061594 7681 maintenance_manager.cc:237] P 29c07deff3f54b10b7ad9a60bc14edeb: Unregistered op MajorDeltaCompactionOp(a61eb64bf7cc417684a8446570d07208) I0703 01:39:30.061650 7681 maintenance_manager.cc:237] P 29c07deff3f54b10b7ad9a60bc14edeb: Unregistered op UndoDeltaBlockGCOp(a61eb64bf7cc417684a8446570d07208) I0703 01:39:30.061717 7681 maintenance_manager.cc:237] P 29c07deff3f54b10b7ad9a60bc14edeb: Unregistered op FlushMRSOp(a61eb64bf7cc417684a8446570d07208) I0703 01:39:30.061754 7681 maintenance_manager.cc:237] P 29c07deff3f54b10b7ad9a60bc14edeb: Unregistered op FlushDeltaMemStoresOp(a61eb64bf7cc417684a8446570d07208) I0703 01:39:30.061781 7681 maintenance_manager.cc:237] P 29c07deff3f54b10b7ad9a60bc14edeb: Unregistered op LogGCOp(a61eb64bf7cc417684a8446570d07208) I0703 01:39:30.061847 7681 raft_consensus.cc:2040] T a61eb64bf7cc417684a8446570d07208 P 29c07deff3f54b10b7ad9a60bc14edeb [term 1 FOLLOWER]: Raft consensus shutting down. I0703 01:39:30.062036 7681 raft_consensus.cc:2067] T a61eb64bf7cc417684a8446570d07208 P 29c07deff3f54b10b7ad9a60bc14edeb [term 1 FOLLOWER]: Raft consensus is shut down! I0703 01:39:30.063118 7681 ts_tablet_manager.cc:1369] T a61eb64bf7cc417684a8446570d07208 P 29c07deff3f54b10b7ad9a60bc14edeb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:39:30.067005 7681 ts_tablet_manager.cc:1382] T a61eb64bf7cc417684a8446570d07208 P 29c07deff3f54b10b7ad9a60bc14edeb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1 I0703 01:39:30.067080 7681 log.cc:986] T a61eb64bf7cc417684a8446570d07208 P 29c07deff3f54b10b7ad9a60bc14edeb: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal/wals/a61eb64bf7cc417684a8446570d07208 I0703 01:39:30.067910 7453 catalog_manager.cc:3103] TS 29c07deff3f54b10b7ad9a60bc14edeb (127.5.240.194:48424): tablet a61eb64bf7cc417684a8446570d07208 (table TestTable [id=24e11f69554b41ef994802e369af7c41]) successfully deleted WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:39:30.192384 7678 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:39:30.192646 7678 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:39:30.194725 7678 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:39:30.194833 7678 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.193 I0703 01:39:30.201815 7678 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.193:42085 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.5.240.193 --webserver_port=58558 --tserver_master_addrs=127.5.240.254:54399 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.193 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:39:30.208420 7678 tablet_server_main.cc:85] Initializing tablet server... I0703 01:39:30.209619 7678 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 138708us I0703 01:39:30.210600 7678 fs_manager.cc:261] Metadata directory not provided I0703 01:39:30.210669 7678 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:39:30.216562 7678 fs_manager.cc:397] Time spent opening directory manager: real 0.004s user 0.001s sys 0.002s I0703 01:39:30.217381 7678 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:39:30.217793 7678 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:39:30.221352 7678 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0703 01:39:30.221422 7678 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal uuid: "d7071ff1216d42808d7cfb3ba8e4ee0b" format_stamp: "Formatted at 2018-07-03 08:39:22 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:30.223124 7678 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:39:30.266224 7678 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:39:30.268146 7678 ts_tablet_manager.cc:344] Loading tablet metadata (0/1 complete) I0703 01:39:30.268201 7678 ts_tablet_manager.cc:1005] T a61eb64bf7cc417684a8446570d07208 P d7071ff1216d42808d7cfb3ba8e4ee0b: Loading tablet metadata I0703 01:39:30.272572 7678 ts_tablet_manager.cc:356] Loaded tablet metadata (1 live tablets) I0703 01:39:30.274444 7678 ts_tablet_manager.cc:1173] T a61eb64bf7cc417684a8446570d07208 P d7071ff1216d42808d7cfb3ba8e4ee0b: Registered tablet (data state: TABLET_DATA_READY) I0703 01:39:30.275825 7696 ts_tablet_manager.cc:1028] T a61eb64bf7cc417684a8446570d07208 P d7071ff1216d42808d7cfb3ba8e4ee0b: Bootstrapping tablet I0703 01:39:30.276826 7678 tablet_server_main.cc:90] Starting tablet server... I0703 01:39:30.277021 7696 tablet_bootstrap.cc:427] T a61eb64bf7cc417684a8446570d07208 P d7071ff1216d42808d7cfb3ba8e4ee0b: Bootstrap starting. I0703 01:39:30.280889 7453 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet a61eb64bf7cc417684a8446570d07208 on 29c07deff3f54b10b7ad9a60bc14edeb (127.5.240.194:48424) (TS 29c07deff3f54b10b7ad9a60bc14edeb not found in new config with opid_index 2) I0703 01:39:30.281571 7611 tablet_service.cc:800] Processing DeleteTablet for tablet a61eb64bf7cc417684a8446570d07208 with delete_type TABLET_DATA_TOMBSTONED (TS 29c07deff3f54b10b7ad9a60bc14edeb not found in new config with opid_index 2) from {username='jenkins'} at 127.5.240.194:55025 I0703 01:39:30.282054 7453 catalog_manager.cc:3103] TS 29c07deff3f54b10b7ad9a60bc14edeb (127.5.240.194:48424): tablet a61eb64bf7cc417684a8446570d07208 (table TestTable [id=24e11f69554b41ef994802e369af7c41]) successfully deleted I0703 01:39:30.305814 7696 log.cc:525] T a61eb64bf7cc417684a8446570d07208 P d7071ff1216d42808d7cfb3ba8e4ee0b: Log is configured to *not* fsync() on all Append() calls I0703 01:39:30.315773 7455 catalog_manager.cc:2903] Scheduling retry of a61eb64bf7cc417684a8446570d07208 Delete Tablet RPC for TS=d7071ff1216d42808d7cfb3ba8e4ee0b with a delay of 2078 ms (attempt = 8) W0703 01:39:30.315855 7455 catalog_manager.cc:2922] Async tablet task a61eb64bf7cc417684a8446570d07208 Delete Tablet RPC for TS=d7071ff1216d42808d7cfb3ba8e4ee0b failed: Not found: failed to reset TS proxy: Could not find TS for UUID d7071ff1216d42808d7cfb3ba8e4ee0b I0703 01:39:30.316263 7696 tablet_bootstrap.cc:427] T a61eb64bf7cc417684a8446570d07208 P d7071ff1216d42808d7cfb3ba8e4ee0b: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:39:30.316759 7696 tablet_bootstrap.cc:427] T a61eb64bf7cc417684a8446570d07208 P d7071ff1216d42808d7cfb3ba8e4ee0b: Bootstrap complete. I0703 01:39:30.317836 7696 ts_tablet_manager.cc:1045] T a61eb64bf7cc417684a8446570d07208 P d7071ff1216d42808d7cfb3ba8e4ee0b: Time spent bootstrapping tablet: real 0.041s user 0.010s sys 0.020s I0703 01:39:30.356739 7696 raft_consensus.cc:325] T a61eb64bf7cc417684a8446570d07208 P d7071ff1216d42808d7cfb3ba8e4ee0b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fbe0bff97843f89529ce5722817fad" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 60719 } } peers { permanent_uuid: "29c07deff3f54b10b7ad9a60bc14edeb" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 48424 } } peers { permanent_uuid: "d7071ff1216d42808d7cfb3ba8e4ee0b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42085 } } I0703 01:39:30.359621 7696 raft_consensus.cc:628] T a61eb64bf7cc417684a8446570d07208 P d7071ff1216d42808d7cfb3ba8e4ee0b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7071ff1216d42808d7cfb3ba8e4ee0b, State: Initialized, Role: FOLLOWER I0703 01:39:30.361163 7696 consensus_queue.cc:227] T a61eb64bf7cc417684a8446570d07208 P d7071ff1216d42808d7cfb3ba8e4ee0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5fbe0bff97843f89529ce5722817fad" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 60719 } } peers { permanent_uuid: "29c07deff3f54b10b7ad9a60bc14edeb" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 48424 } } peers { permanent_uuid: "d7071ff1216d42808d7cfb3ba8e4ee0b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42085 } } I0703 01:39:30.369879 7696 ts_tablet_manager.cc:1073] T a61eb64bf7cc417684a8446570d07208 P d7071ff1216d42808d7cfb3ba8e4ee0b: Time spent starting tablet: real 0.052s user 0.010s sys 0.028s I0703 01:39:30.399941 7678 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.193:42085 I0703 01:39:30.400382 7678 webserver.cc:175] Starting webserver on 127.5.240.193:58558 I0703 01:39:30.400425 7678 webserver.cc:186] Document root disabled I0703 01:39:30.401762 7678 webserver.cc:313] Webserver started. Bound to: http://127.5.240.193:58558/ I0703 01:39:30.402297 7678 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb I0703 01:39:30.403585 7678 tablet_server_main.cc:93] Tablet server successfully started. W0703 01:39:30.403590 7804 tablet.cc:1678] T a61eb64bf7cc417684a8446570d07208 P d7071ff1216d42808d7cfb3ba8e4ee0b: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:39:30.404410 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 7678 I0703 01:39:30.415839 7803 heartbeater.cc:346] Connected to a master server at 127.5.240.254:54399 I0703 01:39:30.416007 7803 heartbeater.cc:426] Registering TS with master... I0703 01:39:30.416507 7803 heartbeater.cc:475] Master 127.5.240.254:54399 requested a full tablet report, sending... I0703 01:39:30.417838 7472 ts_manager.cc:98] Registered new tserver with Master: d7071ff1216d42808d7cfb3ba8e4ee0b (127.5.240.193:42085) I0703 01:39:30.418332 7472 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet a61eb64bf7cc417684a8446570d07208 on d7071ff1216d42808d7cfb3ba8e4ee0b (127.5.240.193:42085) (Replica has no consensus available (current committed config index is 7)) I0703 01:39:30.419766 7472 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.193:56596 I0703 01:39:30.424494 7739 tablet_service.cc:800] Processing DeleteTablet for tablet a61eb64bf7cc417684a8446570d07208 with delete_type TABLET_DATA_TOMBSTONED (Replica has no consensus available (current committed config index is 7)) from {username='jenkins'} at 127.5.240.193:33699 I0703 01:39:30.425323 7809 tablet_replica.cc:261] T a61eb64bf7cc417684a8446570d07208 P d7071ff1216d42808d7cfb3ba8e4ee0b: stopping tablet replica I0703 01:39:30.425873 7809 maintenance_manager.cc:237] P d7071ff1216d42808d7cfb3ba8e4ee0b: Unregistered op CompactRowSetsOp(a61eb64bf7cc417684a8446570d07208) I0703 01:39:30.425936 7809 maintenance_manager.cc:237] P d7071ff1216d42808d7cfb3ba8e4ee0b: Unregistered op MinorDeltaCompactionOp(a61eb64bf7cc417684a8446570d07208) I0703 01:39:30.425971 7809 maintenance_manager.cc:237] P d7071ff1216d42808d7cfb3ba8e4ee0b: Unregistered op MajorDeltaCompactionOp(a61eb64bf7cc417684a8446570d07208) I0703 01:39:30.426005 7809 maintenance_manager.cc:237] P d7071ff1216d42808d7cfb3ba8e4ee0b: Unregistered op UndoDeltaBlockGCOp(a61eb64bf7cc417684a8446570d07208) I0703 01:39:30.426065 7809 maintenance_manager.cc:237] P d7071ff1216d42808d7cfb3ba8e4ee0b: Unregistered op FlushMRSOp(a61eb64bf7cc417684a8446570d07208) I0703 01:39:30.426100 7809 maintenance_manager.cc:237] P d7071ff1216d42808d7cfb3ba8e4ee0b: Unregistered op FlushDeltaMemStoresOp(a61eb64bf7cc417684a8446570d07208) I0703 01:39:30.426127 7809 maintenance_manager.cc:237] P d7071ff1216d42808d7cfb3ba8e4ee0b: Unregistered op LogGCOp(a61eb64bf7cc417684a8446570d07208) I0703 01:39:30.426194 7809 raft_consensus.cc:2040] T a61eb64bf7cc417684a8446570d07208 P d7071ff1216d42808d7cfb3ba8e4ee0b [term 1 FOLLOWER]: Raft consensus shutting down. I0703 01:39:30.426385 7809 raft_consensus.cc:2067] T a61eb64bf7cc417684a8446570d07208 P d7071ff1216d42808d7cfb3ba8e4ee0b [term 1 FOLLOWER]: Raft consensus is shut down! I0703 01:39:30.427444 7809 ts_tablet_manager.cc:1369] T a61eb64bf7cc417684a8446570d07208 P d7071ff1216d42808d7cfb3ba8e4ee0b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:39:30.431433 7809 ts_tablet_manager.cc:1382] T a61eb64bf7cc417684a8446570d07208 P d7071ff1216d42808d7cfb3ba8e4ee0b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1 I0703 01:39:30.431510 7809 log.cc:986] T a61eb64bf7cc417684a8446570d07208 P d7071ff1216d42808d7cfb3ba8e4ee0b: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleFollower.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal/wals/a61eb64bf7cc417684a8446570d07208 I0703 01:39:30.432097 7455 catalog_manager.cc:3103] TS d7071ff1216d42808d7cfb3ba8e4ee0b (127.5.240.193:42085): tablet a61eb64bf7cc417684a8446570d07208 (table TestTable [id=24e11f69554b41ef994802e369af7c41]) successfully deleted I0703 01:39:30.436781 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 7678 I0703 01:39:30.442970 7800 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:39:30.486362 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 7678 I0703 01:39:30.495005 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 7553 W0703 01:39:30.494953 7456 connection.cc:512] server connection from 127.5.240.193:56596 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:56596): Connection reset by peer (error 104) I0703 01:39:30.498592 7672 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:39:30.540406 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 7553 W0703 01:39:30.549016 7455 connection.cc:512] server connection from 127.5.240.194:52130 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:52130): Connection reset by peer (error 104) I0703 01:39:30.549485 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 7016 I0703 01:39:30.550308 7132 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:39:30.593600 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 7016 W0703 01:39:30.601846 7454 connection.cc:512] server connection from 127.5.240.195:34511 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.195:34511): Connection reset by peer (error 104) W0703 01:39:30.602003 7269 connection.cc:512] server connection from 127.5.240.195:53995 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.195:53995): Connection reset by peer (error 104) I0703 01:39:30.602051 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 7138 W0703 01:39:30.602257 7267 connection.cc:512] client connection to 127.5.240.195:33176 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.195:33176): Connection reset by peer (error 104) I0703 01:39:30.602907 7254 generic_service.cc:169] Checking for leaks (request via RPC) W0703 01:39:30.631975 7267 consensus_peers.cc:433] T a61eb64bf7cc417684a8446570d07208 P a5fbe0bff97843f89529ce5722817fad -> Peer d45639fe421a4c0c81abaf8a04d04fa9 (127.5.240.195:33176): Couldn't send request to peer d45639fe421a4c0c81abaf8a04d04fa9 for tablet a61eb64bf7cc417684a8446570d07208. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.195:33176: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. I0703 01:39:30.648010 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 7138 W0703 01:39:30.656189 7454 connection.cc:512] server connection from 127.5.240.196:33426 recv error: Network error: failed to read from TLS socket (remote: 0.0.0.0:0): Connection reset by peer (error 104) W0703 01:39:30.656256 7270 connection.cc:512] client connection to 127.5.240.196:59711 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.196:59711): Connection reset by peer (error 104) W0703 01:39:30.656496 7454 connection.cc:185] Error closing socket: Network error: TlsSocket::Close: Broken pipe I0703 01:39:30.656580 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 7260 W0703 01:39:30.656666 7267 connection.cc:512] server connection from 127.5.240.196:51256 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.196:51256): Connection reset by peer (error 104) I0703 01:39:30.657279 7376 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:39:30.703558 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 7260 W0703 01:39:30.711773 7456 connection.cc:512] server connection from 127.5.240.197:49117 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.197:49117): Connection reset by peer (error 104) I0703 01:39:30.712087 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master pid 7445 I0703 01:39:30.712831 7502 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:39:30.778275 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master with pid 7445 [ OK ] AdminCliTest.TestUnsafeChangeConfigOnSingleFollower (8353 ms) [ RUN ] AdminCliTest.TestUnsafeChangeConfigOnSingleLeader I0703 01:39:30.800484 6083 test_util.cc:208] Using random seed: 1085590352 I0703 01:39:30.802047 6083 ts_itest-base.cc:113] Starting cluster with: I0703 01:39:30.802075 6083 ts_itest-base.cc:114] -------------- I0703 01:39:30.802098 6083 ts_itest-base.cc:115] 5 tablet servers I0703 01:39:30.802117 6083 ts_itest-base.cc:116] 3 replicas per TS I0703 01:39:30.802134 6083 ts_itest-base.cc:117] -------------- I0703 01:39:30.803258 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:50537 --webserver_interface=127.5.240.254 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:39:30.982478 7823 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:39:30.982714 7823 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:39:30.984908 7823 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0703 01:39:30.984946 7823 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:39:30.984972 7823 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0703 01:39:30.984998 7823 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0703 01:39:30.993073 7823 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:50537 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.5.240.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:39:30.998703 7823 master_main.cc:80] Initializing master server... I0703 01:39:30.999756 7823 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 138708us I0703 01:39:31.000814 7823 fs_manager.cc:261] Metadata directory not provided I0703 01:39:31.000869 7823 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:39:31.001085 7823 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:39:31.001143 7823 server_base.cc:433] Attempting to create new FS layout instead I0703 01:39:31.005321 7823 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/instance: uuid: "f23c54bf26ca4ea4a4d3f921196318d6" format_stamp: "Formatted at 2018-07-03 08:39:31 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:31.005856 7823 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "f23c54bf26ca4ea4a4d3f921196318d6" format_stamp: "Formatted at 2018-07-03 08:39:31 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:31.092556 7823 fs_manager.cc:501] Time spent creating directory manager: real 0.086s user 0.003s sys 0.005s I0703 01:39:31.093840 7823 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:39:31.094251 7823 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:39:31.097986 7823 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0703 01:39:31.098042 7823 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal uuid: "f23c54bf26ca4ea4a4d3f921196318d6" format_stamp: "Formatted at 2018-07-03 08:39:31 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:31.098155 7823 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:39:31.125121 7823 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:39:31.126090 7823 master_main.cc:83] Starting Master server... I0703 01:39:31.163293 7823 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.254:50537 I0703 01:39:31.163774 7823 webserver.cc:175] Starting webserver on 127.5.240.254:0 I0703 01:39:31.163820 7823 webserver.cc:186] Document root disabled I0703 01:39:31.165006 7823 webserver.cc:313] Webserver started. Bound to: http://127.5.240.254:53890/ I0703 01:39:31.165601 7823 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb I0703 01:39:31.169252 7882 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:39:31.174283 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master as pid 7823 I0703 01:39:31.175243 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.5.240.193:0 --local_ip_for_outbound_sockets=127.5.240.193 --webserver_interface=127.5.240.193 --webserver_port=0 --tserver_master_addrs=127.5.240.254:50537 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0703 01:39:31.181838 7882 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P f23c54bf26ca4ea4a4d3f921196318d6: Bootstrap starting. I0703 01:39:31.183841 7882 tablet_bootstrap.cc:578] T 00000000000000000000000000000000 P f23c54bf26ca4ea4a4d3f921196318d6: No blocks or log segments found. Creating new log. I0703 01:39:31.184566 7882 log.cc:525] T 00000000000000000000000000000000 P f23c54bf26ca4ea4a4d3f921196318d6: Log is configured to *not* fsync() on all Append() calls I0703 01:39:31.186236 7882 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P f23c54bf26ca4ea4a4d3f921196318d6: No bootstrap required, opened a new log I0703 01:39:31.192421 7882 raft_consensus.cc:325] T 00000000000000000000000000000000 P f23c54bf26ca4ea4a4d3f921196318d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f23c54bf26ca4ea4a4d3f921196318d6" member_type: VOTER } I0703 01:39:31.192634 7882 raft_consensus.cc:351] T 00000000000000000000000000000000 P f23c54bf26ca4ea4a4d3f921196318d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:39:31.192694 7882 raft_consensus.cc:628] T 00000000000000000000000000000000 P f23c54bf26ca4ea4a4d3f921196318d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23c54bf26ca4ea4a4d3f921196318d6, State: Initialized, Role: FOLLOWER I0703 01:39:31.193184 7882 consensus_queue.cc:227] T 00000000000000000000000000000000 P f23c54bf26ca4ea4a4d3f921196318d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f23c54bf26ca4ea4a4d3f921196318d6" member_type: VOTER } I0703 01:39:31.193326 7882 raft_consensus.cc:365] T 00000000000000000000000000000000 P f23c54bf26ca4ea4a4d3f921196318d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:39:31.193403 7882 raft_consensus.cc:457] T 00000000000000000000000000000000 P f23c54bf26ca4ea4a4d3f921196318d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:39:31.193478 7882 raft_consensus.cc:2730] T 00000000000000000000000000000000 P f23c54bf26ca4ea4a4d3f921196318d6: Snoozing failure detection for 1.935s (starting election) I0703 01:39:31.193563 7882 raft_consensus.cc:2782] T 00000000000000000000000000000000 P f23c54bf26ca4ea4a4d3f921196318d6 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:39:31.194914 7882 raft_consensus.cc:479] T 00000000000000000000000000000000 P f23c54bf26ca4ea4a4d3f921196318d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f23c54bf26ca4ea4a4d3f921196318d6" member_type: VOTER } I0703 01:39:31.195272 7882 leader_election.cc:270] T 00000000000000000000000000000000 P f23c54bf26ca4ea4a4d3f921196318d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:39:31.196750 7885 raft_consensus.cc:2730] T 00000000000000000000000000000000 P f23c54bf26ca4ea4a4d3f921196318d6: Snoozing failure detection for 1.526s (election complete) I0703 01:39:31.197005 7885 raft_consensus.cc:2537] T 00000000000000000000000000000000 P f23c54bf26ca4ea4a4d3f921196318d6 [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:39:31.202803 7885 raft_consensus.cc:593] T 00000000000000000000000000000000 P f23c54bf26ca4ea4a4d3f921196318d6 [term 1 LEADER]: Becoming Leader. State: Replica: f23c54bf26ca4ea4a4d3f921196318d6, State: Running, Role: LEADER I0703 01:39:31.203132 7885 consensus_queue.cc:204] T 00000000000000000000000000000000 P f23c54bf26ca4ea4a4d3f921196318d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f23c54bf26ca4ea4a4d3f921196318d6" member_type: VOTER } I0703 01:39:31.206221 7882 sys_catalog.cc:337] T 00000000000000000000000000000000 P f23c54bf26ca4ea4a4d3f921196318d6 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "f23c54bf26ca4ea4a4d3f921196318d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f23c54bf26ca4ea4a4d3f921196318d6" member_type: VOTER } } I0703 01:39:31.206215 7886 sys_catalog.cc:337] T 00000000000000000000000000000000 P f23c54bf26ca4ea4a4d3f921196318d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f23c54bf26ca4ea4a4d3f921196318d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f23c54bf26ca4ea4a4d3f921196318d6" member_type: VOTER } } I0703 01:39:31.207088 7882 sys_catalog.cc:340] T 00000000000000000000000000000000 P f23c54bf26ca4ea4a4d3f921196318d6 [sys.catalog]: This master's current role is: LEADER I0703 01:39:31.207322 7886 sys_catalog.cc:340] T 00000000000000000000000000000000 P f23c54bf26ca4ea4a4d3f921196318d6 [sys.catalog]: This master's current role is: LEADER I0703 01:39:31.207088 7887 sys_catalog.cc:337] T 00000000000000000000000000000000 P f23c54bf26ca4ea4a4d3f921196318d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f23c54bf26ca4ea4a4d3f921196318d6. Latest consensus state: current_term: 1 leader_uuid: "f23c54bf26ca4ea4a4d3f921196318d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f23c54bf26ca4ea4a4d3f921196318d6" member_type: VOTER } } I0703 01:39:31.207650 7887 sys_catalog.cc:340] T 00000000000000000000000000000000 P f23c54bf26ca4ea4a4d3f921196318d6 [sys.catalog]: This master's current role is: LEADER I0703 01:39:31.209945 7882 sys_catalog.cc:424] T 00000000000000000000000000000000 P f23c54bf26ca4ea4a4d3f921196318d6 [sys.catalog]: configured and running, proceeding with master startup. I0703 01:39:31.210279 7889 catalog_manager.cc:1006] Loading table and tablet metadata into memory... I0703 01:39:31.211541 7823 master_main.cc:86] Master server successfully started. I0703 01:39:31.216564 7889 catalog_manager.cc:1017] Initializing Kudu internal certificate authority... I0703 01:39:31.276625 7889 catalog_manager.cc:904] Generated new certificate authority record I0703 01:39:31.278630 7889 catalog_manager.cc:1026] Loading token signing keys... I0703 01:39:31.287058 7889 catalog_manager.cc:4102] T 00000000000000000000000000000000 P f23c54bf26ca4ea4a4d3f921196318d6: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:39:31.349570 7883 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:39:31.349779 7883 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:39:31.351794 7883 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:39:31.351915 7883 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.193 I0703 01:39:31.359210 7883 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.5.240.193 --webserver_port=0 --tserver_master_addrs=127.5.240.254:50537 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.193 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:39:31.365819 7883 tablet_server_main.cc:85] Initializing tablet server... I0703 01:39:31.367049 7883 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 139208us I0703 01:39:31.368043 7883 fs_manager.cc:261] Metadata directory not provided I0703 01:39:31.368106 7883 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:39:31.368351 7883 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:39:31.368417 7883 server_base.cc:433] Attempting to create new FS layout instead I0703 01:39:31.372570 7883 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "8ca84f980eb34212944aaf54d04fd950" format_stamp: "Formatted at 2018-07-03 08:39:31 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:31.373045 7883 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "8ca84f980eb34212944aaf54d04fd950" format_stamp: "Formatted at 2018-07-03 08:39:31 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:31.380887 7883 fs_manager.cc:501] Time spent creating directory manager: real 0.008s user 0.004s sys 0.002s I0703 01:39:31.382067 7883 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:39:31.382477 7883 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:39:31.386590 7883 fs_manager.cc:417] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s I0703 01:39:31.386651 7883 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal uuid: "8ca84f980eb34212944aaf54d04fd950" format_stamp: "Formatted at 2018-07-03 08:39:31 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:31.386770 7883 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:39:31.412279 7883 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:39:31.414809 7883 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:39:31.415275 7883 tablet_server_main.cc:90] Starting tablet server... I0703 01:39:31.516317 7883 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.193:42094 I0703 01:39:31.516803 7883 webserver.cc:175] Starting webserver on 127.5.240.193:0 I0703 01:39:31.516841 7883 webserver.cc:186] Document root disabled I0703 01:39:31.517644 7883 webserver.cc:313] Webserver started. Bound to: http://127.5.240.193:51893/ I0703 01:39:31.518172 7883 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb I0703 01:39:31.522764 7883 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:39:31.528049 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 7883 I0703 01:39:31.528892 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.5.240.194:0 --local_ip_for_outbound_sockets=127.5.240.194 --webserver_interface=127.5.240.194 --webserver_port=0 --tserver_master_addrs=127.5.240.254:50537 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0703 01:39:31.542971 8012 heartbeater.cc:346] Connected to a master server at 127.5.240.254:50537 I0703 01:39:31.543120 8012 heartbeater.cc:426] Registering TS with master... I0703 01:39:31.543920 8012 heartbeater.cc:475] Master 127.5.240.254:50537 requested a full tablet report, sending... I0703 01:39:31.545920 7849 ts_manager.cc:98] Registered new tserver with Master: 8ca84f980eb34212944aaf54d04fd950 (127.5.240.193:42094) I0703 01:39:31.548969 7849 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.193:55905 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:39:31.691210 8016 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:39:31.691442 8016 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:39:31.693650 8016 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:39:31.693758 8016 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.194 I0703 01:39:31.699934 8016 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.194:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.5.240.194 --webserver_port=0 --tserver_master_addrs=127.5.240.254:50537 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.194 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:39:31.706554 8016 tablet_server_main.cc:85] Initializing tablet server... I0703 01:39:31.707756 8016 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 139208us I0703 01:39:31.708712 8016 fs_manager.cc:261] Metadata directory not provided I0703 01:39:31.708775 8016 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:39:31.709373 8016 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:39:31.709435 8016 server_base.cc:433] Attempting to create new FS layout instead I0703 01:39:31.713471 8016 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "eb53f28829e04691a2601f44b09fbbe0" format_stamp: "Formatted at 2018-07-03 08:39:31 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:31.714018 8016 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "eb53f28829e04691a2601f44b09fbbe0" format_stamp: "Formatted at 2018-07-03 08:39:31 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:31.721731 8016 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.004s sys 0.003s I0703 01:39:31.722892 8016 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:39:31.723273 8016 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:39:31.726358 8016 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0703 01:39:31.726420 8016 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal uuid: "eb53f28829e04691a2601f44b09fbbe0" format_stamp: "Formatted at 2018-07-03 08:39:31 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:31.726536 8016 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:39:31.764098 8016 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:39:31.765895 8016 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:39:31.766219 8016 tablet_server_main.cc:90] Starting tablet server... I0703 01:39:31.850239 8016 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.194:39440 I0703 01:39:31.850715 8016 webserver.cc:175] Starting webserver on 127.5.240.194:0 I0703 01:39:31.850754 8016 webserver.cc:186] Document root disabled I0703 01:39:31.851941 8016 webserver.cc:313] Webserver started. Bound to: http://127.5.240.194:59096/ I0703 01:39:31.852470 8016 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb I0703 01:39:31.854308 8016 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:39:31.861420 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 8016 I0703 01:39:31.862231 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.5.240.195:0 --local_ip_for_outbound_sockets=127.5.240.195 --webserver_interface=127.5.240.195 --webserver_port=0 --tserver_master_addrs=127.5.240.254:50537 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0703 01:39:31.866986 8135 heartbeater.cc:346] Connected to a master server at 127.5.240.254:50537 I0703 01:39:31.867121 8135 heartbeater.cc:426] Registering TS with master... I0703 01:39:31.867900 8135 heartbeater.cc:475] Master 127.5.240.254:50537 requested a full tablet report, sending... I0703 01:39:31.868949 7849 ts_manager.cc:98] Registered new tserver with Master: eb53f28829e04691a2601f44b09fbbe0 (127.5.240.194:39440) I0703 01:39:31.869794 7849 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.194:35317 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:39:32.018898 8138 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:39:32.019148 8138 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:39:32.021165 8138 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:39:32.021303 8138 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.195 I0703 01:39:32.027935 8138 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.195:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.5.240.195 --webserver_port=0 --tserver_master_addrs=127.5.240.254:50537 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.195 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:39:32.034605 8138 tablet_server_main.cc:85] Initializing tablet server... I0703 01:39:32.035773 8138 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 139708us I0703 01:39:32.036720 8138 fs_manager.cc:261] Metadata directory not provided I0703 01:39:32.036782 8138 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:39:32.037014 8138 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:39:32.037075 8138 server_base.cc:433] Attempting to create new FS layout instead I0703 01:39:32.041209 8138 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "576ad88f26964932bee2e504cb16970b" format_stamp: "Formatted at 2018-07-03 08:39:32 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:32.041790 8138 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "576ad88f26964932bee2e504cb16970b" format_stamp: "Formatted at 2018-07-03 08:39:32 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:32.048990 8138 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.003s sys 0.003s I0703 01:39:32.050173 8138 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:39:32.050567 8138 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:39:32.054136 8138 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0703 01:39:32.054193 8138 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal uuid: "576ad88f26964932bee2e504cb16970b" format_stamp: "Formatted at 2018-07-03 08:39:32 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:32.054307 8138 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:39:32.096707 8138 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:39:32.098462 8138 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:39:32.098783 8138 tablet_server_main.cc:90] Starting tablet server... I0703 01:39:32.183465 8138 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.195:48979 I0703 01:39:32.183909 8138 webserver.cc:175] Starting webserver on 127.5.240.195:0 I0703 01:39:32.183948 8138 webserver.cc:186] Document root disabled I0703 01:39:32.185142 8138 webserver.cc:313] Webserver started. Bound to: http://127.5.240.195:41885/ I0703 01:39:32.185703 8138 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb I0703 01:39:32.187613 8138 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:39:32.194680 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 8138 I0703 01:39:32.195333 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.5.240.196:0 --local_ip_for_outbound_sockets=127.5.240.196 --webserver_interface=127.5.240.196 --webserver_port=0 --tserver_master_addrs=127.5.240.254:50537 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0703 01:39:32.199676 8257 heartbeater.cc:346] Connected to a master server at 127.5.240.254:50537 I0703 01:39:32.199818 8257 heartbeater.cc:426] Registering TS with master... I0703 01:39:32.200639 8257 heartbeater.cc:475] Master 127.5.240.254:50537 requested a full tablet report, sending... I0703 01:39:32.201612 7849 ts_manager.cc:98] Registered new tserver with Master: 576ad88f26964932bee2e504cb16970b (127.5.240.195:48979) I0703 01:39:32.202503 7849 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.195:34110 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:39:32.352432 8260 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:39:32.352671 8260 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:39:32.354755 8260 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:39:32.354851 8260 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.196 I0703 01:39:32.361440 8260 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.196:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb --webserver_interface=127.5.240.196 --webserver_port=0 --tserver_master_addrs=127.5.240.254:50537 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.196 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:39:32.368177 8260 tablet_server_main.cc:85] Initializing tablet server... I0703 01:39:32.369314 8260 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 139708us I0703 01:39:32.370280 8260 fs_manager.cc:261] Metadata directory not provided I0703 01:39:32.370333 8260 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:39:32.370944 8260 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:39:32.371002 8260 server_base.cc:433] Attempting to create new FS layout instead I0703 01:39:32.375092 8260 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/instance: uuid: "8e286bd098564982987f97b2de7590c8" format_stamp: "Formatted at 2018-07-03 08:39:32 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:32.375623 8260 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal/instance: uuid: "8e286bd098564982987f97b2de7590c8" format_stamp: "Formatted at 2018-07-03 08:39:32 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:32.383352 8260 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.001s sys 0.005s I0703 01:39:32.384524 8260 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:39:32.384958 8260 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:39:32.388016 8260 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s I0703 01:39:32.388074 8260 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal uuid: "8e286bd098564982987f97b2de7590c8" format_stamp: "Formatted at 2018-07-03 08:39:32 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:32.388191 8260 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:39:32.421561 8260 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:39:32.423817 8260 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:39:32.424163 8260 tablet_server_main.cc:90] Starting tablet server... I0703 01:39:32.507131 8260 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.196:49273 I0703 01:39:32.507563 8260 webserver.cc:175] Starting webserver on 127.5.240.196:0 I0703 01:39:32.507618 8260 webserver.cc:186] Document root disabled I0703 01:39:32.508291 8260 webserver.cc:313] Webserver started. Bound to: http://127.5.240.196:58683/ I0703 01:39:32.508829 8260 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb I0703 01:39:32.511449 8260 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:39:32.517333 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 8260 I0703 01:39:32.518110 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data --block_manager=log --rpc_bind_addresses=127.5.240.197:0 --local_ip_for_outbound_sockets=127.5.240.197 --webserver_interface=127.5.240.197 --webserver_port=0 --tserver_master_addrs=127.5.240.254:50537 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0703 01:39:32.523674 8379 heartbeater.cc:346] Connected to a master server at 127.5.240.254:50537 I0703 01:39:32.523834 8379 heartbeater.cc:426] Registering TS with master... I0703 01:39:32.524765 8379 heartbeater.cc:475] Master 127.5.240.254:50537 requested a full tablet report, sending... I0703 01:39:32.525851 7849 ts_manager.cc:98] Registered new tserver with Master: 8e286bd098564982987f97b2de7590c8 (127.5.240.196:49273) I0703 01:39:32.526695 7849 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.196:36671 I0703 01:39:32.551687 8012 heartbeater.cc:467] Master 127.5.240.254:50537 was elected leader, sending a full tablet report... WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:39:32.674726 8382 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:39:32.674940 8382 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:39:32.676905 8382 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:39:32.676993 8382 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.197 I0703 01:39:32.683158 8382 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.197:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb --webserver_interface=127.5.240.197 --webserver_port=0 --tserver_master_addrs=127.5.240.254:50537 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.197 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:39:32.689817 8382 tablet_server_main.cc:85] Initializing tablet server... I0703 01:39:32.690927 8382 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 139708us I0703 01:39:32.691857 8382 fs_manager.cc:261] Metadata directory not provided I0703 01:39:32.691910 8382 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:39:32.692497 8382 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:39:32.692569 8382 server_base.cc:433] Attempting to create new FS layout instead I0703 01:39:32.696642 8382 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/instance: uuid: "4028d6eb9da140bfb94a7fc80bc102c0" format_stamp: "Formatted at 2018-07-03 08:39:32 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:32.697149 8382 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal/instance: uuid: "4028d6eb9da140bfb94a7fc80bc102c0" format_stamp: "Formatted at 2018-07-03 08:39:32 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:32.704457 8382 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.004s sys 0.003s I0703 01:39:32.705652 8382 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:39:32.706032 8382 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:39:32.709589 8382 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s I0703 01:39:32.709638 8382 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal uuid: "4028d6eb9da140bfb94a7fc80bc102c0" format_stamp: "Formatted at 2018-07-03 08:39:32 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:32.709748 8382 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:39:32.758236 8382 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:39:32.760509 8382 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:39:32.760870 8382 tablet_server_main.cc:90] Starting tablet server... I0703 01:39:32.845485 8382 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.197:36566 I0703 01:39:32.845926 8382 webserver.cc:175] Starting webserver on 127.5.240.197:0 I0703 01:39:32.845958 8382 webserver.cc:186] Document root disabled I0703 01:39:32.846642 8382 webserver.cc:313] Webserver started. Bound to: http://127.5.240.197:41458/ I0703 01:39:32.847162 8382 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb I0703 01:39:32.849210 8382 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:39:32.850119 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 8382 I0703 01:39:32.861131 8501 heartbeater.cc:346] Connected to a master server at 127.5.240.254:50537 I0703 01:39:32.861297 8501 heartbeater.cc:426] Registering TS with master... I0703 01:39:32.862306 8501 heartbeater.cc:475] Master 127.5.240.254:50537 requested a full tablet report, sending... I0703 01:39:32.863528 7848 ts_manager.cc:98] Registered new tserver with Master: 4028d6eb9da140bfb94a7fc80bc102c0 (127.5.240.197:36566) I0703 01:39:32.864423 7848 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.197:41525 I0703 01:39:32.864773 6083 external_mini_cluster.cc:490] 5 TS(s) registered with all masters I0703 01:39:32.871603 8135 heartbeater.cc:467] Master 127.5.240.254:50537 was elected leader, sending a full tablet report... I0703 01:39:32.874719 7849 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.5.240.254:45462: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0703 01:39:32.903882 8072 tablet_service.cc:759] Processing CreateTablet for tablet 0b854d2292eb45f7b09f7279fcd23a62 (table=TestTable [id=3578883b7bc34387a7c9a55537c46eed]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:39:32.904614 8072 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 0b854d2292eb45f7b09f7279fcd23a62. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:39:32.904842 8316 tablet_service.cc:759] Processing CreateTablet for tablet 0b854d2292eb45f7b09f7279fcd23a62 (table=TestTable [id=3578883b7bc34387a7c9a55537c46eed]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:39:32.905629 8316 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 0b854d2292eb45f7b09f7279fcd23a62. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:39:32.907629 8438 tablet_service.cc:759] Processing CreateTablet for tablet 0b854d2292eb45f7b09f7279fcd23a62 (table=TestTable [id=3578883b7bc34387a7c9a55537c46eed]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:39:32.908447 8438 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 0b854d2292eb45f7b09f7279fcd23a62. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:39:32.914623 8072 ts_tablet_manager.cc:1173] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0: Registered tablet (data state: TABLET_DATA_READY) I0703 01:39:32.915380 8521 ts_tablet_manager.cc:1028] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0: Bootstrapping tablet I0703 01:39:32.915951 8521 tablet_bootstrap.cc:427] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0: Bootstrap starting. I0703 01:39:32.918051 8521 tablet_bootstrap.cc:578] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0: No blocks or log segments found. Creating new log. I0703 01:39:32.918834 8521 log.cc:525] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0: Log is configured to *not* fsync() on all Append() calls I0703 01:39:32.919798 8316 ts_tablet_manager.cc:1173] T 0b854d2292eb45f7b09f7279fcd23a62 P 8e286bd098564982987f97b2de7590c8: Registered tablet (data state: TABLET_DATA_READY) I0703 01:39:32.920518 8523 ts_tablet_manager.cc:1028] T 0b854d2292eb45f7b09f7279fcd23a62 P 8e286bd098564982987f97b2de7590c8: Bootstrapping tablet I0703 01:39:32.921337 8521 tablet_bootstrap.cc:427] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0: No bootstrap required, opened a new log I0703 01:39:32.921514 8521 ts_tablet_manager.cc:1045] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s I0703 01:39:32.922669 8523 tablet_bootstrap.cc:427] T 0b854d2292eb45f7b09f7279fcd23a62 P 8e286bd098564982987f97b2de7590c8: Bootstrap starting. I0703 01:39:32.923991 8438 ts_tablet_manager.cc:1173] T 0b854d2292eb45f7b09f7279fcd23a62 P 4028d6eb9da140bfb94a7fc80bc102c0: Registered tablet (data state: TABLET_DATA_READY) I0703 01:39:32.925508 8524 ts_tablet_manager.cc:1028] T 0b854d2292eb45f7b09f7279fcd23a62 P 4028d6eb9da140bfb94a7fc80bc102c0: Bootstrapping tablet I0703 01:39:32.925839 8523 tablet_bootstrap.cc:578] T 0b854d2292eb45f7b09f7279fcd23a62 P 8e286bd098564982987f97b2de7590c8: No blocks or log segments found. Creating new log. I0703 01:39:32.926131 8524 tablet_bootstrap.cc:427] T 0b854d2292eb45f7b09f7279fcd23a62 P 4028d6eb9da140bfb94a7fc80bc102c0: Bootstrap starting. I0703 01:39:32.926698 8523 log.cc:525] T 0b854d2292eb45f7b09f7279fcd23a62 P 8e286bd098564982987f97b2de7590c8: Log is configured to *not* fsync() on all Append() calls I0703 01:39:32.929065 8524 tablet_bootstrap.cc:578] T 0b854d2292eb45f7b09f7279fcd23a62 P 4028d6eb9da140bfb94a7fc80bc102c0: No blocks or log segments found. Creating new log. I0703 01:39:32.929877 8523 tablet_bootstrap.cc:427] T 0b854d2292eb45f7b09f7279fcd23a62 P 8e286bd098564982987f97b2de7590c8: No bootstrap required, opened a new log I0703 01:39:32.929982 8524 log.cc:525] T 0b854d2292eb45f7b09f7279fcd23a62 P 4028d6eb9da140bfb94a7fc80bc102c0: Log is configured to *not* fsync() on all Append() calls I0703 01:39:32.930057 8523 ts_tablet_manager.cc:1045] T 0b854d2292eb45f7b09f7279fcd23a62 P 8e286bd098564982987f97b2de7590c8: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s I0703 01:39:32.929818 8521 raft_consensus.cc:325] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4028d6eb9da140bfb94a7fc80bc102c0" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 36566 } } peers { permanent_uuid: "8e286bd098564982987f97b2de7590c8" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 49273 } } peers { permanent_uuid: "eb53f28829e04691a2601f44b09fbbe0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39440 } } I0703 01:39:32.930235 8521 raft_consensus.cc:351] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:39:32.930320 8521 raft_consensus.cc:628] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb53f28829e04691a2601f44b09fbbe0, State: Initialized, Role: FOLLOWER I0703 01:39:32.930830 8521 consensus_queue.cc:227] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4028d6eb9da140bfb94a7fc80bc102c0" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 36566 } } peers { permanent_uuid: "8e286bd098564982987f97b2de7590c8" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 49273 } } peers { permanent_uuid: "eb53f28829e04691a2601f44b09fbbe0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39440 } } I0703 01:39:32.933286 8521 ts_tablet_manager.cc:1073] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s I0703 01:39:32.935174 8524 tablet_bootstrap.cc:427] T 0b854d2292eb45f7b09f7279fcd23a62 P 4028d6eb9da140bfb94a7fc80bc102c0: No bootstrap required, opened a new log I0703 01:39:32.935403 8524 ts_tablet_manager.cc:1045] T 0b854d2292eb45f7b09f7279fcd23a62 P 4028d6eb9da140bfb94a7fc80bc102c0: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s I0703 01:39:32.939287 8523 raft_consensus.cc:325] T 0b854d2292eb45f7b09f7279fcd23a62 P 8e286bd098564982987f97b2de7590c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4028d6eb9da140bfb94a7fc80bc102c0" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 36566 } } peers { permanent_uuid: "8e286bd098564982987f97b2de7590c8" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 49273 } } peers { permanent_uuid: "eb53f28829e04691a2601f44b09fbbe0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39440 } } I0703 01:39:32.939741 8523 raft_consensus.cc:351] T 0b854d2292eb45f7b09f7279fcd23a62 P 8e286bd098564982987f97b2de7590c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:39:32.939838 8523 raft_consensus.cc:628] T 0b854d2292eb45f7b09f7279fcd23a62 P 8e286bd098564982987f97b2de7590c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e286bd098564982987f97b2de7590c8, State: Initialized, Role: FOLLOWER I0703 01:39:32.940327 8523 consensus_queue.cc:227] T 0b854d2292eb45f7b09f7279fcd23a62 P 8e286bd098564982987f97b2de7590c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4028d6eb9da140bfb94a7fc80bc102c0" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 36566 } } peers { permanent_uuid: "8e286bd098564982987f97b2de7590c8" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 49273 } } peers { permanent_uuid: "eb53f28829e04691a2601f44b09fbbe0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39440 } } I0703 01:39:32.941210 8379 heartbeater.cc:467] Master 127.5.240.254:50537 was elected leader, sending a full tablet report... I0703 01:39:32.942230 8523 ts_tablet_manager.cc:1073] T 0b854d2292eb45f7b09f7279fcd23a62 P 8e286bd098564982987f97b2de7590c8: Time spent starting tablet: real 0.012s user 0.008s sys 0.003s I0703 01:39:32.943321 8524 raft_consensus.cc:325] T 0b854d2292eb45f7b09f7279fcd23a62 P 4028d6eb9da140bfb94a7fc80bc102c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4028d6eb9da140bfb94a7fc80bc102c0" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 36566 } } peers { permanent_uuid: "8e286bd098564982987f97b2de7590c8" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 49273 } } peers { permanent_uuid: "eb53f28829e04691a2601f44b09fbbe0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39440 } } I0703 01:39:32.943608 8524 raft_consensus.cc:351] T 0b854d2292eb45f7b09f7279fcd23a62 P 4028d6eb9da140bfb94a7fc80bc102c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:39:32.943677 8524 raft_consensus.cc:628] T 0b854d2292eb45f7b09f7279fcd23a62 P 4028d6eb9da140bfb94a7fc80bc102c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4028d6eb9da140bfb94a7fc80bc102c0, State: Initialized, Role: FOLLOWER I0703 01:39:32.944022 8524 consensus_queue.cc:227] T 0b854d2292eb45f7b09f7279fcd23a62 P 4028d6eb9da140bfb94a7fc80bc102c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4028d6eb9da140bfb94a7fc80bc102c0" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 36566 } } peers { permanent_uuid: "8e286bd098564982987f97b2de7590c8" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 49273 } } peers { permanent_uuid: "eb53f28829e04691a2601f44b09fbbe0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39440 } } I0703 01:39:32.946058 8501 heartbeater.cc:467] Master 127.5.240.254:50537 was elected leader, sending a full tablet report... I0703 01:39:32.946604 8524 ts_tablet_manager.cc:1073] T 0b854d2292eb45f7b09f7279fcd23a62 P 4028d6eb9da140bfb94a7fc80bc102c0: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s I0703 01:39:32.959578 8527 raft_consensus.cc:457] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0703 01:39:32.959679 8527 raft_consensus.cc:2730] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0: Snoozing failure detection for 1.745s (starting election) I0703 01:39:32.959770 8527 raft_consensus.cc:479] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4028d6eb9da140bfb94a7fc80bc102c0" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 36566 } } peers { permanent_uuid: "8e286bd098564982987f97b2de7590c8" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 49273 } } peers { permanent_uuid: "eb53f28829e04691a2601f44b09fbbe0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39440 } } I0703 01:39:32.960885 8527 leader_election.cc:243] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 4028d6eb9da140bfb94a7fc80bc102c0 (127.5.240.197:36566) I0703 01:39:32.961207 8527 leader_election.cc:243] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 8e286bd098564982987f97b2de7590c8 (127.5.240.196:49273) I0703 01:39:32.970793 8336 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "0b854d2292eb45f7b09f7279fcd23a62" candidate_uuid: "eb53f28829e04691a2601f44b09fbbe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e286bd098564982987f97b2de7590c8" is_pre_election: true I0703 01:39:32.971410 8336 raft_consensus.cc:2730] T 0b854d2292eb45f7b09f7279fcd23a62 P 8e286bd098564982987f97b2de7590c8: Snoozing failure detection for 1.530s (vote granted) I0703 01:39:32.971500 8336 raft_consensus.cc:2243] T 0b854d2292eb45f7b09f7279fcd23a62 P 8e286bd098564982987f97b2de7590c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb53f28829e04691a2601f44b09fbbe0 in term 0. I0703 01:39:32.971186 8458 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "0b854d2292eb45f7b09f7279fcd23a62" candidate_uuid: "eb53f28829e04691a2601f44b09fbbe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4028d6eb9da140bfb94a7fc80bc102c0" is_pre_election: true I0703 01:39:32.971823 8458 raft_consensus.cc:2730] T 0b854d2292eb45f7b09f7279fcd23a62 P 4028d6eb9da140bfb94a7fc80bc102c0: Snoozing failure detection for 1.923s (vote granted) I0703 01:39:32.971905 8458 raft_consensus.cc:2243] T 0b854d2292eb45f7b09f7279fcd23a62 P 4028d6eb9da140bfb94a7fc80bc102c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb53f28829e04691a2601f44b09fbbe0 in term 0. I0703 01:39:32.972100 8026 leader_election.cc:379] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 8e286bd098564982987f97b2de7590c8 (127.5.240.196:49273) I0703 01:39:32.972173 8026 leader_election.cc:270] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0703 01:39:32.972553 8024 leader_election.cc:379] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 4028d6eb9da140bfb94a7fc80bc102c0 (127.5.240.197:36566) I0703 01:39:32.972555 8527 raft_consensus.cc:2730] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0: Snoozing failure detection for 2.173s (election complete) I0703 01:39:32.972679 8527 raft_consensus.cc:2537] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0703 01:39:32.972797 8527 raft_consensus.cc:457] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0703 01:39:32.972854 8527 raft_consensus.cc:2730] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0: Snoozing failure detection for 1.955s (starting election) I0703 01:39:32.972911 8527 raft_consensus.cc:2782] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:39:32.974660 8527 raft_consensus.cc:479] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4028d6eb9da140bfb94a7fc80bc102c0" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 36566 } } peers { permanent_uuid: "8e286bd098564982987f97b2de7590c8" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 49273 } } peers { permanent_uuid: "eb53f28829e04691a2601f44b09fbbe0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39440 } } I0703 01:39:32.975001 8527 leader_election.cc:243] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 [CANDIDATE]: Term 1 election: Requesting vote from peer 4028d6eb9da140bfb94a7fc80bc102c0 (127.5.240.197:36566) I0703 01:39:32.975170 8527 leader_election.cc:243] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 [CANDIDATE]: Term 1 election: Requesting vote from peer 8e286bd098564982987f97b2de7590c8 (127.5.240.196:49273) I0703 01:39:32.975729 8458 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "0b854d2292eb45f7b09f7279fcd23a62" candidate_uuid: "eb53f28829e04691a2601f44b09fbbe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4028d6eb9da140bfb94a7fc80bc102c0" I0703 01:39:32.975834 8458 raft_consensus.cc:2782] T 0b854d2292eb45f7b09f7279fcd23a62 P 4028d6eb9da140bfb94a7fc80bc102c0 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:39:32.975925 8458 raft_consensus.cc:2730] T 0b854d2292eb45f7b09f7279fcd23a62 P 4028d6eb9da140bfb94a7fc80bc102c0: Snoozing failure detection for 1.783s (vote granted) I0703 01:39:32.975898 8336 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "0b854d2292eb45f7b09f7279fcd23a62" candidate_uuid: "eb53f28829e04691a2601f44b09fbbe0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e286bd098564982987f97b2de7590c8" I0703 01:39:32.976006 8336 raft_consensus.cc:2782] T 0b854d2292eb45f7b09f7279fcd23a62 P 8e286bd098564982987f97b2de7590c8 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:39:32.976094 8336 raft_consensus.cc:2730] T 0b854d2292eb45f7b09f7279fcd23a62 P 8e286bd098564982987f97b2de7590c8: Snoozing failure detection for 2.141s (vote granted) I0703 01:39:32.977216 8458 raft_consensus.cc:2243] T 0b854d2292eb45f7b09f7279fcd23a62 P 4028d6eb9da140bfb94a7fc80bc102c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb53f28829e04691a2601f44b09fbbe0 in term 1. I0703 01:39:32.977445 8336 raft_consensus.cc:2243] T 0b854d2292eb45f7b09f7279fcd23a62 P 8e286bd098564982987f97b2de7590c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb53f28829e04691a2601f44b09fbbe0 in term 1. I0703 01:39:32.977744 8024 leader_election.cc:379] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 [CANDIDATE]: Term 1 election: Vote granted by peer 4028d6eb9da140bfb94a7fc80bc102c0 (127.5.240.197:36566) I0703 01:39:32.977843 8024 leader_election.cc:270] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:39:32.977939 8026 leader_election.cc:379] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 [CANDIDATE]: Term 1 election: Vote granted by peer 8e286bd098564982987f97b2de7590c8 (127.5.240.196:49273) I0703 01:39:32.978083 8527 raft_consensus.cc:2730] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0: Snoozing failure detection for 1.869s (election complete) I0703 01:39:32.978150 8527 raft_consensus.cc:2537] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:39:32.978984 8527 raft_consensus.cc:593] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 [term 1 LEADER]: Becoming Leader. State: Replica: eb53f28829e04691a2601f44b09fbbe0, State: Running, Role: LEADER I0703 01:39:32.979372 8527 consensus_queue.cc:204] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4028d6eb9da140bfb94a7fc80bc102c0" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 36566 } } peers { permanent_uuid: "8e286bd098564982987f97b2de7590c8" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 49273 } } peers { permanent_uuid: "eb53f28829e04691a2601f44b09fbbe0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39440 } } I0703 01:39:32.983175 7849 catalog_manager.cc:3798] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 reported cstate change: term changed from 0 to 1, leader changed from to eb53f28829e04691a2601f44b09fbbe0 (127.5.240.194). New cstate: current_term: 1 leader_uuid: "eb53f28829e04691a2601f44b09fbbe0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4028d6eb9da140bfb94a7fc80bc102c0" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 36566 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e286bd098564982987f97b2de7590c8" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 49273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb53f28829e04691a2601f44b09fbbe0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39440 } health_report { overall_health: HEALTHY } } } I0703 01:39:32.995748 6083 external_mini_cluster.cc:490] 5 TS(s) registered with all masters I0703 01:39:32.996842 6083 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 8ca84f980eb34212944aaf54d04fd950 to finish bootstrapping I0703 01:39:33.003401 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver eb53f28829e04691a2601f44b09fbbe0 to finish bootstrapping I0703 01:39:33.007817 6083 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 576ad88f26964932bee2e504cb16970b to finish bootstrapping W0703 01:39:33.011771 8380 tablet.cc:1678] T 0b854d2292eb45f7b09f7279fcd23a62 P 8e286bd098564982987f97b2de7590c8: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:39:33.014101 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 8e286bd098564982987f97b2de7590c8 to finish bootstrapping I0703 01:39:33.019374 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 4028d6eb9da140bfb94a7fc80bc102c0 to finish bootstrapping I0703 01:39:33.024256 6083 kudu-admin-test.cc:487] Waiting for Master to see the current replicas... I0703 01:39:33.025492 6083 kudu-admin-test.cc:494] Tablet locations: tablet_id: "0b854d2292eb45f7b09f7279fcd23a62" replicas { ts_info { permanent_uuid: "4028d6eb9da140bfb94a7fc80bc102c0" rpc_addresses { host: "127.5.240.197" port: 36566 } } role: FOLLOWER } replicas { ts_info { permanent_uuid: "8e286bd098564982987f97b2de7590c8" rpc_addresses { host: "127.5.240.196" port: 49273 } } role: FOLLOWER } replicas { ts_info { permanent_uuid: "eb53f28829e04691a2601f44b09fbbe0" rpc_addresses { host: "127.5.240.194" port: 39440 } } role: LEADER } DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } W0703 01:39:33.099112 8502 tablet.cc:1678] T 0b854d2292eb45f7b09f7279fcd23a62 P 4028d6eb9da140bfb94a7fc80bc102c0: Can't schedule compaction. Clean time has not been advanced past its initial value. W0703 01:39:33.105149 8136 tablet.cc:1678] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:39:33.204341 8257 heartbeater.cc:467] Master 127.5.240.254:50537 was elected leader, sending a full tablet report... I0703 01:39:33.427286 8458 raft_consensus.cc:1081] T 0b854d2292eb45f7b09f7279fcd23a62 P 4028d6eb9da140bfb94a7fc80bc102c0 [term 1 FOLLOWER]: Refusing update from remote peer eb53f28829e04691a2601f44b09fbbe0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0703 01:39:33.428436 8527 consensus_queue.cc:962] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4028d6eb9da140bfb94a7fc80bc102c0" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 36566 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0703 01:39:33.434573 8336 raft_consensus.cc:1081] T 0b854d2292eb45f7b09f7279fcd23a62 P 8e286bd098564982987f97b2de7590c8 [term 1 FOLLOWER]: Refusing update from remote peer eb53f28829e04691a2601f44b09fbbe0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0703 01:39:33.435431 8538 consensus_queue.cc:962] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e286bd098564982987f97b2de7590c8" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 49273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0703 01:39:33.439653 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 8382 I0703 01:39:33.440593 8498 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:39:33.484063 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 8382 W0703 01:39:33.492280 7832 connection.cc:512] server connection from 127.5.240.197:41525 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.197:41525): Connection reset by peer (error 104) W0703 01:39:33.492302 8024 connection.cc:512] client connection to 127.5.240.197:36566 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.197:36566): Connection reset by peer (error 104) I0703 01:39:33.493016 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 8260 I0703 01:39:33.493764 8376 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:39:33.536973 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 8260 W0703 01:39:33.544534 8026 connection.cc:512] client connection to 127.5.240.196:49273 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.196:49273): Connection reset by peer (error 104) W0703 01:39:33.544750 7831 connection.cc:512] server connection from 127.5.240.196:36671 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.196:36671): Connection reset by peer (error 104) I0703 01:39:33.544888 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master pid 7823 I0703 01:39:33.545658 7879 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:39:33.608454 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master with pid 7823 W0703 01:39:33.621160 8025 connection.cc:512] client connection to 127.5.240.254:50537 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.254:50537): Connection reset by peer (error 104) W0703 01:39:33.621139 7902 connection.cc:512] client connection to 127.5.240.254:50537 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.254:50537): Connection reset by peer (error 104) W0703 01:39:33.621165 8147 connection.cc:512] client connection to 127.5.240.254:50537 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.254:50537): Connection reset by peer (error 104) W0703 01:39:33.621739 8012 heartbeater.cc:587] Failed to heartbeat to 127.5.240.254:50537: Network error: Failed to send heartbeat to master: failed to read from TLS socket (remote: 127.5.240.254:50537): Connection reset by peer (error 104) I0703 01:39:33.621819 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:50537 --webserver_interface=127.5.240.254 --webserver_port=53890 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:39:33.805371 8543 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:39:33.805596 8543 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:39:33.807904 8543 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0703 01:39:33.807940 8543 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:39:33.807966 8543 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0703 01:39:33.807997 8543 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0703 01:39:33.815495 8543 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:50537 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.5.240.254 --webserver_port=53890 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:39:33.820698 8543 master_main.cc:80] Initializing master server... I0703 01:39:33.821744 8543 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 140208us I0703 01:39:33.822818 8543 fs_manager.cc:261] Metadata directory not provided I0703 01:39:33.822875 8543 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:39:33.830279 8543 fs_manager.cc:397] Time spent opening directory manager: real 0.005s user 0.001s sys 0.003s I0703 01:39:33.831111 8543 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:39:33.831508 8543 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:39:33.835079 8543 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s I0703 01:39:33.835145 8543 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal uuid: "f23c54bf26ca4ea4a4d3f921196318d6" format_stamp: "Formatted at 2018-07-03 08:39:31 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:33.836014 8543 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:39:33.865397 8543 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:39:33.866823 8543 master_main.cc:83] Starting Master server... I0703 01:39:33.905169 8543 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.254:50537 I0703 01:39:33.905593 8543 webserver.cc:175] Starting webserver on 127.5.240.254:53890 I0703 01:39:33.905647 8543 webserver.cc:186] Document root disabled I0703 01:39:33.906798 8543 webserver.cc:313] Webserver started. Bound to: http://127.5.240.254:53890/ I0703 01:39:33.907460 8543 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigOnSingleLeader.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb I0703 01:39:33.914253 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master as pid 8543 I0703 01:39:33.914357 6083 kudu-admin-test.cc:513] Forcing unsafe config change on tserver eb53f28829e04691a2601f44b09fbbe0 I0703 01:39:33.915879 8602 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P f23c54bf26ca4ea4a4d3f921196318d6: Bootstrap starting. I0703 01:39:33.928156 8602 log.cc:525] T 00000000000000000000000000000000 P f23c54bf26ca4ea4a4d3f921196318d6: Log is configured to *not* fsync() on all Append() calls W0703 01:39:33.935608 8024 consensus_peers.cc:433] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 -> Peer 4028d6eb9da140bfb94a7fc80bc102c0 (127.5.240.197:36566): Couldn't send request to peer 4028d6eb9da140bfb94a7fc80bc102c0 for tablet 0b854d2292eb45f7b09f7279fcd23a62. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.197:36566: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. I0703 01:39:33.938146 8602 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P f23c54bf26ca4ea4a4d3f921196318d6: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=1} inserts{seen=4 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:39:33.938639 8602 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P f23c54bf26ca4ea4a4d3f921196318d6: Bootstrap complete. W0703 01:39:33.949057 8026 consensus_peers.cc:433] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 -> Peer 8e286bd098564982987f97b2de7590c8 (127.5.240.196:49273): Couldn't send request to peer 8e286bd098564982987f97b2de7590c8 for tablet 0b854d2292eb45f7b09f7279fcd23a62. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.196:49273: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. I0703 01:39:33.949139 8602 raft_consensus.cc:325] T 00000000000000000000000000000000 P f23c54bf26ca4ea4a4d3f921196318d6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f23c54bf26ca4ea4a4d3f921196318d6" member_type: VOTER } I0703 01:39:33.950887 8602 raft_consensus.cc:628] T 00000000000000000000000000000000 P f23c54bf26ca4ea4a4d3f921196318d6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f23c54bf26ca4ea4a4d3f921196318d6, State: Initialized, Role: FOLLOWER I0703 01:39:33.955310 8602 consensus_queue.cc:227] T 00000000000000000000000000000000 P f23c54bf26ca4ea4a4d3f921196318d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f23c54bf26ca4ea4a4d3f921196318d6" member_type: VOTER } I0703 01:39:33.955608 8602 raft_consensus.cc:365] T 00000000000000000000000000000000 P f23c54bf26ca4ea4a4d3f921196318d6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:39:33.955844 8602 raft_consensus.cc:457] T 00000000000000000000000000000000 P f23c54bf26ca4ea4a4d3f921196318d6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:39:33.956008 8602 raft_consensus.cc:2730] T 00000000000000000000000000000000 P f23c54bf26ca4ea4a4d3f921196318d6: Snoozing failure detection for 1.831s (starting election) I0703 01:39:33.956171 8602 raft_consensus.cc:2782] T 00000000000000000000000000000000 P f23c54bf26ca4ea4a4d3f921196318d6 [term 1 FOLLOWER]: Advancing to term 2 I0703 01:39:33.963976 8602 raft_consensus.cc:479] T 00000000000000000000000000000000 P f23c54bf26ca4ea4a4d3f921196318d6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f23c54bf26ca4ea4a4d3f921196318d6" member_type: VOTER } I0703 01:39:33.964474 8602 leader_election.cc:270] T 00000000000000000000000000000000 P f23c54bf26ca4ea4a4d3f921196318d6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0703 01:39:33.966372 8610 raft_consensus.cc:2730] T 00000000000000000000000000000000 P f23c54bf26ca4ea4a4d3f921196318d6: Snoozing failure detection for 1.748s (election complete) I0703 01:39:33.966560 8610 raft_consensus.cc:2537] T 00000000000000000000000000000000 P f23c54bf26ca4ea4a4d3f921196318d6 [term 2 FOLLOWER]: Leader election won for term 2 I0703 01:39:33.970140 8610 raft_consensus.cc:593] T 00000000000000000000000000000000 P f23c54bf26ca4ea4a4d3f921196318d6 [term 2 LEADER]: Becoming Leader. State: Replica: f23c54bf26ca4ea4a4d3f921196318d6, State: Running, Role: LEADER I0703 01:39:33.970731 8610 consensus_queue.cc:204] T 00000000000000000000000000000000 P f23c54bf26ca4ea4a4d3f921196318d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f23c54bf26ca4ea4a4d3f921196318d6" member_type: VOTER } W0703 01:39:33.971843 8135 heartbeater.cc:587] Failed to heartbeat to 127.5.240.254:50537: Service unavailable: Catalog manager is not initialized. State: Starting I0703 01:39:33.973806 8602 sys_catalog.cc:337] T 00000000000000000000000000000000 P f23c54bf26ca4ea4a4d3f921196318d6 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 2 leader_uuid: "f23c54bf26ca4ea4a4d3f921196318d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f23c54bf26ca4ea4a4d3f921196318d6" member_type: VOTER } } I0703 01:39:33.973603 8611 sys_catalog.cc:337] T 00000000000000000000000000000000 P f23c54bf26ca4ea4a4d3f921196318d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f23c54bf26ca4ea4a4d3f921196318d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f23c54bf26ca4ea4a4d3f921196318d6" member_type: VOTER } } I0703 01:39:33.974319 8602 sys_catalog.cc:340] T 00000000000000000000000000000000 P f23c54bf26ca4ea4a4d3f921196318d6 [sys.catalog]: This master's current role is: LEADER I0703 01:39:33.973803 8612 sys_catalog.cc:337] T 00000000000000000000000000000000 P f23c54bf26ca4ea4a4d3f921196318d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f23c54bf26ca4ea4a4d3f921196318d6. Latest consensus state: current_term: 2 leader_uuid: "f23c54bf26ca4ea4a4d3f921196318d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f23c54bf26ca4ea4a4d3f921196318d6" member_type: VOTER } } I0703 01:39:33.974462 8611 sys_catalog.cc:340] T 00000000000000000000000000000000 P f23c54bf26ca4ea4a4d3f921196318d6 [sys.catalog]: This master's current role is: LEADER I0703 01:39:33.974704 8612 sys_catalog.cc:340] T 00000000000000000000000000000000 P f23c54bf26ca4ea4a4d3f921196318d6 [sys.catalog]: This master's current role is: LEADER I0703 01:39:33.976382 8602 sys_catalog.cc:424] T 00000000000000000000000000000000 P f23c54bf26ca4ea4a4d3f921196318d6 [sys.catalog]: configured and running, proceeding with master startup. I0703 01:39:33.976644 8614 catalog_manager.cc:1006] Loading table and tablet metadata into memory... I0703 01:39:33.977999 8543 master_main.cc:86] Master server successfully started. I0703 01:39:33.982606 8614 catalog_manager.cc:323] Loaded metadata for table TestTable [id=3578883b7bc34387a7c9a55537c46eed] I0703 01:39:33.984692 8614 catalog_manager.cc:377] Loaded metadata for tablet 0b854d2292eb45f7b09f7279fcd23a62 (table TestTable [id=3578883b7bc34387a7c9a55537c46eed]) I0703 01:39:33.984885 8614 catalog_manager.cc:1017] Initializing Kudu internal certificate authority... I0703 01:39:33.990716 8614 catalog_manager.cc:1026] Loading token signing keys... I0703 01:39:33.992249 8614 catalog_manager.cc:4113] T 00000000000000000000000000000000 P f23c54bf26ca4ea4a4d3f921196318d6: Loaded TSK: 0 W0703 01:39:34.140872 8603 flags.cc:406] Enabled unsafe flag: --never_fsync=true I0703 01:39:34.192358 8092 tablet_service.cc:1077] Received UnsafeChangeConfig RPC: dest_uuid: "eb53f28829e04691a2601f44b09fbbe0" tablet_id: "0b854d2292eb45f7b09f7279fcd23a62" caller_id: "kudu-tools" new_config { peers { permanent_uuid: "eb53f28829e04691a2601f44b09fbbe0" } } from {username='jenkins'} at 127.5.240.194:42962 W0703 01:39:34.192916 8092 raft_consensus.cc:2017] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 [term 1 LEADER]: PROCEEDING WITH UNSAFE CONFIG CHANGE ON THIS SERVER, COMMITTED CONFIG: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4028d6eb9da140bfb94a7fc80bc102c0" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 36566 } } peers { permanent_uuid: "8e286bd098564982987f97b2de7590c8" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 49273 } } peers { permanent_uuid: "eb53f28829e04691a2601f44b09fbbe0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39440 } }NEW CONFIG: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb53f28829e04691a2601f44b09fbbe0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39440 } } unsafe_config_change: true I0703 01:39:34.193400 8092 raft_consensus.cc:2777] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 [term 1 LEADER]: Stepping down as leader of term 1 I0703 01:39:34.193449 8092 raft_consensus.cc:628] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: eb53f28829e04691a2601f44b09fbbe0, State: Running, Role: LEADER I0703 01:39:34.193744 8092 consensus_queue.cc:227] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4028d6eb9da140bfb94a7fc80bc102c0" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 36566 } } peers { permanent_uuid: "8e286bd098564982987f97b2de7590c8" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 49273 } } peers { permanent_uuid: "eb53f28829e04691a2601f44b09fbbe0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39440 } } I0703 01:39:34.193950 8092 consensus_peers.cc:455] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 -> Peer 8e286bd098564982987f97b2de7590c8 (127.5.240.196:49273): Closing peer: 8e286bd098564982987f97b2de7590c8 I0703 01:39:34.194016 8092 consensus_peers.cc:455] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 -> Peer 4028d6eb9da140bfb94a7fc80bc102c0 (127.5.240.197:36566): Closing peer: 4028d6eb9da140bfb94a7fc80bc102c0 I0703 01:39:34.194258 8092 raft_consensus.cc:2782] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 [term 1 FOLLOWER]: Advancing to term 2 I0703 01:39:34.199509 8569 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "eb53f28829e04691a2601f44b09fbbe0" instance_seqno: 1530607171833546) as {username='jenkins'} at 127.5.240.194:41798; Asking this server to re-register. I0703 01:39:34.200270 8135 heartbeater.cc:426] Registering TS with master... I0703 01:39:34.200412 8135 heartbeater.cc:475] Master 127.5.240.254:50537 requested a full tablet report, sending... I0703 01:39:34.201885 8569 ts_manager.cc:98] Registered new tserver with Master: eb53f28829e04691a2601f44b09fbbe0 (127.5.240.194:39440) I0703 01:39:34.211539 8569 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "576ad88f26964932bee2e504cb16970b" instance_seqno: 1530607172166945) as {username='jenkins'} at 127.5.240.195:39668; Asking this server to re-register. I0703 01:39:34.212255 8257 heartbeater.cc:426] Registering TS with master... I0703 01:39:34.212411 8257 heartbeater.cc:475] Master 127.5.240.254:50537 requested a full tablet report, sending... I0703 01:39:34.213387 8569 ts_manager.cc:98] Registered new tserver with Master: 576ad88f26964932bee2e504cb16970b (127.5.240.195:48979) I0703 01:39:34.628490 8012 heartbeater.cc:346] Connected to a master server at 127.5.240.254:50537 I0703 01:39:34.629278 8569 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "8ca84f980eb34212944aaf54d04fd950" instance_seqno: 1530607171495946) as {username='jenkins'} at 127.5.240.193:42231; Asking this server to re-register. I0703 01:39:34.629904 8012 heartbeater.cc:426] Registering TS with master... I0703 01:39:34.630028 8012 heartbeater.cc:475] Master 127.5.240.254:50537 requested a full tablet report, sending... I0703 01:39:34.630731 8569 ts_manager.cc:98] Registered new tserver with Master: 8ca84f980eb34212944aaf54d04fd950 (127.5.240.193:42094) I0703 01:39:35.698127 8636 raft_consensus.cc:457] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader kudu-tools) I0703 01:39:35.698215 8636 raft_consensus.cc:2730] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0: Snoozing failure detection for 2.036s (starting election) I0703 01:39:35.698320 8636 raft_consensus.cc:479] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb53f28829e04691a2601f44b09fbbe0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39440 } } unsafe_config_change: true I0703 01:39:35.698523 8636 leader_election.cc:270] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. I0703 01:39:35.698891 8636 raft_consensus.cc:2730] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0: Snoozing failure detection for 2.056s (election complete) I0703 01:39:35.698951 8636 raft_consensus.cc:2537] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 [term 2 FOLLOWER]: Leader pre-election won for term 3 I0703 01:39:35.699012 8636 raft_consensus.cc:457] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader kudu-tools) I0703 01:39:35.699067 8636 raft_consensus.cc:2730] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0: Snoozing failure detection for 1.843s (starting election) I0703 01:39:35.699132 8636 raft_consensus.cc:2782] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 [term 2 FOLLOWER]: Advancing to term 3 I0703 01:39:35.700508 8636 raft_consensus.cc:479] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb53f28829e04691a2601f44b09fbbe0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39440 } } unsafe_config_change: true I0703 01:39:35.700667 8636 leader_election.cc:270] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. I0703 01:39:35.700847 8637 raft_consensus.cc:2730] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0: Snoozing failure detection for 2.174s (election complete) I0703 01:39:35.700912 8637 raft_consensus.cc:2537] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 [term 3 FOLLOWER]: Leader election won for term 3 I0703 01:39:35.701019 8637 raft_consensus.cc:593] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 [term 3 LEADER]: Becoming Leader. State: Replica: eb53f28829e04691a2601f44b09fbbe0, State: Running, Role: LEADER I0703 01:39:35.701262 8637 consensus_queue.cc:204] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 0, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb53f28829e04691a2601f44b09fbbe0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39440 } } unsafe_config_change: true I0703 01:39:35.702311 8637 raft_consensus.cc:2684] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 [term 3 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, VOTER 4028d6eb9da140bfb94a7fc80bc102c0 (127.5.240.197) evicted, VOTER 8e286bd098564982987f97b2de7590c8 (127.5.240.196) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb53f28829e04691a2601f44b09fbbe0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39440 } } unsafe_config_change: true } I0703 01:39:35.703125 8569 catalog_manager.cc:3798] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 reported cstate change: term changed from 1 to 3, now has a pending config: VOTER eb53f28829e04691a2601f44b09fbbe0 (127.5.240.194). New cstate: current_term: 3 leader_uuid: "eb53f28829e04691a2601f44b09fbbe0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4028d6eb9da140bfb94a7fc80bc102c0" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 36566 } } peers { permanent_uuid: "8e286bd098564982987f97b2de7590c8" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 49273 } } peers { permanent_uuid: "eb53f28829e04691a2601f44b09fbbe0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39440 } health_report { overall_health: HEALTHY } } } pending_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb53f28829e04691a2601f44b09fbbe0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39440 } } unsafe_config_change: true } I0703 01:39:35.712800 8569 catalog_manager.cc:3798] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 reported cstate change: config changed from index -1 to 2, VOTER 4028d6eb9da140bfb94a7fc80bc102c0 (127.5.240.197) evicted, VOTER 8e286bd098564982987f97b2de7590c8 (127.5.240.196) evicted, no longer has a pending config: VOTER eb53f28829e04691a2601f44b09fbbe0 (127.5.240.194). New cstate: current_term: 3 leader_uuid: "eb53f28829e04691a2601f44b09fbbe0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eb53f28829e04691a2601f44b09fbbe0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39440 } health_report { overall_health: HEALTHY } } unsafe_config_change: true } I0703 01:39:35.714784 8569 catalog_manager.cc:2903] Scheduling retry of 0b854d2292eb45f7b09f7279fcd23a62 Delete Tablet RPC for TS=4028d6eb9da140bfb94a7fc80bc102c0 with a delay of 26 ms (attempt = 1) W0703 01:39:35.714882 8569 catalog_manager.cc:3957] Failed to send 0b854d2292eb45f7b09f7279fcd23a62 Delete Tablet RPC for TS=4028d6eb9da140bfb94a7fc80bc102c0: Not found: failed to reset TS proxy: Could not find TS for UUID 4028d6eb9da140bfb94a7fc80bc102c0 I0703 01:39:35.714994 8569 catalog_manager.cc:2903] Scheduling retry of 0b854d2292eb45f7b09f7279fcd23a62 Delete Tablet RPC for TS=8e286bd098564982987f97b2de7590c8 with a delay of 21 ms (attempt = 1) W0703 01:39:35.715054 8569 catalog_manager.cc:3957] Failed to send 0b854d2292eb45f7b09f7279fcd23a62 Delete Tablet RPC for TS=8e286bd098564982987f97b2de7590c8: Not found: failed to reset TS proxy: Could not find TS for UUID 8e286bd098564982987f97b2de7590c8 I0703 01:39:35.715417 8569 catalog_manager.cc:3437] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 0b854d2292eb45f7b09f7279fcd23a62 (attempt 1) I0703 01:39:35.721354 8092 consensus_queue.cc:204] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 0, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb53f28829e04691a2601f44b09fbbe0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39440 } } peers { permanent_uuid: "8ca84f980eb34212944aaf54d04fd950" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 42094 } attrs { promote: true } } unsafe_config_change: true I0703 01:39:35.722193 8637 raft_consensus.cc:2684] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index 2 to 4, NON_VOTER 8ca84f980eb34212944aaf54d04fd950 (127.5.240.193) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb53f28829e04691a2601f44b09fbbe0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39440 } } peers { permanent_uuid: "8ca84f980eb34212944aaf54d04fd950" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 42094 } attrs { promote: true } } unsafe_config_change: true } I0703 01:39:35.724849 8552 catalog_manager.cc:3352] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0b854d2292eb45f7b09f7279fcd23a62 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0703 01:39:35.725631 8569 catalog_manager.cc:3798] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 reported cstate change: config changed from index 2 to 4, NON_VOTER 8ca84f980eb34212944aaf54d04fd950 (127.5.240.193) added. New cstate: current_term: 3 leader_uuid: "eb53f28829e04691a2601f44b09fbbe0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eb53f28829e04691a2601f44b09fbbe0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39440 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ca84f980eb34212944aaf54d04fd950" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 42094 } attrs { promote: true } health_report { overall_health: UNKNOWN } } unsafe_config_change: true } I0703 01:39:35.727453 8569 catalog_manager.cc:3437] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 0b854d2292eb45f7b09f7279fcd23a62 (attempt 1) I0703 01:39:35.728392 8092 consensus_peers.cc:455] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 -> Peer 8ca84f980eb34212944aaf54d04fd950 (127.5.240.193:42094): Closing peer: 8ca84f980eb34212944aaf54d04fd950 I0703 01:39:35.728587 8092 consensus_queue.cc:204] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 0, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb53f28829e04691a2601f44b09fbbe0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39440 } } peers { permanent_uuid: "8ca84f980eb34212944aaf54d04fd950" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 42094 } attrs { promote: true } } peers { permanent_uuid: "576ad88f26964932bee2e504cb16970b" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 48979 } attrs { promote: true } } unsafe_config_change: true I0703 01:39:35.730067 8637 raft_consensus.cc:2684] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, NON_VOTER 576ad88f26964932bee2e504cb16970b (127.5.240.195) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb53f28829e04691a2601f44b09fbbe0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39440 } } peers { permanent_uuid: "8ca84f980eb34212944aaf54d04fd950" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 42094 } attrs { promote: true } } peers { permanent_uuid: "576ad88f26964932bee2e504cb16970b" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 48979 } attrs { promote: true } } unsafe_config_change: true } W0703 01:39:35.731178 8024 consensus_peers.cc:433] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 -> Peer 8ca84f980eb34212944aaf54d04fd950 (127.5.240.193:42094): Couldn't send request to peer 8ca84f980eb34212944aaf54d04fd950 for tablet 0b854d2292eb45f7b09f7279fcd23a62. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0b854d2292eb45f7b09f7279fcd23a62. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:39:35.732616 8552 catalog_manager.cc:3352] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0b854d2292eb45f7b09f7279fcd23a62 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0703 01:39:35.733269 8569 catalog_manager.cc:3798] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 reported cstate change: config changed from index 4 to 5, NON_VOTER 576ad88f26964932bee2e504cb16970b (127.5.240.195) added. New cstate: current_term: 3 leader_uuid: "eb53f28829e04691a2601f44b09fbbe0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb53f28829e04691a2601f44b09fbbe0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39440 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ca84f980eb34212944aaf54d04fd950" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 42094 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "576ad88f26964932bee2e504cb16970b" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 48979 } attrs { promote: true } health_report { overall_health: UNKNOWN } } unsafe_config_change: true } I0703 01:39:35.737080 8552 catalog_manager.cc:2903] Scheduling retry of 0b854d2292eb45f7b09f7279fcd23a62 Delete Tablet RPC for TS=8e286bd098564982987f97b2de7590c8 with a delay of 50 ms (attempt = 2) W0703 01:39:35.737146 8552 catalog_manager.cc:2922] Async tablet task 0b854d2292eb45f7b09f7279fcd23a62 Delete Tablet RPC for TS=8e286bd098564982987f97b2de7590c8 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 8e286bd098564982987f97b2de7590c8 W0703 01:39:35.737759 8026 consensus_peers.cc:433] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 -> Peer 576ad88f26964932bee2e504cb16970b (127.5.240.195:48979): Couldn't send request to peer 576ad88f26964932bee2e504cb16970b for tablet 0b854d2292eb45f7b09f7279fcd23a62. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0b854d2292eb45f7b09f7279fcd23a62. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:39:35.741215 8550 catalog_manager.cc:2903] Scheduling retry of 0b854d2292eb45f7b09f7279fcd23a62 Delete Tablet RPC for TS=4028d6eb9da140bfb94a7fc80bc102c0 with a delay of 43 ms (attempt = 2) W0703 01:39:35.741276 8550 catalog_manager.cc:2922] Async tablet task 0b854d2292eb45f7b09f7279fcd23a62 Delete Tablet RPC for TS=4028d6eb9da140bfb94a7fc80bc102c0 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 4028d6eb9da140bfb94a7fc80bc102c0 I0703 01:39:35.784855 8550 catalog_manager.cc:2903] Scheduling retry of 0b854d2292eb45f7b09f7279fcd23a62 Delete Tablet RPC for TS=4028d6eb9da140bfb94a7fc80bc102c0 with a delay of 106 ms (attempt = 3) W0703 01:39:35.784914 8550 catalog_manager.cc:2922] Async tablet task 0b854d2292eb45f7b09f7279fcd23a62 Delete Tablet RPC for TS=4028d6eb9da140bfb94a7fc80bc102c0 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 4028d6eb9da140bfb94a7fc80bc102c0 I0703 01:39:35.787642 8552 catalog_manager.cc:2903] Scheduling retry of 0b854d2292eb45f7b09f7279fcd23a62 Delete Tablet RPC for TS=8e286bd098564982987f97b2de7590c8 with a delay of 96 ms (attempt = 3) W0703 01:39:35.787703 8552 catalog_manager.cc:2922] Async tablet task 0b854d2292eb45f7b09f7279fcd23a62 Delete Tablet RPC for TS=8e286bd098564982987f97b2de7590c8 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 8e286bd098564982987f97b2de7590c8 I0703 01:39:35.884378 8552 catalog_manager.cc:2903] Scheduling retry of 0b854d2292eb45f7b09f7279fcd23a62 Delete Tablet RPC for TS=8e286bd098564982987f97b2de7590c8 with a delay of 174 ms (attempt = 4) W0703 01:39:35.884441 8552 catalog_manager.cc:2922] Async tablet task 0b854d2292eb45f7b09f7279fcd23a62 Delete Tablet RPC for TS=8e286bd098564982987f97b2de7590c8 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 8e286bd098564982987f97b2de7590c8 I0703 01:39:35.891548 8550 catalog_manager.cc:2903] Scheduling retry of 0b854d2292eb45f7b09f7279fcd23a62 Delete Tablet RPC for TS=4028d6eb9da140bfb94a7fc80bc102c0 with a delay of 157 ms (attempt = 4) W0703 01:39:35.891603 8550 catalog_manager.cc:2922] Async tablet task 0b854d2292eb45f7b09f7279fcd23a62 Delete Tablet RPC for TS=4028d6eb9da140bfb94a7fc80bc102c0 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 4028d6eb9da140bfb94a7fc80bc102c0 I0703 01:39:36.049636 8550 catalog_manager.cc:2903] Scheduling retry of 0b854d2292eb45f7b09f7279fcd23a62 Delete Tablet RPC for TS=4028d6eb9da140bfb94a7fc80bc102c0 with a delay of 277 ms (attempt = 5) W0703 01:39:36.049713 8550 catalog_manager.cc:2922] Async tablet task 0b854d2292eb45f7b09f7279fcd23a62 Delete Tablet RPC for TS=4028d6eb9da140bfb94a7fc80bc102c0 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 4028d6eb9da140bfb94a7fc80bc102c0 I0703 01:39:36.059417 8552 catalog_manager.cc:2903] Scheduling retry of 0b854d2292eb45f7b09f7279fcd23a62 Delete Tablet RPC for TS=8e286bd098564982987f97b2de7590c8 with a delay of 276 ms (attempt = 5) W0703 01:39:36.059475 8552 catalog_manager.cc:2922] Async tablet task 0b854d2292eb45f7b09f7279fcd23a62 Delete Tablet RPC for TS=8e286bd098564982987f97b2de7590c8 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 8e286bd098564982987f97b2de7590c8 I0703 01:39:36.222609 8649 ts_tablet_manager.cc:669] T 0b854d2292eb45f7b09f7279fcd23a62 P 8ca84f980eb34212944aaf54d04fd950: Initiating tablet copy from peer eb53f28829e04691a2601f44b09fbbe0 (127.5.240.194:39440) I0703 01:39:36.222869 8649 tablet_copy_client.cc:239] T 0b854d2292eb45f7b09f7279fcd23a62 P 8ca84f980eb34212944aaf54d04fd950: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.194:39440 I0703 01:39:36.228559 8112 tablet_copy_service.cc:135] P eb53f28829e04691a2601f44b09fbbe0: Received BeginTabletCopySession request for tablet 0b854d2292eb45f7b09f7279fcd23a62 from peer 8ca84f980eb34212944aaf54d04fd950 ({username='jenkins'} at 127.5.240.193:48685) I0703 01:39:36.228714 8112 tablet_copy_service.cc:156] P eb53f28829e04691a2601f44b09fbbe0: Beginning new tablet copy session on tablet 0b854d2292eb45f7b09f7279fcd23a62 from peer 8ca84f980eb34212944aaf54d04fd950 at {username='jenkins'} at 127.5.240.193:48685: session id = 8ca84f980eb34212944aaf54d04fd950-0b854d2292eb45f7b09f7279fcd23a62 I0703 01:39:36.230721 8112 tablet_copy_source_session.cc:202] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:39:36.233369 8649 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 0b854d2292eb45f7b09f7279fcd23a62. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:39:36.242825 8649 ts_tablet_manager.cc:1173] T 0b854d2292eb45f7b09f7279fcd23a62 P 8ca84f980eb34212944aaf54d04fd950: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:39:36.243054 8649 tablet_copy_client.cc:566] T 0b854d2292eb45f7b09f7279fcd23a62 P 8ca84f980eb34212944aaf54d04fd950: tablet copy: Starting download of 0 data blocks... I0703 01:39:36.243221 8649 tablet_copy_client.cc:529] T 0b854d2292eb45f7b09f7279fcd23a62 P 8ca84f980eb34212944aaf54d04fd950: tablet copy: Starting download of 1 WAL segments... I0703 01:39:36.244771 8649 tablet_copy_client.cc:405] T 0b854d2292eb45f7b09f7279fcd23a62 P 8ca84f980eb34212944aaf54d04fd950: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:39:36.246387 8649 ts_tablet_manager.cc:1028] T 0b854d2292eb45f7b09f7279fcd23a62 P 8ca84f980eb34212944aaf54d04fd950: Bootstrapping tablet I0703 01:39:36.246628 8649 tablet_bootstrap.cc:427] T 0b854d2292eb45f7b09f7279fcd23a62 P 8ca84f980eb34212944aaf54d04fd950: Bootstrap starting. I0703 01:39:36.250474 8649 log.cc:525] T 0b854d2292eb45f7b09f7279fcd23a62 P 8ca84f980eb34212944aaf54d04fd950: Log is configured to *not* fsync() on all Append() calls I0703 01:39:36.254240 8649 tablet_bootstrap.cc:427] T 0b854d2292eb45f7b09f7279fcd23a62 P 8ca84f980eb34212944aaf54d04fd950: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:39:36.254567 8649 tablet_bootstrap.cc:427] T 0b854d2292eb45f7b09f7279fcd23a62 P 8ca84f980eb34212944aaf54d04fd950: Bootstrap complete. I0703 01:39:36.254807 8649 ts_tablet_manager.cc:1045] T 0b854d2292eb45f7b09f7279fcd23a62 P 8ca84f980eb34212944aaf54d04fd950: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.002s I0703 01:39:36.259673 8649 raft_consensus.cc:325] T 0b854d2292eb45f7b09f7279fcd23a62 P 8ca84f980eb34212944aaf54d04fd950 [term 3 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb53f28829e04691a2601f44b09fbbe0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39440 } } peers { permanent_uuid: "8ca84f980eb34212944aaf54d04fd950" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 42094 } attrs { promote: true } } peers { permanent_uuid: "576ad88f26964932bee2e504cb16970b" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 48979 } attrs { promote: true } } unsafe_config_change: true I0703 01:39:36.259969 8649 raft_consensus.cc:628] T 0b854d2292eb45f7b09f7279fcd23a62 P 8ca84f980eb34212944aaf54d04fd950 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 8ca84f980eb34212944aaf54d04fd950, State: Initialized, Role: LEARNER I0703 01:39:36.260234 8649 consensus_queue.cc:227] T 0b854d2292eb45f7b09f7279fcd23a62 P 8ca84f980eb34212944aaf54d04fd950 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb53f28829e04691a2601f44b09fbbe0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39440 } } peers { permanent_uuid: "8ca84f980eb34212944aaf54d04fd950" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 42094 } attrs { promote: true } } peers { permanent_uuid: "576ad88f26964932bee2e504cb16970b" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 48979 } attrs { promote: true } } unsafe_config_change: true I0703 01:39:36.262159 8649 ts_tablet_manager.cc:1073] T 0b854d2292eb45f7b09f7279fcd23a62 P 8ca84f980eb34212944aaf54d04fd950: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s I0703 01:39:36.263032 8112 tablet_copy_service.cc:325] P eb53f28829e04691a2601f44b09fbbe0: Request end of tablet copy session 8ca84f980eb34212944aaf54d04fd950-0b854d2292eb45f7b09f7279fcd23a62 received from {username='jenkins'} at 127.5.240.193:48685 I0703 01:39:36.263115 8112 tablet_copy_service.cc:413] P eb53f28829e04691a2601f44b09fbbe0: ending tablet copy session 8ca84f980eb34212944aaf54d04fd950-0b854d2292eb45f7b09f7279fcd23a62 on tablet 0b854d2292eb45f7b09f7279fcd23a62 with peer 8ca84f980eb34212944aaf54d04fd950 W0703 01:39:36.273754 8013 tablet.cc:1678] T 0b854d2292eb45f7b09f7279fcd23a62 P 8ca84f980eb34212944aaf54d04fd950: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:39:36.327164 8550 catalog_manager.cc:2903] Scheduling retry of 0b854d2292eb45f7b09f7279fcd23a62 Delete Tablet RPC for TS=4028d6eb9da140bfb94a7fc80bc102c0 with a delay of 546 ms (attempt = 6) W0703 01:39:36.327230 8550 catalog_manager.cc:2922] Async tablet task 0b854d2292eb45f7b09f7279fcd23a62 Delete Tablet RPC for TS=4028d6eb9da140bfb94a7fc80bc102c0 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 4028d6eb9da140bfb94a7fc80bc102c0 I0703 01:39:36.328922 8654 ts_tablet_manager.cc:669] T 0b854d2292eb45f7b09f7279fcd23a62 P 576ad88f26964932bee2e504cb16970b: Initiating tablet copy from peer eb53f28829e04691a2601f44b09fbbe0 (127.5.240.194:39440) I0703 01:39:36.329207 8654 tablet_copy_client.cc:239] T 0b854d2292eb45f7b09f7279fcd23a62 P 576ad88f26964932bee2e504cb16970b: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.194:39440 I0703 01:39:36.333835 8112 tablet_copy_service.cc:135] P eb53f28829e04691a2601f44b09fbbe0: Received BeginTabletCopySession request for tablet 0b854d2292eb45f7b09f7279fcd23a62 from peer 576ad88f26964932bee2e504cb16970b ({username='jenkins'} at 127.5.240.195:35086) I0703 01:39:36.333961 8112 tablet_copy_service.cc:156] P eb53f28829e04691a2601f44b09fbbe0: Beginning new tablet copy session on tablet 0b854d2292eb45f7b09f7279fcd23a62 from peer 576ad88f26964932bee2e504cb16970b at {username='jenkins'} at 127.5.240.195:35086: session id = 576ad88f26964932bee2e504cb16970b-0b854d2292eb45f7b09f7279fcd23a62 I0703 01:39:36.335640 8112 tablet_copy_source_session.cc:202] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:39:36.336493 8552 catalog_manager.cc:2903] Scheduling retry of 0b854d2292eb45f7b09f7279fcd23a62 Delete Tablet RPC for TS=8e286bd098564982987f97b2de7590c8 with a delay of 534 ms (attempt = 6) W0703 01:39:36.336563 8552 catalog_manager.cc:2922] Async tablet task 0b854d2292eb45f7b09f7279fcd23a62 Delete Tablet RPC for TS=8e286bd098564982987f97b2de7590c8 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 8e286bd098564982987f97b2de7590c8 I0703 01:39:36.338165 8654 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 0b854d2292eb45f7b09f7279fcd23a62. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:39:36.348556 8654 ts_tablet_manager.cc:1173] T 0b854d2292eb45f7b09f7279fcd23a62 P 576ad88f26964932bee2e504cb16970b: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:39:36.348848 8654 tablet_copy_client.cc:566] T 0b854d2292eb45f7b09f7279fcd23a62 P 576ad88f26964932bee2e504cb16970b: tablet copy: Starting download of 0 data blocks... I0703 01:39:36.349007 8654 tablet_copy_client.cc:529] T 0b854d2292eb45f7b09f7279fcd23a62 P 576ad88f26964932bee2e504cb16970b: tablet copy: Starting download of 1 WAL segments... I0703 01:39:36.350404 8654 tablet_copy_client.cc:405] T 0b854d2292eb45f7b09f7279fcd23a62 P 576ad88f26964932bee2e504cb16970b: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:39:36.352134 8654 ts_tablet_manager.cc:1028] T 0b854d2292eb45f7b09f7279fcd23a62 P 576ad88f26964932bee2e504cb16970b: Bootstrapping tablet I0703 01:39:36.352351 8654 tablet_bootstrap.cc:427] T 0b854d2292eb45f7b09f7279fcd23a62 P 576ad88f26964932bee2e504cb16970b: Bootstrap starting. I0703 01:39:36.356315 8654 log.cc:525] T 0b854d2292eb45f7b09f7279fcd23a62 P 576ad88f26964932bee2e504cb16970b: Log is configured to *not* fsync() on all Append() calls I0703 01:39:36.360220 8654 tablet_bootstrap.cc:427] T 0b854d2292eb45f7b09f7279fcd23a62 P 576ad88f26964932bee2e504cb16970b: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:39:36.360958 8654 tablet_bootstrap.cc:427] T 0b854d2292eb45f7b09f7279fcd23a62 P 576ad88f26964932bee2e504cb16970b: Bootstrap complete. I0703 01:39:36.361207 8654 ts_tablet_manager.cc:1045] T 0b854d2292eb45f7b09f7279fcd23a62 P 576ad88f26964932bee2e504cb16970b: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.004s I0703 01:39:36.366242 8654 raft_consensus.cc:325] T 0b854d2292eb45f7b09f7279fcd23a62 P 576ad88f26964932bee2e504cb16970b [term 3 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb53f28829e04691a2601f44b09fbbe0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39440 } } peers { permanent_uuid: "8ca84f980eb34212944aaf54d04fd950" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 42094 } attrs { promote: true } } peers { permanent_uuid: "576ad88f26964932bee2e504cb16970b" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 48979 } attrs { promote: true } } unsafe_config_change: true I0703 01:39:36.366535 8654 raft_consensus.cc:628] T 0b854d2292eb45f7b09f7279fcd23a62 P 576ad88f26964932bee2e504cb16970b [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 576ad88f26964932bee2e504cb16970b, State: Initialized, Role: LEARNER I0703 01:39:36.366804 8654 consensus_queue.cc:227] T 0b854d2292eb45f7b09f7279fcd23a62 P 576ad88f26964932bee2e504cb16970b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eb53f28829e04691a2601f44b09fbbe0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39440 } } peers { permanent_uuid: "8ca84f980eb34212944aaf54d04fd950" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 42094 } attrs { promote: true } } peers { permanent_uuid: "576ad88f26964932bee2e504cb16970b" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 48979 } attrs { promote: true } } unsafe_config_change: true I0703 01:39:36.368263 8654 ts_tablet_manager.cc:1073] T 0b854d2292eb45f7b09f7279fcd23a62 P 576ad88f26964932bee2e504cb16970b: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s I0703 01:39:36.369180 8112 tablet_copy_service.cc:325] P eb53f28829e04691a2601f44b09fbbe0: Request end of tablet copy session 576ad88f26964932bee2e504cb16970b-0b854d2292eb45f7b09f7279fcd23a62 received from {username='jenkins'} at 127.5.240.195:35086 I0703 01:39:36.369252 8112 tablet_copy_service.cc:413] P eb53f28829e04691a2601f44b09fbbe0: ending tablet copy session 576ad88f26964932bee2e504cb16970b-0b854d2292eb45f7b09f7279fcd23a62 on tablet 0b854d2292eb45f7b09f7279fcd23a62 with peer 576ad88f26964932bee2e504cb16970b W0703 01:39:36.440201 8258 tablet.cc:1678] T 0b854d2292eb45f7b09f7279fcd23a62 P 576ad88f26964932bee2e504cb16970b: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:39:36.746564 7969 raft_consensus.cc:1026] T 0b854d2292eb45f7b09f7279fcd23a62 P 8ca84f980eb34212944aaf54d04fd950 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.4->[3.5-3.5] Dedup: 3.5->[] I0703 01:39:36.871636 8552 catalog_manager.cc:2903] Scheduling retry of 0b854d2292eb45f7b09f7279fcd23a62 Delete Tablet RPC for TS=8e286bd098564982987f97b2de7590c8 with a delay of 1051 ms (attempt = 7) W0703 01:39:36.871711 8552 catalog_manager.cc:2922] Async tablet task 0b854d2292eb45f7b09f7279fcd23a62 Delete Tablet RPC for TS=8e286bd098564982987f97b2de7590c8 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 8e286bd098564982987f97b2de7590c8 I0703 01:39:36.873950 8214 raft_consensus.cc:1026] T 0b854d2292eb45f7b09f7279fcd23a62 P 576ad88f26964932bee2e504cb16970b [term 3 LEARNER]: Deduplicated request from leader. Original: 3.4->[3.5-3.5] Dedup: 3.5->[] I0703 01:39:36.874092 8550 catalog_manager.cc:2903] Scheduling retry of 0b854d2292eb45f7b09f7279fcd23a62 Delete Tablet RPC for TS=4028d6eb9da140bfb94a7fc80bc102c0 with a delay of 1058 ms (attempt = 7) W0703 01:39:36.874177 8550 catalog_manager.cc:2922] Async tablet task 0b854d2292eb45f7b09f7279fcd23a62 Delete Tablet RPC for TS=4028d6eb9da140bfb94a7fc80bc102c0 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 4028d6eb9da140bfb94a7fc80bc102c0 I0703 01:39:37.239085 8659 raft_consensus.cc:890] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0: attempting to promote NON_VOTER 8ca84f980eb34212944aaf54d04fd950 to VOTER I0703 01:39:37.239401 8659 consensus_peers.cc:455] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 -> Peer 576ad88f26964932bee2e504cb16970b (127.5.240.195:48979): Closing peer: 576ad88f26964932bee2e504cb16970b I0703 01:39:37.239470 8659 consensus_peers.cc:455] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 -> Peer 8ca84f980eb34212944aaf54d04fd950 (127.5.240.193:42094): Closing peer: 8ca84f980eb34212944aaf54d04fd950 I0703 01:39:37.239715 8659 consensus_queue.cc:204] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 0, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb53f28829e04691a2601f44b09fbbe0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39440 } } peers { permanent_uuid: "8ca84f980eb34212944aaf54d04fd950" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42094 } attrs { promote: false } } peers { permanent_uuid: "576ad88f26964932bee2e504cb16970b" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 48979 } attrs { promote: true } } unsafe_config_change: true I0703 01:39:37.241788 7969 raft_consensus.cc:1081] T 0b854d2292eb45f7b09f7279fcd23a62 P 8ca84f980eb34212944aaf54d04fd950 [term 3 LEARNER]: Refusing update from remote peer eb53f28829e04691a2601f44b09fbbe0: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch) I0703 01:39:37.241788 8214 raft_consensus.cc:1081] T 0b854d2292eb45f7b09f7279fcd23a62 P 576ad88f26964932bee2e504cb16970b [term 3 LEARNER]: Refusing update from remote peer eb53f28829e04691a2601f44b09fbbe0: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch) I0703 01:39:37.242455 8647 consensus_queue.cc:962] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ca84f980eb34212944aaf54d04fd950" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42094 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I0703 01:39:37.242702 8658 consensus_queue.cc:962] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "576ad88f26964932bee2e504cb16970b" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 48979 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I0703 01:39:37.246534 7969 raft_consensus.cc:2684] T 0b854d2292eb45f7b09f7279fcd23a62 P 8ca84f980eb34212944aaf54d04fd950 [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, 8ca84f980eb34212944aaf54d04fd950 (127.5.240.193) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb53f28829e04691a2601f44b09fbbe0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39440 } } peers { permanent_uuid: "8ca84f980eb34212944aaf54d04fd950" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42094 } attrs { promote: false } } peers { permanent_uuid: "576ad88f26964932bee2e504cb16970b" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 48979 } attrs { promote: true } } unsafe_config_change: true } I0703 01:39:37.246598 8214 raft_consensus.cc:2684] T 0b854d2292eb45f7b09f7279fcd23a62 P 576ad88f26964932bee2e504cb16970b [term 3 LEARNER]: Committing config change with OpId 3.6: config changed from index 5 to 6, 8ca84f980eb34212944aaf54d04fd950 (127.5.240.193) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb53f28829e04691a2601f44b09fbbe0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39440 } } peers { permanent_uuid: "8ca84f980eb34212944aaf54d04fd950" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42094 } attrs { promote: false } } peers { permanent_uuid: "576ad88f26964932bee2e504cb16970b" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 48979 } attrs { promote: true } } unsafe_config_change: true } I0703 01:39:37.246186 8659 raft_consensus.cc:2684] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 [term 3 LEADER]: Committing config change with OpId 3.6: config changed from index 5 to 6, 8ca84f980eb34212944aaf54d04fd950 (127.5.240.193) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb53f28829e04691a2601f44b09fbbe0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39440 } } peers { permanent_uuid: "8ca84f980eb34212944aaf54d04fd950" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42094 } attrs { promote: false } } peers { permanent_uuid: "576ad88f26964932bee2e504cb16970b" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 48979 } attrs { promote: true } } unsafe_config_change: true } I0703 01:39:37.250391 8569 catalog_manager.cc:3798] T 0b854d2292eb45f7b09f7279fcd23a62 P 576ad88f26964932bee2e504cb16970b reported cstate change: config changed from index 5 to 6, 8ca84f980eb34212944aaf54d04fd950 (127.5.240.193) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "eb53f28829e04691a2601f44b09fbbe0" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eb53f28829e04691a2601f44b09fbbe0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39440 } } peers { permanent_uuid: "8ca84f980eb34212944aaf54d04fd950" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42094 } attrs { promote: false } } peers { permanent_uuid: "576ad88f26964932bee2e504cb16970b" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 48979 } attrs { promote: true } } unsafe_config_change: true } I0703 01:39:37.251093 8658 raft_consensus.cc:890] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0: attempting to promote NON_VOTER 576ad88f26964932bee2e504cb16970b to VOTER I0703 01:39:37.251334 8658 consensus_peers.cc:455] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 -> Peer 576ad88f26964932bee2e504cb16970b (127.5.240.195:48979): Closing peer: 576ad88f26964932bee2e504cb16970b I0703 01:39:37.251422 8658 consensus_peers.cc:455] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 -> Peer 8ca84f980eb34212944aaf54d04fd950 (127.5.240.193:42094): Closing peer: 8ca84f980eb34212944aaf54d04fd950 I0703 01:39:37.251704 8658 consensus_queue.cc:204] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 0, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb53f28829e04691a2601f44b09fbbe0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39440 } } peers { permanent_uuid: "8ca84f980eb34212944aaf54d04fd950" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42094 } attrs { promote: false } } peers { permanent_uuid: "576ad88f26964932bee2e504cb16970b" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 48979 } attrs { promote: false } } unsafe_config_change: true I0703 01:39:37.252578 8647 raft_consensus.cc:853] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 [term 3 LEADER]: attempt to promote peer 576ad88f26964932bee2e504cb16970b: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing. I0703 01:39:37.253628 8214 raft_consensus.cc:1081] T 0b854d2292eb45f7b09f7279fcd23a62 P 576ad88f26964932bee2e504cb16970b [term 3 LEARNER]: Refusing update from remote peer eb53f28829e04691a2601f44b09fbbe0: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch) I0703 01:39:37.253700 7969 raft_consensus.cc:1081] T 0b854d2292eb45f7b09f7279fcd23a62 P 8ca84f980eb34212944aaf54d04fd950 [term 3 FOLLOWER]: Refusing update from remote peer eb53f28829e04691a2601f44b09fbbe0: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch) I0703 01:39:37.254259 8659 consensus_queue.cc:962] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "576ad88f26964932bee2e504cb16970b" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 48979 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I0703 01:39:37.254503 8667 consensus_queue.cc:962] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ca84f980eb34212944aaf54d04fd950" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42094 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I0703 01:39:37.257048 8667 raft_consensus.cc:2684] T 0b854d2292eb45f7b09f7279fcd23a62 P eb53f28829e04691a2601f44b09fbbe0 [term 3 LEADER]: Committing config change with OpId 3.7: config changed from index 6 to 7, 576ad88f26964932bee2e504cb16970b (127.5.240.195) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb53f28829e04691a2601f44b09fbbe0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39440 } } peers { permanent_uuid: "8ca84f980eb34212944aaf54d04fd950" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42094 } attrs { promote: false } } peers { permanent_uuid: "576ad88f26964932bee2e504cb16970b" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 48979 } attrs { promote: false } } unsafe_config_change: true } I0703 01:39:37.258200 8214 raft_consensus.cc:2684] T 0b854d2292eb45f7b09f7279fcd23a62 P 576ad88f26964932bee2e504cb16970b [term 3 FOLLOWER]: Committing config change with OpId 3.7: config changed from index 6 to 7, 576ad88f26964932bee2e504cb16970b (127.5.240.195) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb53f28829e04691a2601f44b09fbbe0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39440 } } peers { permanent_uuid: "8ca84f980eb34212944aaf54d04fd950" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42094 } attrs { promote: false } } peers { permanent_uuid: "576ad88f26964932bee2e504cb16970b" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 48979 } attrs { promote: false } } unsafe_config_change: true } I0703 01:39:37.259188 7969 raft_consensus.cc:2684] T 0b854d2292eb45f7b09f7279fcd23a62 P 8ca84f980eb34212944aaf54d04fd950 [term 3 FOLLOWER]: Committing config change with OpId 3.7: config changed from index 6 to 7, 576ad88f26964932bee2e504cb16970b (127.5.240.195) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb53f28829e04691a2601f44b09fbbe0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39440 } } peers { permanent_uuid: "8ca84f980eb34212944aaf54d04fd950" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42094 } attrs { promote: false } } peers { permanent_uuid: "576ad88f26964932bee2e504cb16970b" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 48979 } attrs { promote: false } } unsafe_config_change: true } I0703 01:39:37.260952 8567 catalog_manager.cc:3798] T 0b854d2292eb45f7b09f7279fcd23a62 P 576ad88f26964932bee2e504cb16970b reported cstate change: config changed from index 6 to 7, 576ad88f26964932bee2e504cb16970b (127.5.240.195) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "eb53f28829e04691a2601f44b09fbbe0" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eb53f28829e04691a2601f44b09fbbe0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39440 } } peers { permanent_uuid: "8ca84f980eb34212944aaf54d04fd950" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42094 } attrs { promote: false } } peers { permanent_uuid: "576ad88f26964932bee2e504cb16970b" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 48979 } attrs { promote: false } } unsafe_config_change: true } I0703 01:39:37.352926 6083 kudu-admin-test.cc:537] Waiting for Master to see new config... I0703 01:39:37.357539 6083 kudu-admin-test.cc:542] Tablet locations: tablet_id: "0b854d2292eb45f7b09f7279fcd23a62" replicas { ts_info { permanent_uuid: "eb53f28829e04691a2601f44b09fbbe0" rpc_addresses { host: "127.5.240.194" port: 39440 } } role: LEADER } replicas { ts_info { permanent_uuid: "8ca84f980eb34212944aaf54d04fd950" rpc_addresses { host: "127.5.240.193" port: 42094 } } role: FOLLOWER } replicas { ts_info { permanent_uuid: "576ad88f26964932bee2e504cb16970b" rpc_addresses { host: "127.5.240.195" port: 48979 } } role: FOLLOWER } DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } I0703 01:39:37.358629 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 7883 I0703 01:39:37.359573 8009 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:39:37.403165 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 7883 W0703 01:39:37.411849 8024 connection.cc:512] client connection to 127.5.240.193:42094 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:42094): Connection reset by peer (error 104) W0703 01:39:37.412197 8024 connection.cc:512] server connection from 127.5.240.193:48685 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:48685): Connection reset by peer (error 104) W0703 01:39:37.412314 8550 connection.cc:512] server connection from 127.5.240.193:42231 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:42231): Connection reset by peer (error 104) I0703 01:39:37.412672 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 8016 I0703 01:39:37.413394 8132 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:39:37.461000 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 8016 W0703 01:39:37.469759 8553 connection.cc:512] server connection from 127.5.240.194:41798 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:41798): Connection reset by peer (error 104) W0703 01:39:37.470093 8146 connection.cc:512] server connection from 127.5.240.194:53440 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:53440): Connection reset by peer (error 104) I0703 01:39:37.470544 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 8138 W0703 01:39:37.470526 8147 connection.cc:512] client connection to 127.5.240.194:39440 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:39440): Connection reset by peer (error 104) I0703 01:39:37.471343 8254 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:39:37.514031 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 8138 W0703 01:39:37.522229 8551 connection.cc:512] server connection from 127.5.240.195:39668 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.195:39668): Connection reset by peer (error 104) I0703 01:39:37.522320 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master pid 8543 I0703 01:39:37.523072 8599 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:39:37.585047 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master with pid 8543 [ OK ] AdminCliTest.TestUnsafeChangeConfigOnSingleLeader (6808 ms) [ RUN ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes I0703 01:39:37.608916 6083 test_util.cc:208] Using random seed: 1092398785 I0703 01:39:37.610433 6083 ts_itest-base.cc:113] Starting cluster with: I0703 01:39:37.610462 6083 ts_itest-base.cc:114] -------------- I0703 01:39:37.610479 6083 ts_itest-base.cc:115] 4 tablet servers I0703 01:39:37.610496 6083 ts_itest-base.cc:116] 3 replicas per TS I0703 01:39:37.610509 6083 ts_itest-base.cc:117] -------------- I0703 01:39:37.611711 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:38609 --webserver_interface=127.5.240.254 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:39:37.791194 8681 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:39:37.791402 8681 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:39:37.793591 8681 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0703 01:39:37.793649 8681 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:39:37.793684 8681 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0703 01:39:37.793711 8681 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0703 01:39:37.801731 8681 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:38609 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.5.240.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:39:37.807287 8681 master_main.cc:80] Initializing master server... I0703 01:39:37.808342 8681 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 142208us I0703 01:39:37.809389 8681 fs_manager.cc:261] Metadata directory not provided I0703 01:39:37.809444 8681 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:39:37.809711 8681 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:39:37.809785 8681 server_base.cc:433] Attempting to create new FS layout instead I0703 01:39:37.813912 8681 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/instance: uuid: "47e88b229dc046b1ac2d0417718d1d24" format_stamp: "Formatted at 2018-07-03 08:39:37 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:37.814399 8681 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "47e88b229dc046b1ac2d0417718d1d24" format_stamp: "Formatted at 2018-07-03 08:39:37 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:37.822331 8681 fs_manager.cc:501] Time spent creating directory manager: real 0.008s user 0.006s sys 0.000s I0703 01:39:37.823519 8681 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:39:37.823920 8681 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:39:37.827493 8681 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s I0703 01:39:37.827543 8681 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal uuid: "47e88b229dc046b1ac2d0417718d1d24" format_stamp: "Formatted at 2018-07-03 08:39:37 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:37.827666 8681 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:39:37.858194 8681 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:39:37.859151 8681 master_main.cc:83] Starting Master server... I0703 01:39:37.895635 8681 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.254:38609 I0703 01:39:37.896085 8681 webserver.cc:175] Starting webserver on 127.5.240.254:0 I0703 01:39:37.896129 8681 webserver.cc:186] Document root disabled I0703 01:39:37.897320 8681 webserver.cc:313] Webserver started. Bound to: http://127.5.240.254:47735/ I0703 01:39:37.897928 8681 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb I0703 01:39:37.901427 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master as pid 8681 I0703 01:39:37.901724 8740 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:39:37.902305 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.5.240.193:0 --local_ip_for_outbound_sockets=127.5.240.193 --webserver_interface=127.5.240.193 --webserver_port=0 --tserver_master_addrs=127.5.240.254:38609 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0703 01:39:37.914296 8740 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 47e88b229dc046b1ac2d0417718d1d24: Bootstrap starting. I0703 01:39:37.916267 8740 tablet_bootstrap.cc:578] T 00000000000000000000000000000000 P 47e88b229dc046b1ac2d0417718d1d24: No blocks or log segments found. Creating new log. I0703 01:39:37.917013 8740 log.cc:525] T 00000000000000000000000000000000 P 47e88b229dc046b1ac2d0417718d1d24: Log is configured to *not* fsync() on all Append() calls I0703 01:39:37.918797 8740 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 47e88b229dc046b1ac2d0417718d1d24: No bootstrap required, opened a new log I0703 01:39:37.924937 8740 raft_consensus.cc:325] T 00000000000000000000000000000000 P 47e88b229dc046b1ac2d0417718d1d24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47e88b229dc046b1ac2d0417718d1d24" member_type: VOTER } I0703 01:39:37.925125 8740 raft_consensus.cc:351] T 00000000000000000000000000000000 P 47e88b229dc046b1ac2d0417718d1d24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:39:37.925184 8740 raft_consensus.cc:628] T 00000000000000000000000000000000 P 47e88b229dc046b1ac2d0417718d1d24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47e88b229dc046b1ac2d0417718d1d24, State: Initialized, Role: FOLLOWER I0703 01:39:37.925694 8740 consensus_queue.cc:227] T 00000000000000000000000000000000 P 47e88b229dc046b1ac2d0417718d1d24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47e88b229dc046b1ac2d0417718d1d24" member_type: VOTER } I0703 01:39:37.925837 8740 raft_consensus.cc:365] T 00000000000000000000000000000000 P 47e88b229dc046b1ac2d0417718d1d24 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:39:37.925911 8740 raft_consensus.cc:457] T 00000000000000000000000000000000 P 47e88b229dc046b1ac2d0417718d1d24 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:39:37.925990 8740 raft_consensus.cc:2730] T 00000000000000000000000000000000 P 47e88b229dc046b1ac2d0417718d1d24: Snoozing failure detection for 1.854s (starting election) I0703 01:39:37.926072 8740 raft_consensus.cc:2782] T 00000000000000000000000000000000 P 47e88b229dc046b1ac2d0417718d1d24 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:39:37.927371 8740 raft_consensus.cc:479] T 00000000000000000000000000000000 P 47e88b229dc046b1ac2d0417718d1d24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47e88b229dc046b1ac2d0417718d1d24" member_type: VOTER } I0703 01:39:37.927748 8740 leader_election.cc:270] T 00000000000000000000000000000000 P 47e88b229dc046b1ac2d0417718d1d24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:39:37.929260 8743 raft_consensus.cc:2730] T 00000000000000000000000000000000 P 47e88b229dc046b1ac2d0417718d1d24: Snoozing failure detection for 1.938s (election complete) I0703 01:39:37.929471 8743 raft_consensus.cc:2537] T 00000000000000000000000000000000 P 47e88b229dc046b1ac2d0417718d1d24 [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:39:37.935151 8743 raft_consensus.cc:593] T 00000000000000000000000000000000 P 47e88b229dc046b1ac2d0417718d1d24 [term 1 LEADER]: Becoming Leader. State: Replica: 47e88b229dc046b1ac2d0417718d1d24, State: Running, Role: LEADER I0703 01:39:37.935645 8743 consensus_queue.cc:204] T 00000000000000000000000000000000 P 47e88b229dc046b1ac2d0417718d1d24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47e88b229dc046b1ac2d0417718d1d24" member_type: VOTER } I0703 01:39:37.938066 8740 sys_catalog.cc:337] T 00000000000000000000000000000000 P 47e88b229dc046b1ac2d0417718d1d24 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "47e88b229dc046b1ac2d0417718d1d24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47e88b229dc046b1ac2d0417718d1d24" member_type: VOTER } } I0703 01:39:37.938408 8740 sys_catalog.cc:340] T 00000000000000000000000000000000 P 47e88b229dc046b1ac2d0417718d1d24 [sys.catalog]: This master's current role is: LEADER I0703 01:39:37.938212 8744 sys_catalog.cc:337] T 00000000000000000000000000000000 P 47e88b229dc046b1ac2d0417718d1d24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47e88b229dc046b1ac2d0417718d1d24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47e88b229dc046b1ac2d0417718d1d24" member_type: VOTER } } I0703 01:39:37.939163 8744 sys_catalog.cc:340] T 00000000000000000000000000000000 P 47e88b229dc046b1ac2d0417718d1d24 [sys.catalog]: This master's current role is: LEADER I0703 01:39:37.939370 8745 sys_catalog.cc:337] T 00000000000000000000000000000000 P 47e88b229dc046b1ac2d0417718d1d24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47e88b229dc046b1ac2d0417718d1d24. Latest consensus state: current_term: 1 leader_uuid: "47e88b229dc046b1ac2d0417718d1d24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47e88b229dc046b1ac2d0417718d1d24" member_type: VOTER } } I0703 01:39:37.939998 8745 sys_catalog.cc:340] T 00000000000000000000000000000000 P 47e88b229dc046b1ac2d0417718d1d24 [sys.catalog]: This master's current role is: LEADER I0703 01:39:37.942471 8740 sys_catalog.cc:424] T 00000000000000000000000000000000 P 47e88b229dc046b1ac2d0417718d1d24 [sys.catalog]: configured and running, proceeding with master startup. I0703 01:39:37.942854 8747 catalog_manager.cc:1006] Loading table and tablet metadata into memory... I0703 01:39:37.944152 8681 master_main.cc:86] Master server successfully started. I0703 01:39:37.949260 8747 catalog_manager.cc:1017] Initializing Kudu internal certificate authority... I0703 01:39:37.973419 8747 catalog_manager.cc:904] Generated new certificate authority record I0703 01:39:37.975296 8747 catalog_manager.cc:1026] Loading token signing keys... I0703 01:39:37.986807 8747 catalog_manager.cc:4102] T 00000000000000000000000000000000 P 47e88b229dc046b1ac2d0417718d1d24: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:39:38.070870 8741 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:39:38.071080 8741 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:39:38.073087 8741 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:39:38.073210 8741 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.193 I0703 01:39:38.080229 8741 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.5.240.193 --webserver_port=0 --tserver_master_addrs=127.5.240.254:38609 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.193 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:39:38.086789 8741 tablet_server_main.cc:85] Initializing tablet server... I0703 01:39:38.087947 8741 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 142708us I0703 01:39:38.088925 8741 fs_manager.cc:261] Metadata directory not provided I0703 01:39:38.088989 8741 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:39:38.089236 8741 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:39:38.089299 8741 server_base.cc:433] Attempting to create new FS layout instead I0703 01:39:38.093438 8741 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "6cf745d60d5445749567bc9afd503edb" format_stamp: "Formatted at 2018-07-03 08:39:38 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:38.093941 8741 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "6cf745d60d5445749567bc9afd503edb" format_stamp: "Formatted at 2018-07-03 08:39:38 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:38.101794 8741 fs_manager.cc:501] Time spent creating directory manager: real 0.008s user 0.001s sys 0.005s I0703 01:39:38.102980 8741 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:39:38.103371 8741 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:39:38.107368 8741 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s I0703 01:39:38.107427 8741 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal uuid: "6cf745d60d5445749567bc9afd503edb" format_stamp: "Formatted at 2018-07-03 08:39:38 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:38.107553 8741 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:39:38.145319 8741 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:39:38.147883 8741 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:39:38.148337 8741 tablet_server_main.cc:90] Starting tablet server... I0703 01:39:38.251240 8741 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.193:54395 I0703 01:39:38.251757 8741 webserver.cc:175] Starting webserver on 127.5.240.193:0 I0703 01:39:38.251797 8741 webserver.cc:186] Document root disabled I0703 01:39:38.253121 8741 webserver.cc:313] Webserver started. Bound to: http://127.5.240.193:60450/ I0703 01:39:38.253679 8741 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb I0703 01:39:38.255170 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 8741 I0703 01:39:38.256052 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.5.240.194:0 --local_ip_for_outbound_sockets=127.5.240.194 --webserver_interface=127.5.240.194 --webserver_port=0 --tserver_master_addrs=127.5.240.254:38609 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0703 01:39:38.264294 8741 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:39:38.275614 8870 heartbeater.cc:346] Connected to a master server at 127.5.240.254:38609 I0703 01:39:38.275765 8870 heartbeater.cc:426] Registering TS with master... I0703 01:39:38.276561 8870 heartbeater.cc:475] Master 127.5.240.254:38609 requested a full tablet report, sending... I0703 01:39:38.278143 8707 ts_manager.cc:98] Registered new tserver with Master: 6cf745d60d5445749567bc9afd503edb (127.5.240.193:54395) I0703 01:39:38.280170 8707 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.193:54336 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:39:38.421284 8874 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:39:38.421541 8874 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:39:38.423569 8874 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:39:38.423691 8874 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.194 I0703 01:39:38.430362 8874 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.194:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.5.240.194 --webserver_port=0 --tserver_master_addrs=127.5.240.254:38609 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.194 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:39:38.436699 8874 tablet_server_main.cc:85] Initializing tablet server... I0703 01:39:38.437865 8874 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 142708us I0703 01:39:38.438838 8874 fs_manager.cc:261] Metadata directory not provided I0703 01:39:38.438901 8874 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:39:38.439127 8874 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:39:38.439191 8874 server_base.cc:433] Attempting to create new FS layout instead I0703 01:39:38.443250 8874 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "27562c9910fa4819a8ecf91dafbf7ce1" format_stamp: "Formatted at 2018-07-03 08:39:38 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:38.443794 8874 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "27562c9910fa4819a8ecf91dafbf7ce1" format_stamp: "Formatted at 2018-07-03 08:39:38 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:38.451560 8874 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.003s sys 0.003s I0703 01:39:38.452736 8874 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:39:38.453126 8874 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:39:38.456739 8874 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0703 01:39:38.456799 8874 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal uuid: "27562c9910fa4819a8ecf91dafbf7ce1" format_stamp: "Formatted at 2018-07-03 08:39:38 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:38.456923 8874 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:39:38.482910 8874 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:39:38.485189 8874 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:39:38.485560 8874 tablet_server_main.cc:90] Starting tablet server... I0703 01:39:38.569703 8874 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.194:38901 I0703 01:39:38.570134 8874 webserver.cc:175] Starting webserver on 127.5.240.194:0 I0703 01:39:38.570170 8874 webserver.cc:186] Document root disabled I0703 01:39:38.570884 8874 webserver.cc:313] Webserver started. Bound to: http://127.5.240.194:36607/ I0703 01:39:38.571403 8874 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb I0703 01:39:38.573247 8874 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:39:38.579154 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 8874 I0703 01:39:38.579807 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.5.240.195:0 --local_ip_for_outbound_sockets=127.5.240.195 --webserver_interface=127.5.240.195 --webserver_port=0 --tserver_master_addrs=127.5.240.254:38609 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0703 01:39:38.585402 8993 heartbeater.cc:346] Connected to a master server at 127.5.240.254:38609 I0703 01:39:38.585582 8993 heartbeater.cc:426] Registering TS with master... I0703 01:39:38.586534 8993 heartbeater.cc:475] Master 127.5.240.254:38609 requested a full tablet report, sending... I0703 01:39:38.587584 8707 ts_manager.cc:98] Registered new tserver with Master: 27562c9910fa4819a8ecf91dafbf7ce1 (127.5.240.194:38901) I0703 01:39:38.588407 8707 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.194:48290 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:39:38.737702 8996 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:39:38.737918 8996 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:39:38.739944 8996 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:39:38.740034 8996 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.195 I0703 01:39:38.746938 8996 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.195:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.5.240.195 --webserver_port=0 --tserver_master_addrs=127.5.240.254:38609 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.195 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:39:38.753396 8996 tablet_server_main.cc:85] Initializing tablet server... I0703 01:39:38.754489 8996 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 142708us I0703 01:39:38.755417 8996 fs_manager.cc:261] Metadata directory not provided I0703 01:39:38.755470 8996 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:39:38.755704 8996 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:39:38.755761 8996 server_base.cc:433] Attempting to create new FS layout instead I0703 01:39:38.759779 8996 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "e52364e0246c4bb4852c27e967e2f4c8" format_stamp: "Formatted at 2018-07-03 08:39:38 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:38.760275 8996 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "e52364e0246c4bb4852c27e967e2f4c8" format_stamp: "Formatted at 2018-07-03 08:39:38 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:38.767726 8996 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.004s sys 0.003s I0703 01:39:38.768893 8996 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:39:38.769255 8996 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:39:38.772774 8996 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s I0703 01:39:38.772825 8996 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal uuid: "e52364e0246c4bb4852c27e967e2f4c8" format_stamp: "Formatted at 2018-07-03 08:39:38 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:38.772933 8996 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:39:38.851371 8996 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:39:38.853716 8996 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:39:38.854085 8996 tablet_server_main.cc:90] Starting tablet server... I0703 01:39:38.935894 8996 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.195:46355 I0703 01:39:38.936300 8996 webserver.cc:175] Starting webserver on 127.5.240.195:0 I0703 01:39:38.936341 8996 webserver.cc:186] Document root disabled I0703 01:39:38.936983 8996 webserver.cc:313] Webserver started. Bound to: http://127.5.240.195:52949/ I0703 01:39:38.937502 8996 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb I0703 01:39:38.939666 8996 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:39:38.941853 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 8996 I0703 01:39:38.942733 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.5.240.196:0 --local_ip_for_outbound_sockets=127.5.240.196 --webserver_interface=127.5.240.196 --webserver_port=0 --tserver_master_addrs=127.5.240.254:38609 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0703 01:39:38.952136 9115 heartbeater.cc:346] Connected to a master server at 127.5.240.254:38609 I0703 01:39:38.952276 9115 heartbeater.cc:426] Registering TS with master... I0703 01:39:38.953073 9115 heartbeater.cc:475] Master 127.5.240.254:38609 requested a full tablet report, sending... I0703 01:39:38.954087 8707 ts_manager.cc:98] Registered new tserver with Master: e52364e0246c4bb4852c27e967e2f4c8 (127.5.240.195:46355) I0703 01:39:38.954944 8707 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.195:51719 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:39:39.098013 9118 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:39:39.098235 9118 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:39:39.100231 9118 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:39:39.100327 9118 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.196 I0703 01:39:39.106803 9118 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.196:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb --webserver_interface=127.5.240.196 --webserver_port=0 --tserver_master_addrs=127.5.240.254:38609 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.196 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:39:39.113363 9118 tablet_server_main.cc:85] Initializing tablet server... I0703 01:39:39.114445 9118 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 143208us I0703 01:39:39.115383 9118 fs_manager.cc:261] Metadata directory not provided I0703 01:39:39.115439 9118 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:39:39.116050 9118 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:39:39.116111 9118 server_base.cc:433] Attempting to create new FS layout instead I0703 01:39:39.120142 9118 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/instance: uuid: "38c3a80a7fa04fcebe7dea0e351bf9b2" format_stamp: "Formatted at 2018-07-03 08:39:39 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:39.120671 9118 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal/instance: uuid: "38c3a80a7fa04fcebe7dea0e351bf9b2" format_stamp: "Formatted at 2018-07-03 08:39:39 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:39.128412 9118 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.003s sys 0.003s I0703 01:39:39.129580 9118 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:39:39.129954 9118 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:39:39.132966 9118 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0703 01:39:39.133015 9118 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal uuid: "38c3a80a7fa04fcebe7dea0e351bf9b2" format_stamp: "Formatted at 2018-07-03 08:39:39 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:39.133121 9118 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:39:39.175202 9118 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:39:39.177419 9118 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:39:39.177752 9118 tablet_server_main.cc:90] Starting tablet server... I0703 01:39:39.262079 9118 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.196:34814 I0703 01:39:39.262490 9118 webserver.cc:175] Starting webserver on 127.5.240.196:0 I0703 01:39:39.262521 9118 webserver.cc:186] Document root disabled I0703 01:39:39.263691 9118 webserver.cc:313] Webserver started. Bound to: http://127.5.240.196:41888/ I0703 01:39:39.264246 9118 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb I0703 01:39:39.266461 9118 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:39:39.274586 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 9118 I0703 01:39:39.278807 9237 heartbeater.cc:346] Connected to a master server at 127.5.240.254:38609 I0703 01:39:39.278961 9237 heartbeater.cc:426] Registering TS with master... I0703 01:39:39.279916 9237 heartbeater.cc:475] Master 127.5.240.254:38609 requested a full tablet report, sending... I0703 01:39:39.281349 8706 ts_manager.cc:98] Registered new tserver with Master: 38c3a80a7fa04fcebe7dea0e351bf9b2 (127.5.240.196:34814) I0703 01:39:39.282246 8870 heartbeater.cc:467] Master 127.5.240.254:38609 was elected leader, sending a full tablet report... I0703 01:39:39.282392 8706 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.196:43733 I0703 01:39:39.283869 6083 external_mini_cluster.cc:490] 4 TS(s) registered with all masters I0703 01:39:39.293246 8706 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.5.240.254:36443: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0703 01:39:39.318214 9052 tablet_service.cc:759] Processing CreateTablet for tablet c6eb9324dee0414bbcb2039e9a23fab5 (table=TestTable [id=3f0cae68e9f5467da8f010c417d00d44]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:39:39.319146 9052 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet c6eb9324dee0414bbcb2039e9a23fab5. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:39:39.319336 9174 tablet_service.cc:759] Processing CreateTablet for tablet c6eb9324dee0414bbcb2039e9a23fab5 (table=TestTable [id=3f0cae68e9f5467da8f010c417d00d44]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:39:39.319665 8930 tablet_service.cc:759] Processing CreateTablet for tablet c6eb9324dee0414bbcb2039e9a23fab5 (table=TestTable [id=3f0cae68e9f5467da8f010c417d00d44]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:39:39.320037 9174 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet c6eb9324dee0414bbcb2039e9a23fab5. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:39:39.320418 8930 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet c6eb9324dee0414bbcb2039e9a23fab5. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:39:39.329771 9174 ts_tablet_manager.cc:1173] T c6eb9324dee0414bbcb2039e9a23fab5 P 38c3a80a7fa04fcebe7dea0e351bf9b2: Registered tablet (data state: TABLET_DATA_READY) I0703 01:39:39.332036 9257 ts_tablet_manager.cc:1028] T c6eb9324dee0414bbcb2039e9a23fab5 P 38c3a80a7fa04fcebe7dea0e351bf9b2: Bootstrapping tablet I0703 01:39:39.332973 9257 tablet_bootstrap.cc:427] T c6eb9324dee0414bbcb2039e9a23fab5 P 38c3a80a7fa04fcebe7dea0e351bf9b2: Bootstrap starting. I0703 01:39:39.334558 8930 ts_tablet_manager.cc:1173] T c6eb9324dee0414bbcb2039e9a23fab5 P 27562c9910fa4819a8ecf91dafbf7ce1: Registered tablet (data state: TABLET_DATA_READY) I0703 01:39:39.334537 9052 ts_tablet_manager.cc:1173] T c6eb9324dee0414bbcb2039e9a23fab5 P e52364e0246c4bb4852c27e967e2f4c8: Registered tablet (data state: TABLET_DATA_READY) I0703 01:39:39.335373 9258 ts_tablet_manager.cc:1028] T c6eb9324dee0414bbcb2039e9a23fab5 P 27562c9910fa4819a8ecf91dafbf7ce1: Bootstrapping tablet I0703 01:39:39.335808 9257 tablet_bootstrap.cc:578] T c6eb9324dee0414bbcb2039e9a23fab5 P 38c3a80a7fa04fcebe7dea0e351bf9b2: No blocks or log segments found. Creating new log. I0703 01:39:39.335959 9258 tablet_bootstrap.cc:427] T c6eb9324dee0414bbcb2039e9a23fab5 P 27562c9910fa4819a8ecf91dafbf7ce1: Bootstrap starting. I0703 01:39:39.336963 9259 ts_tablet_manager.cc:1028] T c6eb9324dee0414bbcb2039e9a23fab5 P e52364e0246c4bb4852c27e967e2f4c8: Bootstrapping tablet I0703 01:39:39.337471 9259 tablet_bootstrap.cc:427] T c6eb9324dee0414bbcb2039e9a23fab5 P e52364e0246c4bb4852c27e967e2f4c8: Bootstrap starting. I0703 01:39:39.338086 9257 log.cc:525] T c6eb9324dee0414bbcb2039e9a23fab5 P 38c3a80a7fa04fcebe7dea0e351bf9b2: Log is configured to *not* fsync() on all Append() calls I0703 01:39:39.339092 9258 tablet_bootstrap.cc:578] T c6eb9324dee0414bbcb2039e9a23fab5 P 27562c9910fa4819a8ecf91dafbf7ce1: No blocks or log segments found. Creating new log. I0703 01:39:39.339970 9258 log.cc:525] T c6eb9324dee0414bbcb2039e9a23fab5 P 27562c9910fa4819a8ecf91dafbf7ce1: Log is configured to *not* fsync() on all Append() calls I0703 01:39:39.340024 9259 tablet_bootstrap.cc:578] T c6eb9324dee0414bbcb2039e9a23fab5 P e52364e0246c4bb4852c27e967e2f4c8: No blocks or log segments found. Creating new log. I0703 01:39:39.340180 9257 tablet_bootstrap.cc:427] T c6eb9324dee0414bbcb2039e9a23fab5 P 38c3a80a7fa04fcebe7dea0e351bf9b2: No bootstrap required, opened a new log I0703 01:39:39.340382 9257 ts_tablet_manager.cc:1045] T c6eb9324dee0414bbcb2039e9a23fab5 P 38c3a80a7fa04fcebe7dea0e351bf9b2: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.001s I0703 01:39:39.341019 9259 log.cc:525] T c6eb9324dee0414bbcb2039e9a23fab5 P e52364e0246c4bb4852c27e967e2f4c8: Log is configured to *not* fsync() on all Append() calls I0703 01:39:39.342175 9258 tablet_bootstrap.cc:427] T c6eb9324dee0414bbcb2039e9a23fab5 P 27562c9910fa4819a8ecf91dafbf7ce1: No bootstrap required, opened a new log I0703 01:39:39.342332 9258 ts_tablet_manager.cc:1045] T c6eb9324dee0414bbcb2039e9a23fab5 P 27562c9910fa4819a8ecf91dafbf7ce1: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.005s I0703 01:39:39.347323 9257 raft_consensus.cc:325] T c6eb9324dee0414bbcb2039e9a23fab5 P 38c3a80a7fa04fcebe7dea0e351bf9b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27562c9910fa4819a8ecf91dafbf7ce1" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38901 } } peers { permanent_uuid: "e52364e0246c4bb4852c27e967e2f4c8" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46355 } } peers { permanent_uuid: "38c3a80a7fa04fcebe7dea0e351bf9b2" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 34814 } } I0703 01:39:39.347666 9257 raft_consensus.cc:351] T c6eb9324dee0414bbcb2039e9a23fab5 P 38c3a80a7fa04fcebe7dea0e351bf9b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:39:39.347735 9257 raft_consensus.cc:628] T c6eb9324dee0414bbcb2039e9a23fab5 P 38c3a80a7fa04fcebe7dea0e351bf9b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38c3a80a7fa04fcebe7dea0e351bf9b2, State: Initialized, Role: FOLLOWER I0703 01:39:39.348155 9257 consensus_queue.cc:227] T c6eb9324dee0414bbcb2039e9a23fab5 P 38c3a80a7fa04fcebe7dea0e351bf9b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27562c9910fa4819a8ecf91dafbf7ce1" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38901 } } peers { permanent_uuid: "e52364e0246c4bb4852c27e967e2f4c8" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46355 } } peers { permanent_uuid: "38c3a80a7fa04fcebe7dea0e351bf9b2" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 34814 } } I0703 01:39:39.348409 9259 tablet_bootstrap.cc:427] T c6eb9324dee0414bbcb2039e9a23fab5 P e52364e0246c4bb4852c27e967e2f4c8: No bootstrap required, opened a new log I0703 01:39:39.348181 9258 raft_consensus.cc:325] T c6eb9324dee0414bbcb2039e9a23fab5 P 27562c9910fa4819a8ecf91dafbf7ce1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27562c9910fa4819a8ecf91dafbf7ce1" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38901 } } peers { permanent_uuid: "e52364e0246c4bb4852c27e967e2f4c8" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46355 } } peers { permanent_uuid: "38c3a80a7fa04fcebe7dea0e351bf9b2" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 34814 } } I0703 01:39:39.348613 9258 raft_consensus.cc:351] T c6eb9324dee0414bbcb2039e9a23fab5 P 27562c9910fa4819a8ecf91dafbf7ce1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:39:39.348656 9259 ts_tablet_manager.cc:1045] T c6eb9324dee0414bbcb2039e9a23fab5 P e52364e0246c4bb4852c27e967e2f4c8: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.002s I0703 01:39:39.348716 9258 raft_consensus.cc:628] T c6eb9324dee0414bbcb2039e9a23fab5 P 27562c9910fa4819a8ecf91dafbf7ce1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27562c9910fa4819a8ecf91dafbf7ce1, State: Initialized, Role: FOLLOWER I0703 01:39:39.349587 9237 heartbeater.cc:467] Master 127.5.240.254:38609 was elected leader, sending a full tablet report... I0703 01:39:39.349994 9257 ts_tablet_manager.cc:1073] T c6eb9324dee0414bbcb2039e9a23fab5 P 38c3a80a7fa04fcebe7dea0e351bf9b2: Time spent starting tablet: real 0.010s user 0.006s sys 0.003s I0703 01:39:39.349311 9258 consensus_queue.cc:227] T c6eb9324dee0414bbcb2039e9a23fab5 P 27562c9910fa4819a8ecf91dafbf7ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27562c9910fa4819a8ecf91dafbf7ce1" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38901 } } peers { permanent_uuid: "e52364e0246c4bb4852c27e967e2f4c8" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46355 } } peers { permanent_uuid: "38c3a80a7fa04fcebe7dea0e351bf9b2" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 34814 } } I0703 01:39:39.350978 8993 heartbeater.cc:467] Master 127.5.240.254:38609 was elected leader, sending a full tablet report... I0703 01:39:39.351893 9258 ts_tablet_manager.cc:1073] T c6eb9324dee0414bbcb2039e9a23fab5 P 27562c9910fa4819a8ecf91dafbf7ce1: Time spent starting tablet: real 0.009s user 0.004s sys 0.004s I0703 01:39:39.356729 9259 raft_consensus.cc:325] T c6eb9324dee0414bbcb2039e9a23fab5 P e52364e0246c4bb4852c27e967e2f4c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27562c9910fa4819a8ecf91dafbf7ce1" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38901 } } peers { permanent_uuid: "e52364e0246c4bb4852c27e967e2f4c8" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46355 } } peers { permanent_uuid: "38c3a80a7fa04fcebe7dea0e351bf9b2" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 34814 } } I0703 01:39:39.357023 9259 raft_consensus.cc:351] T c6eb9324dee0414bbcb2039e9a23fab5 P e52364e0246c4bb4852c27e967e2f4c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:39:39.357085 9259 raft_consensus.cc:628] T c6eb9324dee0414bbcb2039e9a23fab5 P e52364e0246c4bb4852c27e967e2f4c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e52364e0246c4bb4852c27e967e2f4c8, State: Initialized, Role: FOLLOWER I0703 01:39:39.357447 9259 consensus_queue.cc:227] T c6eb9324dee0414bbcb2039e9a23fab5 P e52364e0246c4bb4852c27e967e2f4c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27562c9910fa4819a8ecf91dafbf7ce1" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38901 } } peers { permanent_uuid: "e52364e0246c4bb4852c27e967e2f4c8" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46355 } } peers { permanent_uuid: "38c3a80a7fa04fcebe7dea0e351bf9b2" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 34814 } } I0703 01:39:39.358683 9115 heartbeater.cc:467] Master 127.5.240.254:38609 was elected leader, sending a full tablet report... I0703 01:39:39.359220 9259 ts_tablet_manager.cc:1073] T c6eb9324dee0414bbcb2039e9a23fab5 P e52364e0246c4bb4852c27e967e2f4c8: Time spent starting tablet: real 0.010s user 0.008s sys 0.001s W0703 01:39:39.440006 9116 tablet.cc:1678] T c6eb9324dee0414bbcb2039e9a23fab5 P e52364e0246c4bb4852c27e967e2f4c8: Can't schedule compaction. Clean time has not been advanced past its initial value. W0703 01:39:39.516679 9238 tablet.cc:1678] T c6eb9324dee0414bbcb2039e9a23fab5 P 38c3a80a7fa04fcebe7dea0e351bf9b2: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:39:39.521168 9264 raft_consensus.cc:457] T c6eb9324dee0414bbcb2039e9a23fab5 P 27562c9910fa4819a8ecf91dafbf7ce1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0703 01:39:39.521268 9264 raft_consensus.cc:2730] T c6eb9324dee0414bbcb2039e9a23fab5 P 27562c9910fa4819a8ecf91dafbf7ce1: Snoozing failure detection for 1.867s (starting election) I0703 01:39:39.521363 9264 raft_consensus.cc:479] T c6eb9324dee0414bbcb2039e9a23fab5 P 27562c9910fa4819a8ecf91dafbf7ce1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27562c9910fa4819a8ecf91dafbf7ce1" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38901 } } peers { permanent_uuid: "e52364e0246c4bb4852c27e967e2f4c8" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46355 } } peers { permanent_uuid: "38c3a80a7fa04fcebe7dea0e351bf9b2" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 34814 } } I0703 01:39:39.522521 9264 leader_election.cc:243] T c6eb9324dee0414bbcb2039e9a23fab5 P 27562c9910fa4819a8ecf91dafbf7ce1 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer e52364e0246c4bb4852c27e967e2f4c8 (127.5.240.195:46355) I0703 01:39:39.522836 9264 leader_election.cc:243] T c6eb9324dee0414bbcb2039e9a23fab5 P 27562c9910fa4819a8ecf91dafbf7ce1 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 38c3a80a7fa04fcebe7dea0e351bf9b2 (127.5.240.196:34814) I0703 01:39:39.531950 9072 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "c6eb9324dee0414bbcb2039e9a23fab5" candidate_uuid: "27562c9910fa4819a8ecf91dafbf7ce1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e52364e0246c4bb4852c27e967e2f4c8" is_pre_election: true I0703 01:39:39.532325 9072 raft_consensus.cc:2730] T c6eb9324dee0414bbcb2039e9a23fab5 P e52364e0246c4bb4852c27e967e2f4c8: Snoozing failure detection for 1.852s (vote granted) I0703 01:39:39.532449 9072 raft_consensus.cc:2243] T c6eb9324dee0414bbcb2039e9a23fab5 P e52364e0246c4bb4852c27e967e2f4c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27562c9910fa4819a8ecf91dafbf7ce1 in term 0. I0703 01:39:39.532488 9194 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "c6eb9324dee0414bbcb2039e9a23fab5" candidate_uuid: "27562c9910fa4819a8ecf91dafbf7ce1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38c3a80a7fa04fcebe7dea0e351bf9b2" is_pre_election: true I0703 01:39:39.533215 8881 leader_election.cc:379] T c6eb9324dee0414bbcb2039e9a23fab5 P 27562c9910fa4819a8ecf91dafbf7ce1 [CANDIDATE]: Term 1 pre-election: Vote granted by peer e52364e0246c4bb4852c27e967e2f4c8 (127.5.240.195:46355) I0703 01:39:39.533325 8881 leader_election.cc:270] T c6eb9324dee0414bbcb2039e9a23fab5 P 27562c9910fa4819a8ecf91dafbf7ce1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0703 01:39:39.533207 9194 raft_consensus.cc:2730] T c6eb9324dee0414bbcb2039e9a23fab5 P 38c3a80a7fa04fcebe7dea0e351bf9b2: Snoozing failure detection for 1.625s (vote granted) I0703 01:39:39.533735 9194 raft_consensus.cc:2243] T c6eb9324dee0414bbcb2039e9a23fab5 P 38c3a80a7fa04fcebe7dea0e351bf9b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27562c9910fa4819a8ecf91dafbf7ce1 in term 0. I0703 01:39:39.533774 9264 raft_consensus.cc:2730] T c6eb9324dee0414bbcb2039e9a23fab5 P 27562c9910fa4819a8ecf91dafbf7ce1: Snoozing failure detection for 2.137s (election complete) I0703 01:39:39.533857 9264 raft_consensus.cc:2537] T c6eb9324dee0414bbcb2039e9a23fab5 P 27562c9910fa4819a8ecf91dafbf7ce1 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0703 01:39:39.533936 9264 raft_consensus.cc:457] T c6eb9324dee0414bbcb2039e9a23fab5 P 27562c9910fa4819a8ecf91dafbf7ce1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0703 01:39:39.534021 9264 raft_consensus.cc:2730] T c6eb9324dee0414bbcb2039e9a23fab5 P 27562c9910fa4819a8ecf91dafbf7ce1: Snoozing failure detection for 1.933s (starting election) I0703 01:39:39.534088 9264 raft_consensus.cc:2782] T c6eb9324dee0414bbcb2039e9a23fab5 P 27562c9910fa4819a8ecf91dafbf7ce1 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:39:39.534345 8882 leader_election.cc:379] T c6eb9324dee0414bbcb2039e9a23fab5 P 27562c9910fa4819a8ecf91dafbf7ce1 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 38c3a80a7fa04fcebe7dea0e351bf9b2 (127.5.240.196:34814) I0703 01:39:39.535859 9264 raft_consensus.cc:479] T c6eb9324dee0414bbcb2039e9a23fab5 P 27562c9910fa4819a8ecf91dafbf7ce1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27562c9910fa4819a8ecf91dafbf7ce1" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38901 } } peers { permanent_uuid: "e52364e0246c4bb4852c27e967e2f4c8" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46355 } } peers { permanent_uuid: "38c3a80a7fa04fcebe7dea0e351bf9b2" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 34814 } } I0703 01:39:39.536240 9264 leader_election.cc:243] T c6eb9324dee0414bbcb2039e9a23fab5 P 27562c9910fa4819a8ecf91dafbf7ce1 [CANDIDATE]: Term 1 election: Requesting vote from peer e52364e0246c4bb4852c27e967e2f4c8 (127.5.240.195:46355) I0703 01:39:39.536408 9264 leader_election.cc:243] T c6eb9324dee0414bbcb2039e9a23fab5 P 27562c9910fa4819a8ecf91dafbf7ce1 [CANDIDATE]: Term 1 election: Requesting vote from peer 38c3a80a7fa04fcebe7dea0e351bf9b2 (127.5.240.196:34814) I0703 01:39:39.536962 9072 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "c6eb9324dee0414bbcb2039e9a23fab5" candidate_uuid: "27562c9910fa4819a8ecf91dafbf7ce1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e52364e0246c4bb4852c27e967e2f4c8" I0703 01:39:39.537107 9072 raft_consensus.cc:2782] T c6eb9324dee0414bbcb2039e9a23fab5 P e52364e0246c4bb4852c27e967e2f4c8 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:39:39.537111 9194 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "c6eb9324dee0414bbcb2039e9a23fab5" candidate_uuid: "27562c9910fa4819a8ecf91dafbf7ce1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38c3a80a7fa04fcebe7dea0e351bf9b2" I0703 01:39:39.537206 9072 raft_consensus.cc:2730] T c6eb9324dee0414bbcb2039e9a23fab5 P e52364e0246c4bb4852c27e967e2f4c8: Snoozing failure detection for 1.936s (vote granted) I0703 01:39:39.537250 9194 raft_consensus.cc:2782] T c6eb9324dee0414bbcb2039e9a23fab5 P 38c3a80a7fa04fcebe7dea0e351bf9b2 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:39:39.537351 9194 raft_consensus.cc:2730] T c6eb9324dee0414bbcb2039e9a23fab5 P 38c3a80a7fa04fcebe7dea0e351bf9b2: Snoozing failure detection for 1.819s (vote granted) I0703 01:39:39.539319 9072 raft_consensus.cc:2243] T c6eb9324dee0414bbcb2039e9a23fab5 P e52364e0246c4bb4852c27e967e2f4c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27562c9910fa4819a8ecf91dafbf7ce1 in term 1. I0703 01:39:39.539388 9194 raft_consensus.cc:2243] T c6eb9324dee0414bbcb2039e9a23fab5 P 38c3a80a7fa04fcebe7dea0e351bf9b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27562c9910fa4819a8ecf91dafbf7ce1 in term 1. I0703 01:39:39.539866 8881 leader_election.cc:379] T c6eb9324dee0414bbcb2039e9a23fab5 P 27562c9910fa4819a8ecf91dafbf7ce1 [CANDIDATE]: Term 1 election: Vote granted by peer e52364e0246c4bb4852c27e967e2f4c8 (127.5.240.195:46355) I0703 01:39:39.539960 8881 leader_election.cc:270] T c6eb9324dee0414bbcb2039e9a23fab5 P 27562c9910fa4819a8ecf91dafbf7ce1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:39:39.540060 8882 leader_election.cc:379] T c6eb9324dee0414bbcb2039e9a23fab5 P 27562c9910fa4819a8ecf91dafbf7ce1 [CANDIDATE]: Term 1 election: Vote granted by peer 38c3a80a7fa04fcebe7dea0e351bf9b2 (127.5.240.196:34814) I0703 01:39:39.540242 9264 raft_consensus.cc:2730] T c6eb9324dee0414bbcb2039e9a23fab5 P 27562c9910fa4819a8ecf91dafbf7ce1: Snoozing failure detection for 2.246s (election complete) I0703 01:39:39.540320 9264 raft_consensus.cc:2537] T c6eb9324dee0414bbcb2039e9a23fab5 P 27562c9910fa4819a8ecf91dafbf7ce1 [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:39:39.540850 9264 raft_consensus.cc:593] T c6eb9324dee0414bbcb2039e9a23fab5 P 27562c9910fa4819a8ecf91dafbf7ce1 [term 1 LEADER]: Becoming Leader. State: Replica: 27562c9910fa4819a8ecf91dafbf7ce1, State: Running, Role: LEADER I0703 01:39:39.541221 9264 consensus_queue.cc:204] T c6eb9324dee0414bbcb2039e9a23fab5 P 27562c9910fa4819a8ecf91dafbf7ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27562c9910fa4819a8ecf91dafbf7ce1" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38901 } } peers { permanent_uuid: "e52364e0246c4bb4852c27e967e2f4c8" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46355 } } peers { permanent_uuid: "38c3a80a7fa04fcebe7dea0e351bf9b2" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 34814 } } I0703 01:39:39.545650 8706 catalog_manager.cc:3798] T c6eb9324dee0414bbcb2039e9a23fab5 P 27562c9910fa4819a8ecf91dafbf7ce1 reported cstate change: term changed from 0 to 1, leader changed from to 27562c9910fa4819a8ecf91dafbf7ce1 (127.5.240.194). New cstate: current_term: 1 leader_uuid: "27562c9910fa4819a8ecf91dafbf7ce1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27562c9910fa4819a8ecf91dafbf7ce1" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e52364e0246c4bb4852c27e967e2f4c8" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38c3a80a7fa04fcebe7dea0e351bf9b2" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 34814 } health_report { overall_health: UNKNOWN } } } W0703 01:39:39.573906 8994 tablet.cc:1678] T c6eb9324dee0414bbcb2039e9a23fab5 P 27562c9910fa4819a8ecf91dafbf7ce1: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:39:39.599485 6083 external_mini_cluster.cc:490] 4 TS(s) registered with all masters I0703 01:39:39.600598 6083 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 6cf745d60d5445749567bc9afd503edb to finish bootstrapping I0703 01:39:39.607097 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 27562c9910fa4819a8ecf91dafbf7ce1 to finish bootstrapping I0703 01:39:39.611457 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver e52364e0246c4bb4852c27e967e2f4c8 to finish bootstrapping I0703 01:39:39.616127 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 38c3a80a7fa04fcebe7dea0e351bf9b2 to finish bootstrapping I0703 01:39:39.620383 6083 kudu-admin-test.cc:570] Waiting for Master to see the current replicas... I0703 01:39:39.621399 6083 kudu-admin-test.cc:577] Tablet locations: tablet_id: "c6eb9324dee0414bbcb2039e9a23fab5" replicas { ts_info { permanent_uuid: "27562c9910fa4819a8ecf91dafbf7ce1" rpc_addresses { host: "127.5.240.194" port: 38901 } } role: LEADER } replicas { ts_info { permanent_uuid: "e52364e0246c4bb4852c27e967e2f4c8" rpc_addresses { host: "127.5.240.195" port: 46355 } } role: FOLLOWER } replicas { ts_info { permanent_uuid: "38c3a80a7fa04fcebe7dea0e351bf9b2" rpc_addresses { host: "127.5.240.196" port: 34814 } } role: FOLLOWER } DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } I0703 01:39:39.936940 9072 raft_consensus.cc:1081] T c6eb9324dee0414bbcb2039e9a23fab5 P e52364e0246c4bb4852c27e967e2f4c8 [term 1 FOLLOWER]: Refusing update from remote peer 27562c9910fa4819a8ecf91dafbf7ce1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0703 01:39:39.937801 9264 consensus_queue.cc:962] T c6eb9324dee0414bbcb2039e9a23fab5 P 27562c9910fa4819a8ecf91dafbf7ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e52364e0246c4bb4852c27e967e2f4c8" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0703 01:39:39.943429 9194 raft_consensus.cc:1081] T c6eb9324dee0414bbcb2039e9a23fab5 P 38c3a80a7fa04fcebe7dea0e351bf9b2 [term 1 FOLLOWER]: Refusing update from remote peer 27562c9910fa4819a8ecf91dafbf7ce1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0703 01:39:39.944728 9268 consensus_queue.cc:962] T c6eb9324dee0414bbcb2039e9a23fab5 P 27562c9910fa4819a8ecf91dafbf7ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38c3a80a7fa04fcebe7dea0e351bf9b2" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 34814 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0703 01:39:39.950268 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 8874 I0703 01:39:39.950862 8990 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:39:39.995318 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 8874 W0703 01:39:40.003346 8690 connection.cc:512] server connection from 127.5.240.194:48290 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:48290): Connection reset by peer (error 104) W0703 01:39:40.003396 9006 connection.cc:512] server connection from 127.5.240.194:49492 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:49492): Connection reset by peer (error 104) W0703 01:39:40.004017 9126 connection.cc:512] server connection from 127.5.240.194:54561 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:54561): Connection reset by peer (error 104) I0703 01:39:40.004122 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master pid 8681 I0703 01:39:40.004920 8737 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:39:40.073510 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master with pid 8681 W0703 01:39:40.086329 9125 connection.cc:512] client connection to 127.5.240.254:38609 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.254:38609): Connection reset by peer (error 104) W0703 01:39:40.086329 8758 connection.cc:512] client connection to 127.5.240.254:38609 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.254:38609): Connection reset by peer (error 104) W0703 01:39:40.086743 9003 connection.cc:512] client connection to 127.5.240.254:38609 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.254:38609): Connection reset by peer (error 104) I0703 01:39:40.087345 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:38609 --webserver_interface=127.5.240.254 --webserver_port=47735 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:39:40.267472 9278 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:39:40.267901 9278 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:39:40.270133 9278 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0703 01:39:40.270179 9278 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:39:40.270237 9278 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0703 01:39:40.270283 9278 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0703 01:39:40.278007 9278 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:38609 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.5.240.254 --webserver_port=47735 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:39:40.283701 9278 master_main.cc:80] Initializing master server... I0703 01:39:40.284797 9278 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 143708us W0703 01:39:40.285430 8870 heartbeater.cc:587] Failed to heartbeat to 127.5.240.254:38609: Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.240.254:38609: connect: Connection refused (error 111) I0703 01:39:40.285933 9278 fs_manager.cc:261] Metadata directory not provided I0703 01:39:40.286002 9278 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:39:40.293350 9278 fs_manager.cc:397] Time spent opening directory manager: real 0.005s user 0.002s sys 0.002s I0703 01:39:40.294198 9278 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:39:40.294631 9278 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:39:40.298210 9278 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s I0703 01:39:40.298285 9278 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal uuid: "47e88b229dc046b1ac2d0417718d1d24" format_stamp: "Formatted at 2018-07-03 08:39:37 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:40.299156 9278 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:39:40.332175 9278 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:39:40.333626 9278 master_main.cc:83] Starting Master server... I0703 01:39:40.370268 9278 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.254:38609 I0703 01:39:40.370751 9278 webserver.cc:175] Starting webserver on 127.5.240.254:47735 I0703 01:39:40.370798 9278 webserver.cc:186] Document root disabled I0703 01:39:40.371510 9278 webserver.cc:313] Webserver started. Bound to: http://127.5.240.254:47735/ I0703 01:39:40.372195 9278 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb I0703 01:39:40.377939 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master as pid 9278 I0703 01:39:40.378069 6083 kudu-admin-test.cc:594] Forcing unsafe config change on tserver 38c3a80a7fa04fcebe7dea0e351bf9b2 I0703 01:39:40.380508 9338 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 47e88b229dc046b1ac2d0417718d1d24: Bootstrap starting. I0703 01:39:40.392496 9338 log.cc:525] T 00000000000000000000000000000000 P 47e88b229dc046b1ac2d0417718d1d24: Log is configured to *not* fsync() on all Append() calls I0703 01:39:40.401682 9338 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 47e88b229dc046b1ac2d0417718d1d24: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=1} inserts{seen=4 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:39:40.402091 9338 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 47e88b229dc046b1ac2d0417718d1d24: Bootstrap complete. I0703 01:39:40.411803 9338 raft_consensus.cc:325] T 00000000000000000000000000000000 P 47e88b229dc046b1ac2d0417718d1d24 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47e88b229dc046b1ac2d0417718d1d24" member_type: VOTER } I0703 01:39:40.412530 9338 raft_consensus.cc:628] T 00000000000000000000000000000000 P 47e88b229dc046b1ac2d0417718d1d24 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47e88b229dc046b1ac2d0417718d1d24, State: Initialized, Role: FOLLOWER I0703 01:39:40.412986 9338 consensus_queue.cc:227] T 00000000000000000000000000000000 P 47e88b229dc046b1ac2d0417718d1d24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47e88b229dc046b1ac2d0417718d1d24" member_type: VOTER } I0703 01:39:40.413127 9338 raft_consensus.cc:365] T 00000000000000000000000000000000 P 47e88b229dc046b1ac2d0417718d1d24 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:39:40.413211 9338 raft_consensus.cc:457] T 00000000000000000000000000000000 P 47e88b229dc046b1ac2d0417718d1d24 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:39:40.413293 9338 raft_consensus.cc:2730] T 00000000000000000000000000000000 P 47e88b229dc046b1ac2d0417718d1d24: Snoozing failure detection for 1.705s (starting election) I0703 01:39:40.413383 9338 raft_consensus.cc:2782] T 00000000000000000000000000000000 P 47e88b229dc046b1ac2d0417718d1d24 [term 1 FOLLOWER]: Advancing to term 2 I0703 01:39:40.416918 9338 raft_consensus.cc:479] T 00000000000000000000000000000000 P 47e88b229dc046b1ac2d0417718d1d24 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47e88b229dc046b1ac2d0417718d1d24" member_type: VOTER } I0703 01:39:40.417279 9338 leader_election.cc:270] T 00000000000000000000000000000000 P 47e88b229dc046b1ac2d0417718d1d24 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0703 01:39:40.418876 9341 raft_consensus.cc:2730] T 00000000000000000000000000000000 P 47e88b229dc046b1ac2d0417718d1d24: Snoozing failure detection for 2.087s (election complete) I0703 01:39:40.419180 9341 raft_consensus.cc:2537] T 00000000000000000000000000000000 P 47e88b229dc046b1ac2d0417718d1d24 [term 2 FOLLOWER]: Leader election won for term 2 I0703 01:39:40.420708 9341 raft_consensus.cc:593] T 00000000000000000000000000000000 P 47e88b229dc046b1ac2d0417718d1d24 [term 2 LEADER]: Becoming Leader. State: Replica: 47e88b229dc046b1ac2d0417718d1d24, State: Running, Role: LEADER I0703 01:39:40.421170 9341 consensus_queue.cc:204] T 00000000000000000000000000000000 P 47e88b229dc046b1ac2d0417718d1d24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47e88b229dc046b1ac2d0417718d1d24" member_type: VOTER } I0703 01:39:40.423454 9338 sys_catalog.cc:337] T 00000000000000000000000000000000 P 47e88b229dc046b1ac2d0417718d1d24 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 2 leader_uuid: "47e88b229dc046b1ac2d0417718d1d24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47e88b229dc046b1ac2d0417718d1d24" member_type: VOTER } } I0703 01:39:40.423866 9338 sys_catalog.cc:340] T 00000000000000000000000000000000 P 47e88b229dc046b1ac2d0417718d1d24 [sys.catalog]: This master's current role is: LEADER I0703 01:39:40.423347 9342 sys_catalog.cc:337] T 00000000000000000000000000000000 P 47e88b229dc046b1ac2d0417718d1d24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "47e88b229dc046b1ac2d0417718d1d24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47e88b229dc046b1ac2d0417718d1d24" member_type: VOTER } } I0703 01:39:40.424291 9343 sys_catalog.cc:337] T 00000000000000000000000000000000 P 47e88b229dc046b1ac2d0417718d1d24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47e88b229dc046b1ac2d0417718d1d24. Latest consensus state: current_term: 2 leader_uuid: "47e88b229dc046b1ac2d0417718d1d24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47e88b229dc046b1ac2d0417718d1d24" member_type: VOTER } } I0703 01:39:40.424886 9342 sys_catalog.cc:340] T 00000000000000000000000000000000 P 47e88b229dc046b1ac2d0417718d1d24 [sys.catalog]: This master's current role is: LEADER I0703 01:39:40.425333 9343 sys_catalog.cc:340] T 00000000000000000000000000000000 P 47e88b229dc046b1ac2d0417718d1d24 [sys.catalog]: This master's current role is: LEADER I0703 01:39:40.426429 9338 sys_catalog.cc:424] T 00000000000000000000000000000000 P 47e88b229dc046b1ac2d0417718d1d24 [sys.catalog]: configured and running, proceeding with master startup. I0703 01:39:40.426779 9345 catalog_manager.cc:1006] Loading table and tablet metadata into memory... I0703 01:39:40.436661 9346 catalog_manager.cc:505] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting I0703 01:39:40.437072 9278 master_main.cc:86] Master server successfully started. I0703 01:39:40.443389 9345 catalog_manager.cc:323] Loaded metadata for table TestTable [id=3f0cae68e9f5467da8f010c417d00d44] I0703 01:39:40.445688 9345 catalog_manager.cc:377] Loaded metadata for tablet c6eb9324dee0414bbcb2039e9a23fab5 (table TestTable [id=3f0cae68e9f5467da8f010c417d00d44]) I0703 01:39:40.445932 9345 catalog_manager.cc:1017] Initializing Kudu internal certificate authority... I0703 01:39:40.450687 9345 catalog_manager.cc:1026] Loading token signing keys... I0703 01:39:40.452121 9345 catalog_manager.cc:4113] T 00000000000000000000000000000000 P 47e88b229dc046b1ac2d0417718d1d24: Loaded TSK: 0 W0703 01:39:40.599056 9339 flags.cc:406] Enabled unsafe flag: --never_fsync=true I0703 01:39:40.649766 9194 tablet_service.cc:1077] Received UnsafeChangeConfig RPC: dest_uuid: "38c3a80a7fa04fcebe7dea0e351bf9b2" tablet_id: "c6eb9324dee0414bbcb2039e9a23fab5" caller_id: "kudu-tools" new_config { peers { permanent_uuid: "e52364e0246c4bb4852c27e967e2f4c8" } peers { permanent_uuid: "38c3a80a7fa04fcebe7dea0e351bf9b2" } } from {username='jenkins'} at 127.5.240.196:54175 W0703 01:39:40.650202 9194 raft_consensus.cc:2017] T c6eb9324dee0414bbcb2039e9a23fab5 P 38c3a80a7fa04fcebe7dea0e351bf9b2 [term 1 FOLLOWER]: PROCEEDING WITH UNSAFE CONFIG CHANGE ON THIS SERVER, COMMITTED CONFIG: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27562c9910fa4819a8ecf91dafbf7ce1" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38901 } } peers { permanent_uuid: "e52364e0246c4bb4852c27e967e2f4c8" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46355 } } peers { permanent_uuid: "38c3a80a7fa04fcebe7dea0e351bf9b2" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 34814 } }NEW CONFIG: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e52364e0246c4bb4852c27e967e2f4c8" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46355 } } peers { permanent_uuid: "38c3a80a7fa04fcebe7dea0e351bf9b2" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 34814 } } unsafe_config_change: true I0703 01:39:40.650465 9194 raft_consensus.cc:2782] T c6eb9324dee0414bbcb2039e9a23fab5 P 38c3a80a7fa04fcebe7dea0e351bf9b2 [term 1 FOLLOWER]: Advancing to term 2 I0703 01:39:40.667018 9305 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "38c3a80a7fa04fcebe7dea0e351bf9b2" instance_seqno: 1530607179245222) as {username='jenkins'} at 127.5.240.196:42025; Asking this server to re-register. I0703 01:39:40.668272 9237 heartbeater.cc:426] Registering TS with master... I0703 01:39:40.668419 9237 heartbeater.cc:475] Master 127.5.240.254:38609 requested a full tablet report, sending... I0703 01:39:40.669658 9305 ts_manager.cc:98] Registered new tserver with Master: 38c3a80a7fa04fcebe7dea0e351bf9b2 (127.5.240.196:34814) I0703 01:39:40.949275 9305 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "e52364e0246c4bb4852c27e967e2f4c8" instance_seqno: 1530607178919291) as {username='jenkins'} at 127.5.240.195:55865; Asking this server to re-register. I0703 01:39:40.949880 9115 heartbeater.cc:426] Registering TS with master... I0703 01:39:40.950014 9115 heartbeater.cc:475] Master 127.5.240.254:38609 requested a full tablet report, sending... I0703 01:39:40.950824 9305 ts_manager.cc:98] Registered new tserver with Master: e52364e0246c4bb4852c27e967e2f4c8 (127.5.240.195:46355) I0703 01:39:41.290618 8870 heartbeater.cc:346] Connected to a master server at 127.5.240.254:38609 I0703 01:39:41.291342 9305 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "6cf745d60d5445749567bc9afd503edb" instance_seqno: 1530607178231533) as {username='jenkins'} at 127.5.240.193:57351; Asking this server to re-register. I0703 01:39:41.291883 8870 heartbeater.cc:426] Registering TS with master... I0703 01:39:41.292002 8870 heartbeater.cc:475] Master 127.5.240.254:38609 requested a full tablet report, sending... I0703 01:39:41.292762 9305 ts_manager.cc:98] Registered new tserver with Master: 6cf745d60d5445749567bc9afd503edb (127.5.240.193:54395) I0703 01:39:41.444200 9370 raft_consensus.cc:457] T c6eb9324dee0414bbcb2039e9a23fab5 P e52364e0246c4bb4852c27e967e2f4c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 27562c9910fa4819a8ecf91dafbf7ce1) I0703 01:39:41.444283 9370 raft_consensus.cc:2730] T c6eb9324dee0414bbcb2039e9a23fab5 P e52364e0246c4bb4852c27e967e2f4c8: Snoozing failure detection for 1.765s (starting election) I0703 01:39:41.444375 9370 raft_consensus.cc:479] T c6eb9324dee0414bbcb2039e9a23fab5 P e52364e0246c4bb4852c27e967e2f4c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27562c9910fa4819a8ecf91dafbf7ce1" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38901 } } peers { permanent_uuid: "e52364e0246c4bb4852c27e967e2f4c8" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46355 } } peers { permanent_uuid: "38c3a80a7fa04fcebe7dea0e351bf9b2" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 34814 } } I0703 01:39:41.445122 9370 leader_election.cc:243] T c6eb9324dee0414bbcb2039e9a23fab5 P e52364e0246c4bb4852c27e967e2f4c8 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 27562c9910fa4819a8ecf91dafbf7ce1 (127.5.240.194:38901) I0703 01:39:41.445462 9370 leader_election.cc:243] T c6eb9324dee0414bbcb2039e9a23fab5 P e52364e0246c4bb4852c27e967e2f4c8 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 38c3a80a7fa04fcebe7dea0e351bf9b2 (127.5.240.196:34814) W0703 01:39:41.446131 9003 leader_election.cc:299] T c6eb9324dee0414bbcb2039e9a23fab5 P e52364e0246c4bb4852c27e967e2f4c8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 27562c9910fa4819a8ecf91dafbf7ce1 (127.5.240.194:38901): Network error: Client connection negotiation failed: client connection to 127.5.240.194:38901: connect: Connection refused (error 111) I0703 01:39:41.450213 9194 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "c6eb9324dee0414bbcb2039e9a23fab5" candidate_uuid: "e52364e0246c4bb4852c27e967e2f4c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "38c3a80a7fa04fcebe7dea0e351bf9b2" is_pre_election: true I0703 01:39:41.450392 9194 raft_consensus.cc:2204] T c6eb9324dee0414bbcb2039e9a23fab5 P 38c3a80a7fa04fcebe7dea0e351bf9b2 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e52364e0246c4bb4852c27e967e2f4c8 for term 2 because replica is either leader or believes a valid leader to be alive. I0703 01:39:41.451283 9004 leader_election.cc:393] T c6eb9324dee0414bbcb2039e9a23fab5 P e52364e0246c4bb4852c27e967e2f4c8 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 38c3a80a7fa04fcebe7dea0e351bf9b2 (127.5.240.196:34814). Message: Invalid argument: T c6eb9324dee0414bbcb2039e9a23fab5 P 38c3a80a7fa04fcebe7dea0e351bf9b2 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e52364e0246c4bb4852c27e967e2f4c8 for term 2 because replica is either leader or believes a valid leader to be alive. I0703 01:39:41.451370 9004 leader_election.cc:270] T c6eb9324dee0414bbcb2039e9a23fab5 P e52364e0246c4bb4852c27e967e2f4c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0703 01:39:41.451653 9370 raft_consensus.cc:2730] T c6eb9324dee0414bbcb2039e9a23fab5 P e52364e0246c4bb4852c27e967e2f4c8: Snoozing failure detection for 1.671s (election complete) I0703 01:39:41.451792 9370 raft_consensus.cc:2782] T c6eb9324dee0414bbcb2039e9a23fab5 P e52364e0246c4bb4852c27e967e2f4c8 [term 1 FOLLOWER]: Advancing to term 2 I0703 01:39:41.453127 9370 raft_consensus.cc:2485] T c6eb9324dee0414bbcb2039e9a23fab5 P e52364e0246c4bb4852c27e967e2f4c8 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0703 01:39:42.153513 9372 raft_consensus.cc:457] T c6eb9324dee0414bbcb2039e9a23fab5 P 38c3a80a7fa04fcebe7dea0e351bf9b2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader kudu-tools) I0703 01:39:42.153612 9372 raft_consensus.cc:2730] T c6eb9324dee0414bbcb2039e9a23fab5 P 38c3a80a7fa04fcebe7dea0e351bf9b2: Snoozing failure detection for 1.554s (starting election) I0703 01:39:42.153705 9372 raft_consensus.cc:479] T c6eb9324dee0414bbcb2039e9a23fab5 P 38c3a80a7fa04fcebe7dea0e351bf9b2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e52364e0246c4bb4852c27e967e2f4c8" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46355 } } peers { permanent_uuid: "38c3a80a7fa04fcebe7dea0e351bf9b2" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 34814 } } unsafe_config_change: true I0703 01:39:42.154325 9372 leader_election.cc:243] T c6eb9324dee0414bbcb2039e9a23fab5 P 38c3a80a7fa04fcebe7dea0e351bf9b2 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer e52364e0246c4bb4852c27e967e2f4c8 (127.5.240.195:46355) I0703 01:39:42.158730 9072 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "c6eb9324dee0414bbcb2039e9a23fab5" candidate_uuid: "38c3a80a7fa04fcebe7dea0e351bf9b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e52364e0246c4bb4852c27e967e2f4c8" is_pre_election: true I0703 01:39:42.158869 9072 raft_consensus.cc:2730] T c6eb9324dee0414bbcb2039e9a23fab5 P e52364e0246c4bb4852c27e967e2f4c8: Snoozing failure detection for 3.170s (vote granted) I0703 01:39:42.158936 9072 raft_consensus.cc:2243] T c6eb9324dee0414bbcb2039e9a23fab5 P e52364e0246c4bb4852c27e967e2f4c8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38c3a80a7fa04fcebe7dea0e351bf9b2 in term 2. I0703 01:39:42.159385 9125 leader_election.cc:379] T c6eb9324dee0414bbcb2039e9a23fab5 P 38c3a80a7fa04fcebe7dea0e351bf9b2 [CANDIDATE]: Term 3 pre-election: Vote granted by peer e52364e0246c4bb4852c27e967e2f4c8 (127.5.240.195:46355) I0703 01:39:42.159462 9125 leader_election.cc:270] T c6eb9324dee0414bbcb2039e9a23fab5 P 38c3a80a7fa04fcebe7dea0e351bf9b2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. I0703 01:39:42.159734 9372 raft_consensus.cc:2730] T c6eb9324dee0414bbcb2039e9a23fab5 P 38c3a80a7fa04fcebe7dea0e351bf9b2: Snoozing failure detection for 1.521s (election complete) I0703 01:39:42.159827 9372 raft_consensus.cc:2537] T c6eb9324dee0414bbcb2039e9a23fab5 P 38c3a80a7fa04fcebe7dea0e351bf9b2 [term 2 FOLLOWER]: Leader pre-election won for term 3 I0703 01:39:42.159881 9372 raft_consensus.cc:457] T c6eb9324dee0414bbcb2039e9a23fab5 P 38c3a80a7fa04fcebe7dea0e351bf9b2 [term 2 FOLLOWER]: Starting leader election (detected failure of leader kudu-tools) I0703 01:39:42.159930 9372 raft_consensus.cc:2730] T c6eb9324dee0414bbcb2039e9a23fab5 P 38c3a80a7fa04fcebe7dea0e351bf9b2: Snoozing failure detection for 1.511s (starting election) I0703 01:39:42.159987 9372 raft_consensus.cc:2782] T c6eb9324dee0414bbcb2039e9a23fab5 P 38c3a80a7fa04fcebe7dea0e351bf9b2 [term 2 FOLLOWER]: Advancing to term 3 I0703 01:39:42.161355 9372 raft_consensus.cc:479] T c6eb9324dee0414bbcb2039e9a23fab5 P 38c3a80a7fa04fcebe7dea0e351bf9b2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e52364e0246c4bb4852c27e967e2f4c8" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46355 } } peers { permanent_uuid: "38c3a80a7fa04fcebe7dea0e351bf9b2" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 34814 } } unsafe_config_change: true I0703 01:39:42.161660 9372 leader_election.cc:243] T c6eb9324dee0414bbcb2039e9a23fab5 P 38c3a80a7fa04fcebe7dea0e351bf9b2 [CANDIDATE]: Term 3 election: Requesting vote from peer e52364e0246c4bb4852c27e967e2f4c8 (127.5.240.195:46355) I0703 01:39:42.162278 9072 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "c6eb9324dee0414bbcb2039e9a23fab5" candidate_uuid: "38c3a80a7fa04fcebe7dea0e351bf9b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e52364e0246c4bb4852c27e967e2f4c8" I0703 01:39:42.162385 9072 raft_consensus.cc:2782] T c6eb9324dee0414bbcb2039e9a23fab5 P e52364e0246c4bb4852c27e967e2f4c8 [term 2 FOLLOWER]: Advancing to term 3 I0703 01:39:42.162449 9072 raft_consensus.cc:2730] T c6eb9324dee0414bbcb2039e9a23fab5 P e52364e0246c4bb4852c27e967e2f4c8: Snoozing failure detection for 2.946s (vote granted) I0703 01:39:42.163745 9072 raft_consensus.cc:2243] T c6eb9324dee0414bbcb2039e9a23fab5 P e52364e0246c4bb4852c27e967e2f4c8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38c3a80a7fa04fcebe7dea0e351bf9b2 in term 3. I0703 01:39:42.164088 9125 leader_election.cc:379] T c6eb9324dee0414bbcb2039e9a23fab5 P 38c3a80a7fa04fcebe7dea0e351bf9b2 [CANDIDATE]: Term 3 election: Vote granted by peer e52364e0246c4bb4852c27e967e2f4c8 (127.5.240.195:46355) I0703 01:39:42.164147 9125 leader_election.cc:270] T c6eb9324dee0414bbcb2039e9a23fab5 P 38c3a80a7fa04fcebe7dea0e351bf9b2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. I0703 01:39:42.164314 9372 raft_consensus.cc:2730] T c6eb9324dee0414bbcb2039e9a23fab5 P 38c3a80a7fa04fcebe7dea0e351bf9b2: Snoozing failure detection for 1.500s (election complete) I0703 01:39:42.164379 9372 raft_consensus.cc:2537] T c6eb9324dee0414bbcb2039e9a23fab5 P 38c3a80a7fa04fcebe7dea0e351bf9b2 [term 3 FOLLOWER]: Leader election won for term 3 I0703 01:39:42.164680 9372 raft_consensus.cc:593] T c6eb9324dee0414bbcb2039e9a23fab5 P 38c3a80a7fa04fcebe7dea0e351bf9b2 [term 3 LEADER]: Becoming Leader. State: Replica: 38c3a80a7fa04fcebe7dea0e351bf9b2, State: Running, Role: LEADER I0703 01:39:42.165001 9372 consensus_queue.cc:204] T c6eb9324dee0414bbcb2039e9a23fab5 P 38c3a80a7fa04fcebe7dea0e351bf9b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 0, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e52364e0246c4bb4852c27e967e2f4c8" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46355 } } peers { permanent_uuid: "38c3a80a7fa04fcebe7dea0e351bf9b2" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 34814 } } unsafe_config_change: true I0703 01:39:42.167646 9305 catalog_manager.cc:3798] T c6eb9324dee0414bbcb2039e9a23fab5 P 38c3a80a7fa04fcebe7dea0e351bf9b2 reported cstate change: term changed from 1 to 3, leader changed from 27562c9910fa4819a8ecf91dafbf7ce1 (127.5.240.194) to 38c3a80a7fa04fcebe7dea0e351bf9b2 (127.5.240.196), now has a pending config: VOTER e52364e0246c4bb4852c27e967e2f4c8 (127.5.240.195), VOTER 38c3a80a7fa04fcebe7dea0e351bf9b2 (127.5.240.196). New cstate: current_term: 3 leader_uuid: "38c3a80a7fa04fcebe7dea0e351bf9b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27562c9910fa4819a8ecf91dafbf7ce1" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38901 } } peers { permanent_uuid: "e52364e0246c4bb4852c27e967e2f4c8" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38c3a80a7fa04fcebe7dea0e351bf9b2" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 34814 } health_report { overall_health: HEALTHY } } } pending_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e52364e0246c4bb4852c27e967e2f4c8" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46355 } } peers { permanent_uuid: "38c3a80a7fa04fcebe7dea0e351bf9b2" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 34814 } } unsafe_config_change: true } I0703 01:39:42.699560 9072 raft_consensus.cc:1081] T c6eb9324dee0414bbcb2039e9a23fab5 P e52364e0246c4bb4852c27e967e2f4c8 [term 3 FOLLOWER]: Refusing update from remote peer 38c3a80a7fa04fcebe7dea0e351bf9b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch) I0703 01:39:42.700189 9381 consensus_queue.cc:962] T c6eb9324dee0414bbcb2039e9a23fab5 P 38c3a80a7fa04fcebe7dea0e351bf9b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e52364e0246c4bb4852c27e967e2f4c8" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s I0703 01:39:42.703434 9384 raft_consensus.cc:2684] T c6eb9324dee0414bbcb2039e9a23fab5 P 38c3a80a7fa04fcebe7dea0e351bf9b2 [term 3 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, VOTER 27562c9910fa4819a8ecf91dafbf7ce1 (127.5.240.194) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e52364e0246c4bb4852c27e967e2f4c8" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46355 } } peers { permanent_uuid: "38c3a80a7fa04fcebe7dea0e351bf9b2" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 34814 } } unsafe_config_change: true } I0703 01:39:42.704211 9072 raft_consensus.cc:2684] T c6eb9324dee0414bbcb2039e9a23fab5 P e52364e0246c4bb4852c27e967e2f4c8 [term 3 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, VOTER 27562c9910fa4819a8ecf91dafbf7ce1 (127.5.240.194) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e52364e0246c4bb4852c27e967e2f4c8" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46355 } } peers { permanent_uuid: "38c3a80a7fa04fcebe7dea0e351bf9b2" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 34814 } } unsafe_config_change: true } I0703 01:39:42.706753 9305 catalog_manager.cc:3798] T c6eb9324dee0414bbcb2039e9a23fab5 P 38c3a80a7fa04fcebe7dea0e351bf9b2 reported cstate change: config changed from index -1 to 2, VOTER 27562c9910fa4819a8ecf91dafbf7ce1 (127.5.240.194) evicted, no longer has a pending config: VOTER e52364e0246c4bb4852c27e967e2f4c8 (127.5.240.195), VOTER 38c3a80a7fa04fcebe7dea0e351bf9b2 (127.5.240.196). New cstate: current_term: 3 leader_uuid: "38c3a80a7fa04fcebe7dea0e351bf9b2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e52364e0246c4bb4852c27e967e2f4c8" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38c3a80a7fa04fcebe7dea0e351bf9b2" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 34814 } health_report { overall_health: HEALTHY } } unsafe_config_change: true } I0703 01:39:42.710373 9305 catalog_manager.cc:2903] Scheduling retry of c6eb9324dee0414bbcb2039e9a23fab5 Delete Tablet RPC for TS=27562c9910fa4819a8ecf91dafbf7ce1 with a delay of 26 ms (attempt = 1) W0703 01:39:42.710537 9305 catalog_manager.cc:3957] Failed to send c6eb9324dee0414bbcb2039e9a23fab5 Delete Tablet RPC for TS=27562c9910fa4819a8ecf91dafbf7ce1: Not found: failed to reset TS proxy: Could not find TS for UUID 27562c9910fa4819a8ecf91dafbf7ce1 I0703 01:39:42.710917 9305 catalog_manager.cc:3437] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet c6eb9324dee0414bbcb2039e9a23fab5 (attempt 1) I0703 01:39:42.713246 9304 catalog_manager.cc:3437] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet c6eb9324dee0414bbcb2039e9a23fab5 (attempt 1) I0703 01:39:42.717720 9194 consensus_peers.cc:455] T c6eb9324dee0414bbcb2039e9a23fab5 P 38c3a80a7fa04fcebe7dea0e351bf9b2 -> Peer e52364e0246c4bb4852c27e967e2f4c8 (127.5.240.195:46355): Closing peer: e52364e0246c4bb4852c27e967e2f4c8 I0703 01:39:42.717939 9194 consensus_queue.cc:204] T c6eb9324dee0414bbcb2039e9a23fab5 P 38c3a80a7fa04fcebe7dea0e351bf9b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 0, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e52364e0246c4bb4852c27e967e2f4c8" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46355 } } peers { permanent_uuid: "38c3a80a7fa04fcebe7dea0e351bf9b2" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 34814 } } peers { permanent_uuid: "6cf745d60d5445749567bc9afd503edb" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 54395 } attrs { promote: true } } unsafe_config_change: true I0703 01:39:42.719959 9072 raft_consensus.cc:1081] T c6eb9324dee0414bbcb2039e9a23fab5 P e52364e0246c4bb4852c27e967e2f4c8 [term 3 FOLLOWER]: Refusing update from remote peer 38c3a80a7fa04fcebe7dea0e351bf9b2: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch) I0703 01:39:42.720881 9384 consensus_queue.cc:962] T c6eb9324dee0414bbcb2039e9a23fab5 P 38c3a80a7fa04fcebe7dea0e351bf9b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e52364e0246c4bb4852c27e967e2f4c8" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I0703 01:39:42.721652 9287 catalog_manager.cc:3369] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet c6eb9324dee0414bbcb2039e9a23fab5 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:NON_VOTER failed with leader 38c3a80a7fa04fcebe7dea0e351bf9b2 (127.5.240.196:34814) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time. Committed config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e52364e0246c4bb4852c27e967e2f4c8" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46355 } } peers { permanent_uuid: "38c3a80a7fa04fcebe7dea0e351bf9b2" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 34814 } } unsafe_config_change: true. Pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e52364e0246c4bb4852c27e967e2f4c8" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46355 } } peers { permanent_uuid: "38c3a80a7fa04fcebe7dea0e351bf9b2" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 34814 } } peers { permanent_uuid: "6cf745d60d5445749567bc9afd503edb" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 54395 } attrs { promote: true } } unsafe_config_change: true I0703 01:39:42.721755 9287 catalog_manager.cc:2903] Scheduling retry of ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet c6eb9324dee0414bbcb2039e9a23fab5 with cas_config_opid_index 2 with a delay of 21 ms (attempt = 1) I0703 01:39:42.723031 9381 raft_consensus.cc:2684] T c6eb9324dee0414bbcb2039e9a23fab5 P 38c3a80a7fa04fcebe7dea0e351bf9b2 [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index 2 to 4, NON_VOTER 6cf745d60d5445749567bc9afd503edb (127.5.240.193) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e52364e0246c4bb4852c27e967e2f4c8" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46355 } } peers { permanent_uuid: "38c3a80a7fa04fcebe7dea0e351bf9b2" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 34814 } } peers { permanent_uuid: "6cf745d60d5445749567bc9afd503edb" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 54395 } attrs { promote: true } } unsafe_config_change: true } I0703 01:39:42.724318 9072 raft_consensus.cc:2684] T c6eb9324dee0414bbcb2039e9a23fab5 P e52364e0246c4bb4852c27e967e2f4c8 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 2 to 4, NON_VOTER 6cf745d60d5445749567bc9afd503edb (127.5.240.193) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e52364e0246c4bb4852c27e967e2f4c8" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46355 } } peers { permanent_uuid: "38c3a80a7fa04fcebe7dea0e351bf9b2" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 34814 } } peers { permanent_uuid: "6cf745d60d5445749567bc9afd503edb" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 54395 } attrs { promote: true } } unsafe_config_change: true } W0703 01:39:42.725785 9128 consensus_peers.cc:433] T c6eb9324dee0414bbcb2039e9a23fab5 P 38c3a80a7fa04fcebe7dea0e351bf9b2 -> Peer 6cf745d60d5445749567bc9afd503edb (127.5.240.193:54395): Couldn't send request to peer 6cf745d60d5445749567bc9afd503edb for tablet c6eb9324dee0414bbcb2039e9a23fab5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c6eb9324dee0414bbcb2039e9a23fab5. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:39:42.726835 9287 catalog_manager.cc:3352] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet c6eb9324dee0414bbcb2039e9a23fab5 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0703 01:39:42.727754 9304 catalog_manager.cc:3798] T c6eb9324dee0414bbcb2039e9a23fab5 P e52364e0246c4bb4852c27e967e2f4c8 reported cstate change: config changed from index 2 to 4, NON_VOTER 6cf745d60d5445749567bc9afd503edb (127.5.240.193) added. New cstate: current_term: 3 leader_uuid: "38c3a80a7fa04fcebe7dea0e351bf9b2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e52364e0246c4bb4852c27e967e2f4c8" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46355 } } peers { permanent_uuid: "38c3a80a7fa04fcebe7dea0e351bf9b2" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 34814 } } peers { permanent_uuid: "6cf745d60d5445749567bc9afd503edb" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 54395 } attrs { promote: true } } unsafe_config_change: true } I0703 01:39:42.736912 9286 catalog_manager.cc:2903] Scheduling retry of c6eb9324dee0414bbcb2039e9a23fab5 Delete Tablet RPC for TS=27562c9910fa4819a8ecf91dafbf7ce1 with a delay of 52 ms (attempt = 2) W0703 01:39:42.736974 9286 catalog_manager.cc:2922] Async tablet task c6eb9324dee0414bbcb2039e9a23fab5 Delete Tablet RPC for TS=27562c9910fa4819a8ecf91dafbf7ce1 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 27562c9910fa4819a8ecf91dafbf7ce1 I0703 01:39:42.743377 9287 catalog_manager.cc:3385] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet c6eb9324dee0414bbcb2039e9a23fab5 with cas_config_opid_index 2: aborting the task: latest config opid_index 4; task opid_index 2 I0703 01:39:42.789392 9286 catalog_manager.cc:2903] Scheduling retry of c6eb9324dee0414bbcb2039e9a23fab5 Delete Tablet RPC for TS=27562c9910fa4819a8ecf91dafbf7ce1 with a delay of 82 ms (attempt = 3) W0703 01:39:42.789453 9286 catalog_manager.cc:2922] Async tablet task c6eb9324dee0414bbcb2039e9a23fab5 Delete Tablet RPC for TS=27562c9910fa4819a8ecf91dafbf7ce1 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 27562c9910fa4819a8ecf91dafbf7ce1 I0703 01:39:42.871969 9286 catalog_manager.cc:2903] Scheduling retry of c6eb9324dee0414bbcb2039e9a23fab5 Delete Tablet RPC for TS=27562c9910fa4819a8ecf91dafbf7ce1 with a delay of 139 ms (attempt = 4) W0703 01:39:42.872025 9286 catalog_manager.cc:2922] Async tablet task c6eb9324dee0414bbcb2039e9a23fab5 Delete Tablet RPC for TS=27562c9910fa4819a8ecf91dafbf7ce1 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 27562c9910fa4819a8ecf91dafbf7ce1 I0703 01:39:43.012017 9286 catalog_manager.cc:2903] Scheduling retry of c6eb9324dee0414bbcb2039e9a23fab5 Delete Tablet RPC for TS=27562c9910fa4819a8ecf91dafbf7ce1 with a delay of 298 ms (attempt = 5) W0703 01:39:43.012089 9286 catalog_manager.cc:2922] Async tablet task c6eb9324dee0414bbcb2039e9a23fab5 Delete Tablet RPC for TS=27562c9910fa4819a8ecf91dafbf7ce1 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 27562c9910fa4819a8ecf91dafbf7ce1 I0703 01:39:43.258360 9393 ts_tablet_manager.cc:669] T c6eb9324dee0414bbcb2039e9a23fab5 P 6cf745d60d5445749567bc9afd503edb: Initiating tablet copy from peer 38c3a80a7fa04fcebe7dea0e351bf9b2 (127.5.240.196:34814) I0703 01:39:43.258644 9393 tablet_copy_client.cc:239] T c6eb9324dee0414bbcb2039e9a23fab5 P 6cf745d60d5445749567bc9afd503edb: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.196:34814 I0703 01:39:43.263386 9214 tablet_copy_service.cc:135] P 38c3a80a7fa04fcebe7dea0e351bf9b2: Received BeginTabletCopySession request for tablet c6eb9324dee0414bbcb2039e9a23fab5 from peer 6cf745d60d5445749567bc9afd503edb ({username='jenkins'} at 127.5.240.193:52566) I0703 01:39:43.263515 9214 tablet_copy_service.cc:156] P 38c3a80a7fa04fcebe7dea0e351bf9b2: Beginning new tablet copy session on tablet c6eb9324dee0414bbcb2039e9a23fab5 from peer 6cf745d60d5445749567bc9afd503edb at {username='jenkins'} at 127.5.240.193:52566: session id = 6cf745d60d5445749567bc9afd503edb-c6eb9324dee0414bbcb2039e9a23fab5 I0703 01:39:43.265468 9214 tablet_copy_source_session.cc:202] T c6eb9324dee0414bbcb2039e9a23fab5 P 38c3a80a7fa04fcebe7dea0e351bf9b2: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:39:43.268119 9393 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet c6eb9324dee0414bbcb2039e9a23fab5. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:39:43.278518 9393 ts_tablet_manager.cc:1173] T c6eb9324dee0414bbcb2039e9a23fab5 P 6cf745d60d5445749567bc9afd503edb: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:39:43.278777 9393 tablet_copy_client.cc:566] T c6eb9324dee0414bbcb2039e9a23fab5 P 6cf745d60d5445749567bc9afd503edb: tablet copy: Starting download of 0 data blocks... I0703 01:39:43.278972 9393 tablet_copy_client.cc:529] T c6eb9324dee0414bbcb2039e9a23fab5 P 6cf745d60d5445749567bc9afd503edb: tablet copy: Starting download of 1 WAL segments... I0703 01:39:43.280932 9393 tablet_copy_client.cc:405] T c6eb9324dee0414bbcb2039e9a23fab5 P 6cf745d60d5445749567bc9afd503edb: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:39:43.282601 9393 ts_tablet_manager.cc:1028] T c6eb9324dee0414bbcb2039e9a23fab5 P 6cf745d60d5445749567bc9afd503edb: Bootstrapping tablet I0703 01:39:43.282824 9393 tablet_bootstrap.cc:427] T c6eb9324dee0414bbcb2039e9a23fab5 P 6cf745d60d5445749567bc9afd503edb: Bootstrap starting. I0703 01:39:43.286747 9393 log.cc:525] T c6eb9324dee0414bbcb2039e9a23fab5 P 6cf745d60d5445749567bc9afd503edb: Log is configured to *not* fsync() on all Append() calls I0703 01:39:43.290393 9393 tablet_bootstrap.cc:427] T c6eb9324dee0414bbcb2039e9a23fab5 P 6cf745d60d5445749567bc9afd503edb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:39:43.290755 9393 tablet_bootstrap.cc:427] T c6eb9324dee0414bbcb2039e9a23fab5 P 6cf745d60d5445749567bc9afd503edb: Bootstrap complete. I0703 01:39:43.290974 9393 ts_tablet_manager.cc:1045] T c6eb9324dee0414bbcb2039e9a23fab5 P 6cf745d60d5445749567bc9afd503edb: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s I0703 01:39:43.296150 9393 raft_consensus.cc:325] T c6eb9324dee0414bbcb2039e9a23fab5 P 6cf745d60d5445749567bc9afd503edb [term 3 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e52364e0246c4bb4852c27e967e2f4c8" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46355 } } peers { permanent_uuid: "38c3a80a7fa04fcebe7dea0e351bf9b2" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 34814 } } peers { permanent_uuid: "6cf745d60d5445749567bc9afd503edb" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 54395 } attrs { promote: true } } unsafe_config_change: true I0703 01:39:43.296566 9393 raft_consensus.cc:628] T c6eb9324dee0414bbcb2039e9a23fab5 P 6cf745d60d5445749567bc9afd503edb [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 6cf745d60d5445749567bc9afd503edb, State: Initialized, Role: LEARNER I0703 01:39:43.296836 9393 consensus_queue.cc:227] T c6eb9324dee0414bbcb2039e9a23fab5 P 6cf745d60d5445749567bc9afd503edb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e52364e0246c4bb4852c27e967e2f4c8" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46355 } } peers { permanent_uuid: "38c3a80a7fa04fcebe7dea0e351bf9b2" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 34814 } } peers { permanent_uuid: "6cf745d60d5445749567bc9afd503edb" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 54395 } attrs { promote: true } } unsafe_config_change: true I0703 01:39:43.298655 9393 ts_tablet_manager.cc:1073] T c6eb9324dee0414bbcb2039e9a23fab5 P 6cf745d60d5445749567bc9afd503edb: Time spent starting tablet: real 0.008s user 0.006s sys 0.002s I0703 01:39:43.299371 9214 tablet_copy_service.cc:325] P 38c3a80a7fa04fcebe7dea0e351bf9b2: Request end of tablet copy session 6cf745d60d5445749567bc9afd503edb-c6eb9324dee0414bbcb2039e9a23fab5 received from {username='jenkins'} at 127.5.240.193:52566 I0703 01:39:43.299464 9214 tablet_copy_service.cc:413] P 38c3a80a7fa04fcebe7dea0e351bf9b2: ending tablet copy session 6cf745d60d5445749567bc9afd503edb-c6eb9324dee0414bbcb2039e9a23fab5 on tablet c6eb9324dee0414bbcb2039e9a23fab5 with peer 6cf745d60d5445749567bc9afd503edb I0703 01:39:43.311337 9286 catalog_manager.cc:2903] Scheduling retry of c6eb9324dee0414bbcb2039e9a23fab5 Delete Tablet RPC for TS=27562c9910fa4819a8ecf91dafbf7ce1 with a delay of 544 ms (attempt = 6) W0703 01:39:43.311401 9286 catalog_manager.cc:2922] Async tablet task c6eb9324dee0414bbcb2039e9a23fab5 Delete Tablet RPC for TS=27562c9910fa4819a8ecf91dafbf7ce1 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 27562c9910fa4819a8ecf91dafbf7ce1 W0703 01:39:43.517565 8871 tablet.cc:1678] T c6eb9324dee0414bbcb2039e9a23fab5 P 6cf745d60d5445749567bc9afd503edb: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:39:43.847990 8827 raft_consensus.cc:1026] T c6eb9324dee0414bbcb2039e9a23fab5 P 6cf745d60d5445749567bc9afd503edb [term 3 LEARNER]: Deduplicated request from leader. Original: 3.3->[3.4-3.4] Dedup: 3.4->[] I0703 01:39:43.856186 9286 catalog_manager.cc:2903] Scheduling retry of c6eb9324dee0414bbcb2039e9a23fab5 Delete Tablet RPC for TS=27562c9910fa4819a8ecf91dafbf7ce1 with a delay of 1070 ms (attempt = 7) W0703 01:39:43.856259 9286 catalog_manager.cc:2922] Async tablet task c6eb9324dee0414bbcb2039e9a23fab5 Delete Tablet RPC for TS=27562c9910fa4819a8ecf91dafbf7ce1 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 27562c9910fa4819a8ecf91dafbf7ce1 I0703 01:39:44.234160 9399 raft_consensus.cc:890] T c6eb9324dee0414bbcb2039e9a23fab5 P 38c3a80a7fa04fcebe7dea0e351bf9b2: attempting to promote NON_VOTER 6cf745d60d5445749567bc9afd503edb to VOTER I0703 01:39:44.234462 9399 consensus_peers.cc:455] T c6eb9324dee0414bbcb2039e9a23fab5 P 38c3a80a7fa04fcebe7dea0e351bf9b2 -> Peer 6cf745d60d5445749567bc9afd503edb (127.5.240.193:54395): Closing peer: 6cf745d60d5445749567bc9afd503edb I0703 01:39:44.234555 9399 consensus_peers.cc:455] T c6eb9324dee0414bbcb2039e9a23fab5 P 38c3a80a7fa04fcebe7dea0e351bf9b2 -> Peer e52364e0246c4bb4852c27e967e2f4c8 (127.5.240.195:46355): Closing peer: e52364e0246c4bb4852c27e967e2f4c8 I0703 01:39:44.234776 9399 consensus_queue.cc:204] T c6eb9324dee0414bbcb2039e9a23fab5 P 38c3a80a7fa04fcebe7dea0e351bf9b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 0, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e52364e0246c4bb4852c27e967e2f4c8" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46355 } } peers { permanent_uuid: "38c3a80a7fa04fcebe7dea0e351bf9b2" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 34814 } } peers { permanent_uuid: "6cf745d60d5445749567bc9afd503edb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54395 } attrs { promote: false } } unsafe_config_change: true I0703 01:39:44.236758 9072 raft_consensus.cc:1081] T c6eb9324dee0414bbcb2039e9a23fab5 P e52364e0246c4bb4852c27e967e2f4c8 [term 3 FOLLOWER]: Refusing update from remote peer 38c3a80a7fa04fcebe7dea0e351bf9b2: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch) I0703 01:39:44.236871 8827 raft_consensus.cc:1081] T c6eb9324dee0414bbcb2039e9a23fab5 P 6cf745d60d5445749567bc9afd503edb [term 3 LEARNER]: Refusing update from remote peer 38c3a80a7fa04fcebe7dea0e351bf9b2: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch) I0703 01:39:44.237411 9398 consensus_queue.cc:962] T c6eb9324dee0414bbcb2039e9a23fab5 P 38c3a80a7fa04fcebe7dea0e351bf9b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e52364e0246c4bb4852c27e967e2f4c8" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I0703 01:39:44.237753 9381 consensus_queue.cc:962] T c6eb9324dee0414bbcb2039e9a23fab5 P 38c3a80a7fa04fcebe7dea0e351bf9b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cf745d60d5445749567bc9afd503edb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I0703 01:39:44.239985 9381 raft_consensus.cc:2684] T c6eb9324dee0414bbcb2039e9a23fab5 P 38c3a80a7fa04fcebe7dea0e351bf9b2 [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 6cf745d60d5445749567bc9afd503edb (127.5.240.193) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e52364e0246c4bb4852c27e967e2f4c8" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46355 } } peers { permanent_uuid: "38c3a80a7fa04fcebe7dea0e351bf9b2" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 34814 } } peers { permanent_uuid: "6cf745d60d5445749567bc9afd503edb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54395 } attrs { promote: false } } unsafe_config_change: true } I0703 01:39:44.241225 8827 raft_consensus.cc:2684] T c6eb9324dee0414bbcb2039e9a23fab5 P 6cf745d60d5445749567bc9afd503edb [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 6cf745d60d5445749567bc9afd503edb (127.5.240.193) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e52364e0246c4bb4852c27e967e2f4c8" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46355 } } peers { permanent_uuid: "38c3a80a7fa04fcebe7dea0e351bf9b2" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 34814 } } peers { permanent_uuid: "6cf745d60d5445749567bc9afd503edb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54395 } attrs { promote: false } } unsafe_config_change: true } I0703 01:39:44.241592 9072 raft_consensus.cc:2684] T c6eb9324dee0414bbcb2039e9a23fab5 P e52364e0246c4bb4852c27e967e2f4c8 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, 6cf745d60d5445749567bc9afd503edb (127.5.240.193) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e52364e0246c4bb4852c27e967e2f4c8" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46355 } } peers { permanent_uuid: "38c3a80a7fa04fcebe7dea0e351bf9b2" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 34814 } } peers { permanent_uuid: "6cf745d60d5445749567bc9afd503edb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54395 } attrs { promote: false } } unsafe_config_change: true } I0703 01:39:44.244451 9305 catalog_manager.cc:3798] T c6eb9324dee0414bbcb2039e9a23fab5 P 38c3a80a7fa04fcebe7dea0e351bf9b2 reported cstate change: config changed from index 4 to 5, 6cf745d60d5445749567bc9afd503edb (127.5.240.193) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "38c3a80a7fa04fcebe7dea0e351bf9b2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e52364e0246c4bb4852c27e967e2f4c8" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38c3a80a7fa04fcebe7dea0e351bf9b2" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 34814 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6cf745d60d5445749567bc9afd503edb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54395 } attrs { promote: false } health_report { overall_health: HEALTHY } } unsafe_config_change: true } I0703 01:39:44.322428 6083 kudu-admin-test.cc:619] Waiting for Master to see new config... I0703 01:39:44.327854 6083 kudu-admin-test.cc:624] Tablet locations: tablet_id: "c6eb9324dee0414bbcb2039e9a23fab5" replicas { ts_info { permanent_uuid: "e52364e0246c4bb4852c27e967e2f4c8" rpc_addresses { host: "127.5.240.195" port: 46355 } } role: FOLLOWER } replicas { ts_info { permanent_uuid: "38c3a80a7fa04fcebe7dea0e351bf9b2" rpc_addresses { host: "127.5.240.196" port: 34814 } } role: LEADER } replicas { ts_info { permanent_uuid: "6cf745d60d5445749567bc9afd503edb" rpc_addresses { host: "127.5.240.193" port: 54395 } } role: FOLLOWER } DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } I0703 01:39:44.328675 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 8741 I0703 01:39:44.329603 8867 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:39:44.375936 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 8741 W0703 01:39:44.384325 9128 connection.cc:512] client connection to 127.5.240.193:54395 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:54395): Connection reset by peer (error 104) W0703 01:39:44.384379 9127 connection.cc:512] server connection from 127.5.240.193:52566 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:52566): Connection reset by peer (error 104) W0703 01:39:44.384515 9288 connection.cc:512] server connection from 127.5.240.193:57351 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:57351): Connection reset by peer (error 104) I0703 01:39:44.384832 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 8996 I0703 01:39:44.385634 9112 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:39:44.429391 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 8996 W0703 01:39:44.437870 9128 connection.cc:512] server connection from 127.5.240.195:42119 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.195:42119): Connection reset by peer (error 104) I0703 01:39:44.437921 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 9118 W0703 01:39:44.437929 9289 connection.cc:512] server connection from 127.5.240.195:55865 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.195:55865): Connection reset by peer (error 104) W0703 01:39:44.438030 9125 connection.cc:512] client connection to 127.5.240.195:46355 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.195:46355): Connection reset by peer (error 104) I0703 01:39:44.438827 9234 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:39:44.484963 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 9118 W0703 01:39:44.493111 9286 connection.cc:512] server connection from 127.5.240.196:42025 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.196:42025): Connection reset by peer (error 104) I0703 01:39:44.493278 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master pid 9278 I0703 01:39:44.494033 9335 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:39:44.555956 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master with pid 9278 [ OK ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes (6970 ms) [ RUN ] AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig I0703 01:39:44.578727 6083 test_util.cc:208] Using random seed: 1099368603 I0703 01:39:44.580255 6083 ts_itest-base.cc:113] Starting cluster with: I0703 01:39:44.580278 6083 ts_itest-base.cc:114] -------------- I0703 01:39:44.580296 6083 ts_itest-base.cc:115] 8 tablet servers I0703 01:39:44.580308 6083 ts_itest-base.cc:116] 5 replicas per TS I0703 01:39:44.580323 6083 ts_itest-base.cc:117] -------------- I0703 01:39:44.581408 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:54359 --webserver_interface=127.5.240.254 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:39:44.760921 9420 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:39:44.761132 9420 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:39:44.763324 9420 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0703 01:39:44.763360 9420 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:39:44.763391 9420 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0703 01:39:44.763415 9420 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0703 01:39:44.771165 9420 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:54359 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.5.240.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:39:44.776724 9420 master_main.cc:80] Initializing master server... I0703 01:39:44.777778 9420 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 145708us I0703 01:39:44.778846 9420 fs_manager.cc:261] Metadata directory not provided I0703 01:39:44.778898 9420 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:39:44.779120 9420 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:39:44.779177 9420 server_base.cc:433] Attempting to create new FS layout instead I0703 01:39:44.783324 9420 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/instance: uuid: "43575583f1234fdd9a8d9f6431ac9e65" format_stamp: "Formatted at 2018-07-03 08:39:44 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:44.783857 9420 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "43575583f1234fdd9a8d9f6431ac9e65" format_stamp: "Formatted at 2018-07-03 08:39:44 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:44.791885 9420 fs_manager.cc:501] Time spent creating directory manager: real 0.008s user 0.003s sys 0.004s I0703 01:39:44.793074 9420 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:39:44.793450 9420 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:39:44.796602 9420 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0703 01:39:44.796653 9420 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal uuid: "43575583f1234fdd9a8d9f6431ac9e65" format_stamp: "Formatted at 2018-07-03 08:39:44 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:44.796748 9420 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:39:44.844244 9420 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:39:44.845252 9420 master_main.cc:83] Starting Master server... I0703 01:39:44.882839 9420 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.254:54359 I0703 01:39:44.883298 9420 webserver.cc:175] Starting webserver on 127.5.240.254:0 I0703 01:39:44.883344 9420 webserver.cc:186] Document root disabled I0703 01:39:44.884027 9420 webserver.cc:313] Webserver started. Bound to: http://127.5.240.254:34888/ I0703 01:39:44.884636 9420 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb I0703 01:39:44.888273 9479 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:39:44.891934 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master as pid 9420 I0703 01:39:44.892854 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.5.240.193:0 --local_ip_for_outbound_sockets=127.5.240.193 --webserver_interface=127.5.240.193 --webserver_port=0 --tserver_master_addrs=127.5.240.254:54359 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0703 01:39:44.901013 9479 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 43575583f1234fdd9a8d9f6431ac9e65: Bootstrap starting. I0703 01:39:44.903017 9479 tablet_bootstrap.cc:578] T 00000000000000000000000000000000 P 43575583f1234fdd9a8d9f6431ac9e65: No blocks or log segments found. Creating new log. I0703 01:39:44.903782 9479 log.cc:525] T 00000000000000000000000000000000 P 43575583f1234fdd9a8d9f6431ac9e65: Log is configured to *not* fsync() on all Append() calls I0703 01:39:44.906206 9479 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 43575583f1234fdd9a8d9f6431ac9e65: No bootstrap required, opened a new log I0703 01:39:44.912665 9479 raft_consensus.cc:325] T 00000000000000000000000000000000 P 43575583f1234fdd9a8d9f6431ac9e65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43575583f1234fdd9a8d9f6431ac9e65" member_type: VOTER } I0703 01:39:44.912864 9479 raft_consensus.cc:351] T 00000000000000000000000000000000 P 43575583f1234fdd9a8d9f6431ac9e65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:39:44.912923 9479 raft_consensus.cc:628] T 00000000000000000000000000000000 P 43575583f1234fdd9a8d9f6431ac9e65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43575583f1234fdd9a8d9f6431ac9e65, State: Initialized, Role: FOLLOWER I0703 01:39:44.913416 9479 consensus_queue.cc:227] T 00000000000000000000000000000000 P 43575583f1234fdd9a8d9f6431ac9e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43575583f1234fdd9a8d9f6431ac9e65" member_type: VOTER } I0703 01:39:44.913569 9479 raft_consensus.cc:365] T 00000000000000000000000000000000 P 43575583f1234fdd9a8d9f6431ac9e65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:39:44.913663 9479 raft_consensus.cc:457] T 00000000000000000000000000000000 P 43575583f1234fdd9a8d9f6431ac9e65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:39:44.913743 9479 raft_consensus.cc:2730] T 00000000000000000000000000000000 P 43575583f1234fdd9a8d9f6431ac9e65: Snoozing failure detection for 1.629s (starting election) I0703 01:39:44.913826 9479 raft_consensus.cc:2782] T 00000000000000000000000000000000 P 43575583f1234fdd9a8d9f6431ac9e65 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:39:44.915223 9479 raft_consensus.cc:479] T 00000000000000000000000000000000 P 43575583f1234fdd9a8d9f6431ac9e65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43575583f1234fdd9a8d9f6431ac9e65" member_type: VOTER } I0703 01:39:44.915592 9479 leader_election.cc:270] T 00000000000000000000000000000000 P 43575583f1234fdd9a8d9f6431ac9e65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:39:44.917127 9482 raft_consensus.cc:2730] T 00000000000000000000000000000000 P 43575583f1234fdd9a8d9f6431ac9e65: Snoozing failure detection for 2.185s (election complete) I0703 01:39:44.917660 9482 raft_consensus.cc:2537] T 00000000000000000000000000000000 P 43575583f1234fdd9a8d9f6431ac9e65 [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:39:44.919107 9482 raft_consensus.cc:593] T 00000000000000000000000000000000 P 43575583f1234fdd9a8d9f6431ac9e65 [term 1 LEADER]: Becoming Leader. State: Replica: 43575583f1234fdd9a8d9f6431ac9e65, State: Running, Role: LEADER I0703 01:39:44.919456 9482 consensus_queue.cc:204] T 00000000000000000000000000000000 P 43575583f1234fdd9a8d9f6431ac9e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43575583f1234fdd9a8d9f6431ac9e65" member_type: VOTER } I0703 01:39:44.922240 9479 sys_catalog.cc:337] T 00000000000000000000000000000000 P 43575583f1234fdd9a8d9f6431ac9e65 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "43575583f1234fdd9a8d9f6431ac9e65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43575583f1234fdd9a8d9f6431ac9e65" member_type: VOTER } } I0703 01:39:44.922237 9484 sys_catalog.cc:337] T 00000000000000000000000000000000 P 43575583f1234fdd9a8d9f6431ac9e65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43575583f1234fdd9a8d9f6431ac9e65. Latest consensus state: current_term: 1 leader_uuid: "43575583f1234fdd9a8d9f6431ac9e65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43575583f1234fdd9a8d9f6431ac9e65" member_type: VOTER } } I0703 01:39:44.923168 9479 sys_catalog.cc:340] T 00000000000000000000000000000000 P 43575583f1234fdd9a8d9f6431ac9e65 [sys.catalog]: This master's current role is: LEADER I0703 01:39:44.923368 9484 sys_catalog.cc:340] T 00000000000000000000000000000000 P 43575583f1234fdd9a8d9f6431ac9e65 [sys.catalog]: This master's current role is: LEADER I0703 01:39:44.922843 9483 sys_catalog.cc:337] T 00000000000000000000000000000000 P 43575583f1234fdd9a8d9f6431ac9e65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "43575583f1234fdd9a8d9f6431ac9e65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43575583f1234fdd9a8d9f6431ac9e65" member_type: VOTER } } I0703 01:39:44.923642 9483 sys_catalog.cc:340] T 00000000000000000000000000000000 P 43575583f1234fdd9a8d9f6431ac9e65 [sys.catalog]: This master's current role is: LEADER I0703 01:39:44.925925 9479 sys_catalog.cc:424] T 00000000000000000000000000000000 P 43575583f1234fdd9a8d9f6431ac9e65 [sys.catalog]: configured and running, proceeding with master startup. I0703 01:39:44.926257 9486 catalog_manager.cc:1006] Loading table and tablet metadata into memory... I0703 01:39:44.926987 9420 master_main.cc:86] Master server successfully started. I0703 01:39:44.932811 9486 catalog_manager.cc:1017] Initializing Kudu internal certificate authority... I0703 01:39:44.984764 9486 catalog_manager.cc:904] Generated new certificate authority record I0703 01:39:44.986714 9486 catalog_manager.cc:1026] Loading token signing keys... I0703 01:39:44.999030 9486 catalog_manager.cc:4102] T 00000000000000000000000000000000 P 43575583f1234fdd9a8d9f6431ac9e65: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:39:45.067970 9480 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:39:45.068212 9480 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:39:45.070235 9480 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:39:45.070358 9480 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.193 I0703 01:39:45.077872 9480 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.5.240.193 --webserver_port=0 --tserver_master_addrs=127.5.240.254:54359 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.193 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:39:45.084595 9480 tablet_server_main.cc:85] Initializing tablet server... I0703 01:39:45.085765 9480 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 146208us I0703 01:39:45.086720 9480 fs_manager.cc:261] Metadata directory not provided I0703 01:39:45.086784 9480 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:39:45.087024 9480 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:39:45.087090 9480 server_base.cc:433] Attempting to create new FS layout instead I0703 01:39:45.091253 9480 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "a382b1ef929b411b825a324ad47ecaa9" format_stamp: "Formatted at 2018-07-03 08:39:45 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:45.091761 9480 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "a382b1ef929b411b825a324ad47ecaa9" format_stamp: "Formatted at 2018-07-03 08:39:45 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:45.099601 9480 fs_manager.cc:501] Time spent creating directory manager: real 0.008s user 0.001s sys 0.005s I0703 01:39:45.100775 9480 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:39:45.101169 9480 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:39:45.105317 9480 fs_manager.cc:417] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s I0703 01:39:45.105376 9480 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal uuid: "a382b1ef929b411b825a324ad47ecaa9" format_stamp: "Formatted at 2018-07-03 08:39:45 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:45.105499 9480 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:39:45.130671 9480 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:39:45.133191 9480 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:39:45.133671 9480 tablet_server_main.cc:90] Starting tablet server... I0703 01:39:45.228437 9480 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.193:48335 I0703 01:39:45.228942 9480 webserver.cc:175] Starting webserver on 127.5.240.193:0 I0703 01:39:45.228982 9480 webserver.cc:186] Document root disabled I0703 01:39:45.230324 9480 webserver.cc:313] Webserver started. Bound to: http://127.5.240.193:60893/ I0703 01:39:45.230875 9480 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb I0703 01:39:45.232928 9480 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:39:45.236037 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 9480 I0703 01:39:45.236876 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.5.240.194:0 --local_ip_for_outbound_sockets=127.5.240.194 --webserver_interface=127.5.240.194 --webserver_port=0 --tserver_master_addrs=127.5.240.254:54359 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0703 01:39:45.255165 9609 heartbeater.cc:346] Connected to a master server at 127.5.240.254:54359 I0703 01:39:45.255342 9609 heartbeater.cc:426] Registering TS with master... I0703 01:39:45.256273 9609 heartbeater.cc:475] Master 127.5.240.254:54359 requested a full tablet report, sending... I0703 01:39:45.258622 9446 ts_manager.cc:98] Registered new tserver with Master: a382b1ef929b411b825a324ad47ecaa9 (127.5.240.193:48335) I0703 01:39:45.261703 9446 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.193:49586 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:39:45.400023 9613 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:39:45.400260 9613 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:39:45.402274 9613 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:39:45.402381 9613 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.194 I0703 01:39:45.408727 9613 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.194:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.5.240.194 --webserver_port=0 --tserver_master_addrs=127.5.240.254:54359 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.194 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:39:45.415275 9613 tablet_server_main.cc:85] Initializing tablet server... I0703 01:39:45.416415 9613 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 146208us I0703 01:39:45.417377 9613 fs_manager.cc:261] Metadata directory not provided I0703 01:39:45.417440 9613 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:39:45.418077 9613 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:39:45.418140 9613 server_base.cc:433] Attempting to create new FS layout instead I0703 01:39:45.422233 9613 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "08ad9c3e352f492f9acb05219b72d0e0" format_stamp: "Formatted at 2018-07-03 08:39:45 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:45.422778 9613 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "08ad9c3e352f492f9acb05219b72d0e0" format_stamp: "Formatted at 2018-07-03 08:39:45 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:45.430532 9613 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.003s sys 0.004s I0703 01:39:45.431717 9613 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:39:45.432097 9613 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:39:45.435204 9613 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s I0703 01:39:45.435262 9613 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal uuid: "08ad9c3e352f492f9acb05219b72d0e0" format_stamp: "Formatted at 2018-07-03 08:39:45 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:45.435379 9613 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:39:45.509277 9613 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:39:45.511751 9613 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:39:45.512104 9613 tablet_server_main.cc:90] Starting tablet server... I0703 01:39:45.596083 9613 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.194:42244 I0703 01:39:45.596530 9613 webserver.cc:175] Starting webserver on 127.5.240.194:0 I0703 01:39:45.596566 9613 webserver.cc:186] Document root disabled I0703 01:39:45.597241 9613 webserver.cc:313] Webserver started. Bound to: http://127.5.240.194:48492/ I0703 01:39:45.597795 9613 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb I0703 01:39:45.600124 9613 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:39:45.600973 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 9613 I0703 01:39:45.601546 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.5.240.195:0 --local_ip_for_outbound_sockets=127.5.240.195 --webserver_interface=127.5.240.195 --webserver_port=0 --tserver_master_addrs=127.5.240.254:54359 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0703 01:39:45.612288 9732 heartbeater.cc:346] Connected to a master server at 127.5.240.254:54359 I0703 01:39:45.612401 9732 heartbeater.cc:426] Registering TS with master... I0703 01:39:45.613193 9732 heartbeater.cc:475] Master 127.5.240.254:54359 requested a full tablet report, sending... I0703 01:39:45.614203 9446 ts_manager.cc:98] Registered new tserver with Master: 08ad9c3e352f492f9acb05219b72d0e0 (127.5.240.194:42244) I0703 01:39:45.615067 9446 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.194:42036 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:39:45.759048 9735 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:39:45.759280 9735 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:39:45.761318 9735 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:39:45.761428 9735 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.195 I0703 01:39:45.767920 9735 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.195:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.5.240.195 --webserver_port=0 --tserver_master_addrs=127.5.240.254:54359 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.195 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:39:45.774431 9735 tablet_server_main.cc:85] Initializing tablet server... I0703 01:39:45.775568 9735 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 146208us I0703 01:39:45.776515 9735 fs_manager.cc:261] Metadata directory not provided I0703 01:39:45.776600 9735 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:39:45.776839 9735 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:39:45.776901 9735 server_base.cc:433] Attempting to create new FS layout instead I0703 01:39:45.781028 9735 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "937ef3ea804c4775a2a090e89e55f0fd" format_stamp: "Formatted at 2018-07-03 08:39:45 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:45.781599 9735 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "937ef3ea804c4775a2a090e89e55f0fd" format_stamp: "Formatted at 2018-07-03 08:39:45 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:45.788769 9735 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.004s sys 0.002s I0703 01:39:45.789927 9735 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:39:45.790325 9735 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:39:45.793937 9735 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0703 01:39:45.793993 9735 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal uuid: "937ef3ea804c4775a2a090e89e55f0fd" format_stamp: "Formatted at 2018-07-03 08:39:45 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:45.794111 9735 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:39:45.864086 9735 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:39:45.866374 9735 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:39:45.866750 9735 tablet_server_main.cc:90] Starting tablet server... I0703 01:39:45.949647 9735 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.195:47698 I0703 01:39:45.950088 9735 webserver.cc:175] Starting webserver on 127.5.240.195:0 I0703 01:39:45.950125 9735 webserver.cc:186] Document root disabled I0703 01:39:45.950788 9735 webserver.cc:313] Webserver started. Bound to: http://127.5.240.195:42132/ I0703 01:39:45.951304 9735 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb I0703 01:39:45.952967 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 9735 I0703 01:39:45.953568 9735 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:39:45.953680 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.5.240.196:0 --local_ip_for_outbound_sockets=127.5.240.196 --webserver_interface=127.5.240.196 --webserver_port=0 --tserver_master_addrs=127.5.240.254:54359 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0703 01:39:45.966078 9854 heartbeater.cc:346] Connected to a master server at 127.5.240.254:54359 I0703 01:39:45.966248 9854 heartbeater.cc:426] Registering TS with master... I0703 01:39:45.967165 9854 heartbeater.cc:475] Master 127.5.240.254:54359 requested a full tablet report, sending... I0703 01:39:45.968240 9446 ts_manager.cc:98] Registered new tserver with Master: 937ef3ea804c4775a2a090e89e55f0fd (127.5.240.195:47698) I0703 01:39:45.969553 9446 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.195:34496 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:39:46.112210 9857 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:39:46.112421 9857 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:39:46.114545 9857 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:39:46.114670 9857 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.196 I0703 01:39:46.120921 9857 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.196:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb --webserver_interface=127.5.240.196 --webserver_port=0 --tserver_master_addrs=127.5.240.254:54359 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.196 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:39:46.127359 9857 tablet_server_main.cc:85] Initializing tablet server... I0703 01:39:46.128479 9857 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 146708us I0703 01:39:46.129420 9857 fs_manager.cc:261] Metadata directory not provided I0703 01:39:46.129477 9857 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:39:46.130120 9857 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:39:46.130180 9857 server_base.cc:433] Attempting to create new FS layout instead I0703 01:39:46.134220 9857 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/instance: uuid: "37b8cbcdace54a91a27cd7bbb1eefe17" format_stamp: "Formatted at 2018-07-03 08:39:46 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:46.134742 9857 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal/instance: uuid: "37b8cbcdace54a91a27cd7bbb1eefe17" format_stamp: "Formatted at 2018-07-03 08:39:46 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:46.142915 9857 fs_manager.cc:501] Time spent creating directory manager: real 0.008s user 0.001s sys 0.006s I0703 01:39:46.144106 9857 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:39:46.144490 9857 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:39:46.147548 9857 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0703 01:39:46.147605 9857 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal uuid: "37b8cbcdace54a91a27cd7bbb1eefe17" format_stamp: "Formatted at 2018-07-03 08:39:46 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:46.147734 9857 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:39:46.192893 9857 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:39:46.195240 9857 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:39:46.195577 9857 tablet_server_main.cc:90] Starting tablet server... I0703 01:39:46.264778 9609 heartbeater.cc:467] Master 127.5.240.254:54359 was elected leader, sending a full tablet report... I0703 01:39:46.280225 9857 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.196:47502 I0703 01:39:46.280661 9857 webserver.cc:175] Starting webserver on 127.5.240.196:0 I0703 01:39:46.280692 9857 webserver.cc:186] Document root disabled I0703 01:39:46.281358 9857 webserver.cc:313] Webserver started. Bound to: http://127.5.240.196:49551/ I0703 01:39:46.281894 9857 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb I0703 01:39:46.284238 9857 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:39:46.286315 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 9857 I0703 01:39:46.286892 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data --block_manager=log --rpc_bind_addresses=127.5.240.197:0 --local_ip_for_outbound_sockets=127.5.240.197 --webserver_interface=127.5.240.197 --webserver_port=0 --tserver_master_addrs=127.5.240.254:54359 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0703 01:39:46.295250 9976 heartbeater.cc:346] Connected to a master server at 127.5.240.254:54359 I0703 01:39:46.295414 9976 heartbeater.cc:426] Registering TS with master... I0703 01:39:46.296334 9976 heartbeater.cc:475] Master 127.5.240.254:54359 requested a full tablet report, sending... I0703 01:39:46.297451 9446 ts_manager.cc:98] Registered new tserver with Master: 37b8cbcdace54a91a27cd7bbb1eefe17 (127.5.240.196:47502) I0703 01:39:46.298749 9446 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.196:37344 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:39:46.446054 9979 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:39:46.446288 9979 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:39:46.448297 9979 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:39:46.448410 9979 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.197 I0703 01:39:46.455446 9979 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.197:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb --webserver_interface=127.5.240.197 --webserver_port=0 --tserver_master_addrs=127.5.240.254:54359 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.197 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:39:46.461977 9979 tablet_server_main.cc:85] Initializing tablet server... I0703 01:39:46.463099 9979 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 146708us I0703 01:39:46.464063 9979 fs_manager.cc:261] Metadata directory not provided I0703 01:39:46.464126 9979 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:39:46.464355 9979 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:39:46.464419 9979 server_base.cc:433] Attempting to create new FS layout instead I0703 01:39:46.468598 9979 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/instance: uuid: "3254ace7bc954d2ab9972a2723f45b0c" format_stamp: "Formatted at 2018-07-03 08:39:46 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:46.469169 9979 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal/instance: uuid: "3254ace7bc954d2ab9972a2723f45b0c" format_stamp: "Formatted at 2018-07-03 08:39:46 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:46.476451 9979 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.004s sys 0.002s I0703 01:39:46.477628 9979 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:39:46.478013 9979 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:39:46.481652 9979 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s I0703 01:39:46.481711 9979 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal uuid: "3254ace7bc954d2ab9972a2723f45b0c" format_stamp: "Formatted at 2018-07-03 08:39:46 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:46.481827 9979 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:39:46.514958 9979 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:39:46.517168 9979 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:39:46.517519 9979 tablet_server_main.cc:90] Starting tablet server... I0703 01:39:46.599871 9979 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.197:54985 I0703 01:39:46.600297 9979 webserver.cc:175] Starting webserver on 127.5.240.197:0 I0703 01:39:46.600334 9979 webserver.cc:186] Document root disabled I0703 01:39:46.601578 9979 webserver.cc:313] Webserver started. Bound to: http://127.5.240.197:53153/ I0703 01:39:46.602123 9979 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb I0703 01:39:46.603915 9979 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:39:46.608572 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 9979 I0703 01:39:46.609339 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data --block_manager=log --rpc_bind_addresses=127.5.240.198:0 --local_ip_for_outbound_sockets=127.5.240.198 --webserver_interface=127.5.240.198 --webserver_port=0 --tserver_master_addrs=127.5.240.254:54359 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0703 01:39:46.617110 9732 heartbeater.cc:467] Master 127.5.240.254:54359 was elected leader, sending a full tablet report... I0703 01:39:46.618731 10098 heartbeater.cc:346] Connected to a master server at 127.5.240.254:54359 I0703 01:39:46.618870 10098 heartbeater.cc:426] Registering TS with master... I0703 01:39:46.619673 10098 heartbeater.cc:475] Master 127.5.240.254:54359 requested a full tablet report, sending... I0703 01:39:46.620741 9446 ts_manager.cc:98] Registered new tserver with Master: 3254ace7bc954d2ab9972a2723f45b0c (127.5.240.197:54985) I0703 01:39:46.621508 9446 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.197:46189 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:39:46.769176 10101 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:39:46.769385 10101 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:39:46.771453 10101 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:39:46.771543 10101 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.198 I0703 01:39:46.778019 10101 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.198:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data/info.pb --webserver_interface=127.5.240.198 --webserver_port=0 --tserver_master_addrs=127.5.240.254:54359 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.198 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:39:46.784585 10101 tablet_server_main.cc:85] Initializing tablet server... I0703 01:39:46.785698 10101 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 146708us I0703 01:39:46.786623 10101 fs_manager.cc:261] Metadata directory not provided I0703 01:39:46.786675 10101 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:39:46.787261 10101 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:39:46.787317 10101 server_base.cc:433] Attempting to create new FS layout instead I0703 01:39:46.791501 10101 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data/instance: uuid: "7bb94c92677448c39c50efc79e39459a" format_stamp: "Formatted at 2018-07-03 08:39:46 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:46.792026 10101 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/wal/instance: uuid: "7bb94c92677448c39c50efc79e39459a" format_stamp: "Formatted at 2018-07-03 08:39:46 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:46.799681 10101 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.003s sys 0.004s I0703 01:39:46.800868 10101 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:39:46.801252 10101 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:39:46.804344 10101 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0703 01:39:46.804396 10101 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/wal uuid: "7bb94c92677448c39c50efc79e39459a" format_stamp: "Formatted at 2018-07-03 08:39:46 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:46.804520 10101 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:39:46.867841 10101 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:39:46.870055 10101 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:39:46.870385 10101 tablet_server_main.cc:90] Starting tablet server... I0703 01:39:46.954001 10101 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.198:39947 I0703 01:39:46.954452 10101 webserver.cc:175] Starting webserver on 127.5.240.198:0 I0703 01:39:46.954490 10101 webserver.cc:186] Document root disabled I0703 01:39:46.955706 10101 webserver.cc:313] Webserver started. Bound to: http://127.5.240.198:43766/ I0703 01:39:46.956244 10101 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data/info.pb I0703 01:39:46.958612 10101 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:39:46.963115 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 10101 I0703 01:39:46.963709 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data --block_manager=log --rpc_bind_addresses=127.5.240.199:0 --local_ip_for_outbound_sockets=127.5.240.199 --webserver_interface=127.5.240.199 --webserver_port=0 --tserver_master_addrs=127.5.240.254:54359 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0703 01:39:46.970101 10220 heartbeater.cc:346] Connected to a master server at 127.5.240.254:54359 I0703 01:39:46.970269 10220 heartbeater.cc:426] Registering TS with master... I0703 01:39:46.971215 10220 heartbeater.cc:475] Master 127.5.240.254:54359 requested a full tablet report, sending... I0703 01:39:46.972048 9854 heartbeater.cc:467] Master 127.5.240.254:54359 was elected leader, sending a full tablet report... I0703 01:39:46.972853 9446 ts_manager.cc:98] Registered new tserver with Master: 7bb94c92677448c39c50efc79e39459a (127.5.240.198:39947) I0703 01:39:46.973878 9446 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.198:52490 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:39:47.122535 10223 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:39:47.122771 10223 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:39:47.124749 10223 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:39:47.124837 10223 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.199 I0703 01:39:47.131844 10223 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.199:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data/info.pb --webserver_interface=127.5.240.199 --webserver_port=0 --tserver_master_addrs=127.5.240.254:54359 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.199 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:39:47.138270 10223 tablet_server_main.cc:85] Initializing tablet server... I0703 01:39:47.139362 10223 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 147208us I0703 01:39:47.140297 10223 fs_manager.cc:261] Metadata directory not provided I0703 01:39:47.140350 10223 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:39:47.140578 10223 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:39:47.140636 10223 server_base.cc:433] Attempting to create new FS layout instead I0703 01:39:47.144735 10223 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data/instance: uuid: "add911cdebcf44b8a64256c228929d30" format_stamp: "Formatted at 2018-07-03 08:39:47 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:47.145241 10223 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/wal/instance: uuid: "add911cdebcf44b8a64256c228929d30" format_stamp: "Formatted at 2018-07-03 08:39:47 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:47.152385 10223 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.003s sys 0.004s I0703 01:39:47.153576 10223 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:39:47.153944 10223 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:39:47.157464 10223 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s I0703 01:39:47.157515 10223 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/wal uuid: "add911cdebcf44b8a64256c228929d30" format_stamp: "Formatted at 2018-07-03 08:39:47 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:47.157641 10223 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:39:47.209914 10223 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:39:47.211733 10223 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:39:47.212036 10223 tablet_server_main.cc:90] Starting tablet server... I0703 01:39:47.299912 10223 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.199:59090 I0703 01:39:47.300366 10223 webserver.cc:175] Starting webserver on 127.5.240.199:0 I0703 01:39:47.300402 10223 webserver.cc:186] Document root disabled I0703 01:39:47.301069 9976 heartbeater.cc:467] Master 127.5.240.254:54359 was elected leader, sending a full tablet report... I0703 01:39:47.301734 10223 webserver.cc:313] Webserver started. Bound to: http://127.5.240.199:58757/ I0703 01:39:47.302412 10223 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data/info.pb I0703 01:39:47.304916 10223 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:39:47.306175 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 10223 I0703 01:39:47.306944 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-7/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-7/data --block_manager=log --rpc_bind_addresses=127.5.240.200:0 --local_ip_for_outbound_sockets=127.5.240.200 --webserver_interface=127.5.240.200 --webserver_port=0 --tserver_master_addrs=127.5.240.254:54359 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-7/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-7/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0703 01:39:47.316824 10342 heartbeater.cc:346] Connected to a master server at 127.5.240.254:54359 I0703 01:39:47.317013 10342 heartbeater.cc:426] Registering TS with master... I0703 01:39:47.317957 10342 heartbeater.cc:475] Master 127.5.240.254:54359 requested a full tablet report, sending... I0703 01:39:47.319017 9446 ts_manager.cc:98] Registered new tserver with Master: add911cdebcf44b8a64256c228929d30 (127.5.240.199:59090) I0703 01:39:47.319864 9446 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.199:38364 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:39:47.465061 10345 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:39:47.465291 10345 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:39:47.467355 10345 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:39:47.467445 10345 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.200 I0703 01:39:47.474403 10345 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-7/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-7/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.200:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-7/data/info.pb --webserver_interface=127.5.240.200 --webserver_port=0 --tserver_master_addrs=127.5.240.254:54359 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.200 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-7/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:39:47.480988 10345 tablet_server_main.cc:85] Initializing tablet server... I0703 01:39:47.482095 10345 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 147208us I0703 01:39:47.483057 10345 fs_manager.cc:261] Metadata directory not provided I0703 01:39:47.483109 10345 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:39:47.483325 10345 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:39:47.483382 10345 server_base.cc:433] Attempting to create new FS layout instead I0703 01:39:47.487511 10345 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-7/data/instance: uuid: "d33993c1f2c148d9ae2504a60ed04173" format_stamp: "Formatted at 2018-07-03 08:39:47 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:47.488052 10345 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-7/wal/instance: uuid: "d33993c1f2c148d9ae2504a60ed04173" format_stamp: "Formatted at 2018-07-03 08:39:47 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:47.495365 10345 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.004s sys 0.002s I0703 01:39:47.496522 10345 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:39:47.496948 10345 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:39:47.500457 10345 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s I0703 01:39:47.500506 10345 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-7/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-7/wal uuid: "d33993c1f2c148d9ae2504a60ed04173" format_stamp: "Formatted at 2018-07-03 08:39:47 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:47.500635 10345 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-7/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-7/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-7/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:39:47.526989 10345 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:39:47.529193 10345 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:39:47.529561 10345 tablet_server_main.cc:90] Starting tablet server... I0703 01:39:47.614439 10345 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.200:54337 I0703 01:39:47.614897 10345 webserver.cc:175] Starting webserver on 127.5.240.200:0 I0703 01:39:47.614930 10345 webserver.cc:186] Document root disabled I0703 01:39:47.615607 10345 webserver.cc:313] Webserver started. Bound to: http://127.5.240.200:34634/ I0703 01:39:47.616120 10345 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-7/data/info.pb I0703 01:39:47.617902 10345 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:39:47.619244 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 10345 I0703 01:39:47.623756 10098 heartbeater.cc:467] Master 127.5.240.254:54359 was elected leader, sending a full tablet report... I0703 01:39:47.630792 10464 heartbeater.cc:346] Connected to a master server at 127.5.240.254:54359 I0703 01:39:47.630929 10464 heartbeater.cc:426] Registering TS with master... I0703 01:39:47.631759 10464 heartbeater.cc:475] Master 127.5.240.254:54359 requested a full tablet report, sending... I0703 01:39:47.632951 9446 ts_manager.cc:98] Registered new tserver with Master: d33993c1f2c148d9ae2504a60ed04173 (127.5.240.200:54337) I0703 01:39:47.633797 9446 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.200:36435 I0703 01:39:47.635159 6083 external_mini_cluster.cc:490] 8 TS(s) registered with all masters I0703 01:39:47.645665 9446 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.5.240.254:56275: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 5 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0703 01:39:47.681500 9546 tablet_service.cc:759] Processing CreateTablet for tablet a87a2a21d927456d930023d9775aec80 (table=TestTable [id=d45069b9f7d0418dbe6b2bc688216e4f]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:39:47.682433 9546 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet a87a2a21d927456d930023d9775aec80. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:39:47.682525 10157 tablet_service.cc:759] Processing CreateTablet for tablet a87a2a21d927456d930023d9775aec80 (table=TestTable [id=d45069b9f7d0418dbe6b2bc688216e4f]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:39:47.683266 10157 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet a87a2a21d927456d930023d9775aec80. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:39:47.697302 9546 ts_tablet_manager.cc:1173] T a87a2a21d927456d930023d9775aec80 P a382b1ef929b411b825a324ad47ecaa9: Registered tablet (data state: TABLET_DATA_READY) I0703 01:39:47.697685 9669 tablet_service.cc:759] Processing CreateTablet for tablet a87a2a21d927456d930023d9775aec80 (table=TestTable [id=d45069b9f7d0418dbe6b2bc688216e4f]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:39:47.698468 10157 ts_tablet_manager.cc:1173] T a87a2a21d927456d930023d9775aec80 P 7bb94c92677448c39c50efc79e39459a: Registered tablet (data state: TABLET_DATA_READY) I0703 01:39:47.698542 9669 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet a87a2a21d927456d930023d9775aec80. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:39:47.702713 10488 ts_tablet_manager.cc:1028] T a87a2a21d927456d930023d9775aec80 P a382b1ef929b411b825a324ad47ecaa9: Bootstrapping tablet I0703 01:39:47.703003 10488 tablet_bootstrap.cc:427] T a87a2a21d927456d930023d9775aec80 P a382b1ef929b411b825a324ad47ecaa9: Bootstrap starting. I0703 01:39:47.704000 9791 tablet_service.cc:759] Processing CreateTablet for tablet a87a2a21d927456d930023d9775aec80 (table=TestTable [id=d45069b9f7d0418dbe6b2bc688216e4f]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:39:47.704847 9791 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet a87a2a21d927456d930023d9775aec80. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:39:47.706642 10488 tablet_bootstrap.cc:578] T a87a2a21d927456d930023d9775aec80 P a382b1ef929b411b825a324ad47ecaa9: No blocks or log segments found. Creating new log. I0703 01:39:47.707547 10488 log.cc:525] T a87a2a21d927456d930023d9775aec80 P a382b1ef929b411b825a324ad47ecaa9: Log is configured to *not* fsync() on all Append() calls I0703 01:39:47.707909 10489 ts_tablet_manager.cc:1028] T a87a2a21d927456d930023d9775aec80 P 7bb94c92677448c39c50efc79e39459a: Bootstrapping tablet I0703 01:39:47.708238 9913 tablet_service.cc:759] Processing CreateTablet for tablet a87a2a21d927456d930023d9775aec80 (table=TestTable [id=d45069b9f7d0418dbe6b2bc688216e4f]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:39:47.708925 10489 tablet_bootstrap.cc:427] T a87a2a21d927456d930023d9775aec80 P 7bb94c92677448c39c50efc79e39459a: Bootstrap starting. I0703 01:39:47.709012 9913 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet a87a2a21d927456d930023d9775aec80. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:39:47.711913 10489 tablet_bootstrap.cc:578] T a87a2a21d927456d930023d9775aec80 P 7bb94c92677448c39c50efc79e39459a: No blocks or log segments found. Creating new log. I0703 01:39:47.712858 9669 ts_tablet_manager.cc:1173] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0: Registered tablet (data state: TABLET_DATA_READY) I0703 01:39:47.712992 10489 log.cc:525] T a87a2a21d927456d930023d9775aec80 P 7bb94c92677448c39c50efc79e39459a: Log is configured to *not* fsync() on all Append() calls I0703 01:39:47.715427 10488 tablet_bootstrap.cc:427] T a87a2a21d927456d930023d9775aec80 P a382b1ef929b411b825a324ad47ecaa9: No bootstrap required, opened a new log I0703 01:39:47.715651 10488 ts_tablet_manager.cc:1045] T a87a2a21d927456d930023d9775aec80 P a382b1ef929b411b825a324ad47ecaa9: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s I0703 01:39:47.719373 10492 ts_tablet_manager.cc:1028] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0: Bootstrapping tablet I0703 01:39:47.720173 10489 tablet_bootstrap.cc:427] T a87a2a21d927456d930023d9775aec80 P 7bb94c92677448c39c50efc79e39459a: No bootstrap required, opened a new log I0703 01:39:47.720432 10489 ts_tablet_manager.cc:1045] T a87a2a21d927456d930023d9775aec80 P 7bb94c92677448c39c50efc79e39459a: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s I0703 01:39:47.723991 10488 raft_consensus.cc:325] T a87a2a21d927456d930023d9775aec80 P a382b1ef929b411b825a324ad47ecaa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } } peers { permanent_uuid: "a382b1ef929b411b825a324ad47ecaa9" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 48335 } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } } peers { permanent_uuid: "37b8cbcdace54a91a27cd7bbb1eefe17" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 47502 } } peers { permanent_uuid: "937ef3ea804c4775a2a090e89e55f0fd" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 47698 } } I0703 01:39:47.724521 10488 raft_consensus.cc:351] T a87a2a21d927456d930023d9775aec80 P a382b1ef929b411b825a324ad47ecaa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:39:47.724643 10488 raft_consensus.cc:628] T a87a2a21d927456d930023d9775aec80 P a382b1ef929b411b825a324ad47ecaa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a382b1ef929b411b825a324ad47ecaa9, State: Initialized, Role: FOLLOWER I0703 01:39:47.725211 10488 consensus_queue.cc:227] T a87a2a21d927456d930023d9775aec80 P a382b1ef929b411b825a324ad47ecaa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } } peers { permanent_uuid: "a382b1ef929b411b825a324ad47ecaa9" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 48335 } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } } peers { permanent_uuid: "37b8cbcdace54a91a27cd7bbb1eefe17" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 47502 } } peers { permanent_uuid: "937ef3ea804c4775a2a090e89e55f0fd" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 47698 } } I0703 01:39:47.725913 10492 tablet_bootstrap.cc:427] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0: Bootstrap starting. I0703 01:39:47.726125 9913 ts_tablet_manager.cc:1173] T a87a2a21d927456d930023d9775aec80 P 37b8cbcdace54a91a27cd7bbb1eefe17: Registered tablet (data state: TABLET_DATA_READY) I0703 01:39:47.727751 9791 ts_tablet_manager.cc:1173] T a87a2a21d927456d930023d9775aec80 P 937ef3ea804c4775a2a090e89e55f0fd: Registered tablet (data state: TABLET_DATA_READY) I0703 01:39:47.728404 10492 tablet_bootstrap.cc:578] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0: No blocks or log segments found. Creating new log. I0703 01:39:47.730293 10495 ts_tablet_manager.cc:1028] T a87a2a21d927456d930023d9775aec80 P 937ef3ea804c4775a2a090e89e55f0fd: Bootstrapping tablet I0703 01:39:47.731178 10494 ts_tablet_manager.cc:1028] T a87a2a21d927456d930023d9775aec80 P 37b8cbcdace54a91a27cd7bbb1eefe17: Bootstrapping tablet I0703 01:39:47.731940 10488 ts_tablet_manager.cc:1073] T a87a2a21d927456d930023d9775aec80 P a382b1ef929b411b825a324ad47ecaa9: Time spent starting tablet: real 0.016s user 0.006s sys 0.008s I0703 01:39:47.731628 10489 raft_consensus.cc:325] T a87a2a21d927456d930023d9775aec80 P 7bb94c92677448c39c50efc79e39459a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } } peers { permanent_uuid: "a382b1ef929b411b825a324ad47ecaa9" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 48335 } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } } peers { permanent_uuid: "37b8cbcdace54a91a27cd7bbb1eefe17" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 47502 } } peers { permanent_uuid: "937ef3ea804c4775a2a090e89e55f0fd" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 47698 } } I0703 01:39:47.732334 10494 tablet_bootstrap.cc:427] T a87a2a21d927456d930023d9775aec80 P 37b8cbcdace54a91a27cd7bbb1eefe17: Bootstrap starting. I0703 01:39:47.732414 10495 tablet_bootstrap.cc:427] T a87a2a21d927456d930023d9775aec80 P 937ef3ea804c4775a2a090e89e55f0fd: Bootstrap starting. I0703 01:39:47.733103 10492 log.cc:525] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0: Log is configured to *not* fsync() on all Append() calls I0703 01:39:47.733415 10489 raft_consensus.cc:351] T a87a2a21d927456d930023d9775aec80 P 7bb94c92677448c39c50efc79e39459a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:39:47.733508 10489 raft_consensus.cc:628] T a87a2a21d927456d930023d9775aec80 P 7bb94c92677448c39c50efc79e39459a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bb94c92677448c39c50efc79e39459a, State: Initialized, Role: FOLLOWER I0703 01:39:47.733978 10489 consensus_queue.cc:227] T a87a2a21d927456d930023d9775aec80 P 7bb94c92677448c39c50efc79e39459a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } } peers { permanent_uuid: "a382b1ef929b411b825a324ad47ecaa9" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 48335 } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } } peers { permanent_uuid: "37b8cbcdace54a91a27cd7bbb1eefe17" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 47502 } } peers { permanent_uuid: "937ef3ea804c4775a2a090e89e55f0fd" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 47698 } } W0703 01:39:47.734616 9610 tablet.cc:1678] T a87a2a21d927456d930023d9775aec80 P a382b1ef929b411b825a324ad47ecaa9: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:39:47.735471 10220 heartbeater.cc:467] Master 127.5.240.254:54359 was elected leader, sending a full tablet report... I0703 01:39:47.735996 10489 ts_tablet_manager.cc:1073] T a87a2a21d927456d930023d9775aec80 P 7bb94c92677448c39c50efc79e39459a: Time spent starting tablet: real 0.015s user 0.007s sys 0.003s I0703 01:39:47.736275 10494 tablet_bootstrap.cc:578] T a87a2a21d927456d930023d9775aec80 P 37b8cbcdace54a91a27cd7bbb1eefe17: No blocks or log segments found. Creating new log. I0703 01:39:47.737026 10495 tablet_bootstrap.cc:578] T a87a2a21d927456d930023d9775aec80 P 937ef3ea804c4775a2a090e89e55f0fd: No blocks or log segments found. Creating new log. I0703 01:39:47.737167 10494 log.cc:525] T a87a2a21d927456d930023d9775aec80 P 37b8cbcdace54a91a27cd7bbb1eefe17: Log is configured to *not* fsync() on all Append() calls I0703 01:39:47.738692 10495 log.cc:525] T a87a2a21d927456d930023d9775aec80 P 937ef3ea804c4775a2a090e89e55f0fd: Log is configured to *not* fsync() on all Append() calls I0703 01:39:47.739665 10494 tablet_bootstrap.cc:427] T a87a2a21d927456d930023d9775aec80 P 37b8cbcdace54a91a27cd7bbb1eefe17: No bootstrap required, opened a new log I0703 01:39:47.739897 10494 ts_tablet_manager.cc:1045] T a87a2a21d927456d930023d9775aec80 P 37b8cbcdace54a91a27cd7bbb1eefe17: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.001s I0703 01:39:47.740550 10492 tablet_bootstrap.cc:427] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0: No bootstrap required, opened a new log I0703 01:39:47.740783 10492 ts_tablet_manager.cc:1045] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.003s I0703 01:39:47.741045 10495 tablet_bootstrap.cc:427] T a87a2a21d927456d930023d9775aec80 P 937ef3ea804c4775a2a090e89e55f0fd: No bootstrap required, opened a new log I0703 01:39:47.741237 10495 ts_tablet_manager.cc:1045] T a87a2a21d927456d930023d9775aec80 P 937ef3ea804c4775a2a090e89e55f0fd: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.002s I0703 01:39:47.746628 10494 raft_consensus.cc:325] T a87a2a21d927456d930023d9775aec80 P 37b8cbcdace54a91a27cd7bbb1eefe17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } } peers { permanent_uuid: "a382b1ef929b411b825a324ad47ecaa9" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 48335 } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } } peers { permanent_uuid: "37b8cbcdace54a91a27cd7bbb1eefe17" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 47502 } } peers { permanent_uuid: "937ef3ea804c4775a2a090e89e55f0fd" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 47698 } } I0703 01:39:47.746978 10494 raft_consensus.cc:351] T a87a2a21d927456d930023d9775aec80 P 37b8cbcdace54a91a27cd7bbb1eefe17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:39:47.747042 10494 raft_consensus.cc:628] T a87a2a21d927456d930023d9775aec80 P 37b8cbcdace54a91a27cd7bbb1eefe17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37b8cbcdace54a91a27cd7bbb1eefe17, State: Initialized, Role: FOLLOWER I0703 01:39:47.747443 10494 consensus_queue.cc:227] T a87a2a21d927456d930023d9775aec80 P 37b8cbcdace54a91a27cd7bbb1eefe17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } } peers { permanent_uuid: "a382b1ef929b411b825a324ad47ecaa9" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 48335 } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } } peers { permanent_uuid: "37b8cbcdace54a91a27cd7bbb1eefe17" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 47502 } } peers { permanent_uuid: "937ef3ea804c4775a2a090e89e55f0fd" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 47698 } } I0703 01:39:47.749398 10494 ts_tablet_manager.cc:1073] T a87a2a21d927456d930023d9775aec80 P 37b8cbcdace54a91a27cd7bbb1eefe17: Time spent starting tablet: real 0.009s user 0.005s sys 0.004s I0703 01:39:47.749434 10492 raft_consensus.cc:325] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } } peers { permanent_uuid: "a382b1ef929b411b825a324ad47ecaa9" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 48335 } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } } peers { permanent_uuid: "37b8cbcdace54a91a27cd7bbb1eefe17" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 47502 } } peers { permanent_uuid: "937ef3ea804c4775a2a090e89e55f0fd" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 47698 } } I0703 01:39:47.749891 10492 raft_consensus.cc:351] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:39:47.749642 10495 raft_consensus.cc:325] T a87a2a21d927456d930023d9775aec80 P 937ef3ea804c4775a2a090e89e55f0fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } } peers { permanent_uuid: "a382b1ef929b411b825a324ad47ecaa9" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 48335 } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } } peers { permanent_uuid: "37b8cbcdace54a91a27cd7bbb1eefe17" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 47502 } } peers { permanent_uuid: "937ef3ea804c4775a2a090e89e55f0fd" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 47698 } } I0703 01:39:47.749966 10492 raft_consensus.cc:628] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08ad9c3e352f492f9acb05219b72d0e0, State: Initialized, Role: FOLLOWER I0703 01:39:47.750049 10495 raft_consensus.cc:351] T a87a2a21d927456d930023d9775aec80 P 937ef3ea804c4775a2a090e89e55f0fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:39:47.750118 10495 raft_consensus.cc:628] T a87a2a21d927456d930023d9775aec80 P 937ef3ea804c4775a2a090e89e55f0fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 937ef3ea804c4775a2a090e89e55f0fd, State: Initialized, Role: FOLLOWER I0703 01:39:47.750372 10492 consensus_queue.cc:227] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } } peers { permanent_uuid: "a382b1ef929b411b825a324ad47ecaa9" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 48335 } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } } peers { permanent_uuid: "37b8cbcdace54a91a27cd7bbb1eefe17" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 47502 } } peers { permanent_uuid: "937ef3ea804c4775a2a090e89e55f0fd" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 47698 } } I0703 01:39:47.750522 10495 consensus_queue.cc:227] T a87a2a21d927456d930023d9775aec80 P 937ef3ea804c4775a2a090e89e55f0fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } } peers { permanent_uuid: "a382b1ef929b411b825a324ad47ecaa9" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 48335 } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } } peers { permanent_uuid: "37b8cbcdace54a91a27cd7bbb1eefe17" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 47502 } } peers { permanent_uuid: "937ef3ea804c4775a2a090e89e55f0fd" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 47698 } } I0703 01:39:47.752336 10492 ts_tablet_manager.cc:1073] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0: Time spent starting tablet: real 0.011s user 0.006s sys 0.005s I0703 01:39:47.753388 10495 ts_tablet_manager.cc:1073] T a87a2a21d927456d930023d9775aec80 P 937ef3ea804c4775a2a090e89e55f0fd: Time spent starting tablet: real 0.012s user 0.009s sys 0.003s I0703 01:39:47.767526 10493 raft_consensus.cc:457] T a87a2a21d927456d930023d9775aec80 P a382b1ef929b411b825a324ad47ecaa9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0703 01:39:47.767634 10493 raft_consensus.cc:2730] T a87a2a21d927456d930023d9775aec80 P a382b1ef929b411b825a324ad47ecaa9: Snoozing failure detection for 1.968s (starting election) I0703 01:39:47.767727 10493 raft_consensus.cc:479] T a87a2a21d927456d930023d9775aec80 P a382b1ef929b411b825a324ad47ecaa9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } } peers { permanent_uuid: "a382b1ef929b411b825a324ad47ecaa9" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 48335 } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } } peers { permanent_uuid: "37b8cbcdace54a91a27cd7bbb1eefe17" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 47502 } } peers { permanent_uuid: "937ef3ea804c4775a2a090e89e55f0fd" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 47698 } } I0703 01:39:47.768970 10493 leader_election.cc:243] T a87a2a21d927456d930023d9775aec80 P a382b1ef929b411b825a324ad47ecaa9 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 7bb94c92677448c39c50efc79e39459a (127.5.240.198:39947) I0703 01:39:47.769263 10493 leader_election.cc:243] T a87a2a21d927456d930023d9775aec80 P a382b1ef929b411b825a324ad47ecaa9 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 08ad9c3e352f492f9acb05219b72d0e0 (127.5.240.194:42244) I0703 01:39:47.769413 10493 leader_election.cc:243] T a87a2a21d927456d930023d9775aec80 P a382b1ef929b411b825a324ad47ecaa9 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 37b8cbcdace54a91a27cd7bbb1eefe17 (127.5.240.196:47502) I0703 01:39:47.769703 10493 leader_election.cc:243] T a87a2a21d927456d930023d9775aec80 P a382b1ef929b411b825a324ad47ecaa9 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 937ef3ea804c4775a2a090e89e55f0fd (127.5.240.195:47698) W0703 01:39:47.786168 9977 tablet.cc:1678] T a87a2a21d927456d930023d9775aec80 P 37b8cbcdace54a91a27cd7bbb1eefe17: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:39:47.788178 9811 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "a87a2a21d927456d930023d9775aec80" candidate_uuid: "a382b1ef929b411b825a324ad47ecaa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "937ef3ea804c4775a2a090e89e55f0fd" is_pre_election: true I0703 01:39:47.788921 9811 raft_consensus.cc:2730] T a87a2a21d927456d930023d9775aec80 P 937ef3ea804c4775a2a090e89e55f0fd: Snoozing failure detection for 1.621s (vote granted) I0703 01:39:47.789028 9811 raft_consensus.cc:2243] T a87a2a21d927456d930023d9775aec80 P 937ef3ea804c4775a2a090e89e55f0fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a382b1ef929b411b825a324ad47ecaa9 in term 0. I0703 01:39:47.789778 9498 leader_election.cc:379] T a87a2a21d927456d930023d9775aec80 P a382b1ef929b411b825a324ad47ecaa9 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 937ef3ea804c4775a2a090e89e55f0fd (127.5.240.195:47698) I0703 01:39:47.789330 9933 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "a87a2a21d927456d930023d9775aec80" candidate_uuid: "a382b1ef929b411b825a324ad47ecaa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37b8cbcdace54a91a27cd7bbb1eefe17" is_pre_election: true I0703 01:39:47.789791 10177 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "a87a2a21d927456d930023d9775aec80" candidate_uuid: "a382b1ef929b411b825a324ad47ecaa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bb94c92677448c39c50efc79e39459a" is_pre_election: true I0703 01:39:47.790014 9933 raft_consensus.cc:2730] T a87a2a21d927456d930023d9775aec80 P 37b8cbcdace54a91a27cd7bbb1eefe17: Snoozing failure detection for 1.738s (vote granted) I0703 01:39:47.790077 10177 raft_consensus.cc:2730] T a87a2a21d927456d930023d9775aec80 P 7bb94c92677448c39c50efc79e39459a: Snoozing failure detection for 1.577s (vote granted) I0703 01:39:47.790086 9933 raft_consensus.cc:2243] T a87a2a21d927456d930023d9775aec80 P 37b8cbcdace54a91a27cd7bbb1eefe17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a382b1ef929b411b825a324ad47ecaa9 in term 0. I0703 01:39:47.790171 10177 raft_consensus.cc:2243] T a87a2a21d927456d930023d9775aec80 P 7bb94c92677448c39c50efc79e39459a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a382b1ef929b411b825a324ad47ecaa9 in term 0. I0703 01:39:47.790727 9497 leader_election.cc:379] T a87a2a21d927456d930023d9775aec80 P a382b1ef929b411b825a324ad47ecaa9 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 37b8cbcdace54a91a27cd7bbb1eefe17 (127.5.240.196:47502) I0703 01:39:47.790835 9499 leader_election.cc:379] T a87a2a21d927456d930023d9775aec80 P a382b1ef929b411b825a324ad47ecaa9 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 7bb94c92677448c39c50efc79e39459a (127.5.240.198:39947) I0703 01:39:47.790911 9499 leader_election.cc:270] T a87a2a21d927456d930023d9775aec80 P a382b1ef929b411b825a324ad47ecaa9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0703 01:39:47.791260 10493 raft_consensus.cc:2730] T a87a2a21d927456d930023d9775aec80 P a382b1ef929b411b825a324ad47ecaa9: Snoozing failure detection for 1.641s (election complete) I0703 01:39:47.791402 10493 raft_consensus.cc:2537] T a87a2a21d927456d930023d9775aec80 P a382b1ef929b411b825a324ad47ecaa9 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0703 01:39:47.791517 10493 raft_consensus.cc:457] T a87a2a21d927456d930023d9775aec80 P a382b1ef929b411b825a324ad47ecaa9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0703 01:39:47.791602 10493 raft_consensus.cc:2730] T a87a2a21d927456d930023d9775aec80 P a382b1ef929b411b825a324ad47ecaa9: Snoozing failure detection for 1.659s (starting election) I0703 01:39:47.791666 10493 raft_consensus.cc:2782] T a87a2a21d927456d930023d9775aec80 P a382b1ef929b411b825a324ad47ecaa9 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:39:47.792698 9689 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "a87a2a21d927456d930023d9775aec80" candidate_uuid: "a382b1ef929b411b825a324ad47ecaa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08ad9c3e352f492f9acb05219b72d0e0" is_pre_election: true I0703 01:39:47.793306 9689 raft_consensus.cc:2730] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0: Snoozing failure detection for 2.062s (vote granted) I0703 01:39:47.793397 9689 raft_consensus.cc:2243] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a382b1ef929b411b825a324ad47ecaa9 in term 0. I0703 01:39:47.794062 9499 leader_election.cc:379] T a87a2a21d927456d930023d9775aec80 P a382b1ef929b411b825a324ad47ecaa9 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 08ad9c3e352f492f9acb05219b72d0e0 (127.5.240.194:42244) I0703 01:39:47.793959 10493 raft_consensus.cc:479] T a87a2a21d927456d930023d9775aec80 P a382b1ef929b411b825a324ad47ecaa9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } } peers { permanent_uuid: "a382b1ef929b411b825a324ad47ecaa9" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 48335 } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } } peers { permanent_uuid: "37b8cbcdace54a91a27cd7bbb1eefe17" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 47502 } } peers { permanent_uuid: "937ef3ea804c4775a2a090e89e55f0fd" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 47698 } } I0703 01:39:47.794528 10493 leader_election.cc:243] T a87a2a21d927456d930023d9775aec80 P a382b1ef929b411b825a324ad47ecaa9 [CANDIDATE]: Term 1 election: Requesting vote from peer 7bb94c92677448c39c50efc79e39459a (127.5.240.198:39947) I0703 01:39:47.794699 10493 leader_election.cc:243] T a87a2a21d927456d930023d9775aec80 P a382b1ef929b411b825a324ad47ecaa9 [CANDIDATE]: Term 1 election: Requesting vote from peer 08ad9c3e352f492f9acb05219b72d0e0 (127.5.240.194:42244) I0703 01:39:47.794823 10493 leader_election.cc:243] T a87a2a21d927456d930023d9775aec80 P a382b1ef929b411b825a324ad47ecaa9 [CANDIDATE]: Term 1 election: Requesting vote from peer 37b8cbcdace54a91a27cd7bbb1eefe17 (127.5.240.196:47502) I0703 01:39:47.795022 10493 leader_election.cc:243] T a87a2a21d927456d930023d9775aec80 P a382b1ef929b411b825a324ad47ecaa9 [CANDIDATE]: Term 1 election: Requesting vote from peer 937ef3ea804c4775a2a090e89e55f0fd (127.5.240.195:47698) I0703 01:39:47.795344 10177 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "a87a2a21d927456d930023d9775aec80" candidate_uuid: "a382b1ef929b411b825a324ad47ecaa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bb94c92677448c39c50efc79e39459a" I0703 01:39:47.795487 10177 raft_consensus.cc:2782] T a87a2a21d927456d930023d9775aec80 P 7bb94c92677448c39c50efc79e39459a [term 0 FOLLOWER]: Advancing to term 1 I0703 01:39:47.795619 10177 raft_consensus.cc:2730] T a87a2a21d927456d930023d9775aec80 P 7bb94c92677448c39c50efc79e39459a: Snoozing failure detection for 1.874s (vote granted) I0703 01:39:47.795588 9689 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "a87a2a21d927456d930023d9775aec80" candidate_uuid: "a382b1ef929b411b825a324ad47ecaa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08ad9c3e352f492f9acb05219b72d0e0" I0703 01:39:47.795596 9933 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "a87a2a21d927456d930023d9775aec80" candidate_uuid: "a382b1ef929b411b825a324ad47ecaa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37b8cbcdace54a91a27cd7bbb1eefe17" I0703 01:39:47.795722 9689 raft_consensus.cc:2782] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:39:47.795753 9933 raft_consensus.cc:2782] T a87a2a21d927456d930023d9775aec80 P 37b8cbcdace54a91a27cd7bbb1eefe17 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:39:47.795826 9689 raft_consensus.cc:2730] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0: Snoozing failure detection for 1.695s (vote granted) I0703 01:39:47.795841 9933 raft_consensus.cc:2730] T a87a2a21d927456d930023d9775aec80 P 37b8cbcdace54a91a27cd7bbb1eefe17: Snoozing failure detection for 2.068s (vote granted) I0703 01:39:47.795903 9811 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "a87a2a21d927456d930023d9775aec80" candidate_uuid: "a382b1ef929b411b825a324ad47ecaa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "937ef3ea804c4775a2a090e89e55f0fd" I0703 01:39:47.796036 9811 raft_consensus.cc:2782] T a87a2a21d927456d930023d9775aec80 P 937ef3ea804c4775a2a090e89e55f0fd [term 0 FOLLOWER]: Advancing to term 1 I0703 01:39:47.796130 9811 raft_consensus.cc:2730] T a87a2a21d927456d930023d9775aec80 P 937ef3ea804c4775a2a090e89e55f0fd: Snoozing failure detection for 2.033s (vote granted) I0703 01:39:47.797682 10177 raft_consensus.cc:2243] T a87a2a21d927456d930023d9775aec80 P 7bb94c92677448c39c50efc79e39459a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a382b1ef929b411b825a324ad47ecaa9 in term 1. I0703 01:39:47.798430 9933 raft_consensus.cc:2243] T a87a2a21d927456d930023d9775aec80 P 37b8cbcdace54a91a27cd7bbb1eefe17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a382b1ef929b411b825a324ad47ecaa9 in term 1. I0703 01:39:47.798494 9811 raft_consensus.cc:2243] T a87a2a21d927456d930023d9775aec80 P 937ef3ea804c4775a2a090e89e55f0fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a382b1ef929b411b825a324ad47ecaa9 in term 1. I0703 01:39:47.798629 9689 raft_consensus.cc:2243] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a382b1ef929b411b825a324ad47ecaa9 in term 1. I0703 01:39:47.798971 9497 leader_election.cc:379] T a87a2a21d927456d930023d9775aec80 P a382b1ef929b411b825a324ad47ecaa9 [CANDIDATE]: Term 1 election: Vote granted by peer 37b8cbcdace54a91a27cd7bbb1eefe17 (127.5.240.196:47502) I0703 01:39:47.799085 9498 leader_election.cc:379] T a87a2a21d927456d930023d9775aec80 P a382b1ef929b411b825a324ad47ecaa9 [CANDIDATE]: Term 1 election: Vote granted by peer 937ef3ea804c4775a2a090e89e55f0fd (127.5.240.195:47698) I0703 01:39:47.799178 9498 leader_election.cc:270] T a87a2a21d927456d930023d9775aec80 P a382b1ef929b411b825a324ad47ecaa9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:39:47.799293 9499 leader_election.cc:379] T a87a2a21d927456d930023d9775aec80 P a382b1ef929b411b825a324ad47ecaa9 [CANDIDATE]: Term 1 election: Vote granted by peer 7bb94c92677448c39c50efc79e39459a (127.5.240.198:39947) I0703 01:39:47.799381 10493 raft_consensus.cc:2730] T a87a2a21d927456d930023d9775aec80 P a382b1ef929b411b825a324ad47ecaa9: Snoozing failure detection for 1.512s (election complete) I0703 01:39:47.799511 10493 raft_consensus.cc:2537] T a87a2a21d927456d930023d9775aec80 P a382b1ef929b411b825a324ad47ecaa9 [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:39:47.799515 9499 leader_election.cc:379] T a87a2a21d927456d930023d9775aec80 P a382b1ef929b411b825a324ad47ecaa9 [CANDIDATE]: Term 1 election: Vote granted by peer 08ad9c3e352f492f9acb05219b72d0e0 (127.5.240.194:42244) I0703 01:39:47.799978 10493 raft_consensus.cc:593] T a87a2a21d927456d930023d9775aec80 P a382b1ef929b411b825a324ad47ecaa9 [term 1 LEADER]: Becoming Leader. State: Replica: a382b1ef929b411b825a324ad47ecaa9, State: Running, Role: LEADER I0703 01:39:47.800384 10493 consensus_queue.cc:204] T a87a2a21d927456d930023d9775aec80 P a382b1ef929b411b825a324ad47ecaa9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } } peers { permanent_uuid: "a382b1ef929b411b825a324ad47ecaa9" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 48335 } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } } peers { permanent_uuid: "37b8cbcdace54a91a27cd7bbb1eefe17" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 47502 } } peers { permanent_uuid: "937ef3ea804c4775a2a090e89e55f0fd" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 47698 } } I0703 01:39:47.805413 9446 catalog_manager.cc:3798] T a87a2a21d927456d930023d9775aec80 P a382b1ef929b411b825a324ad47ecaa9 reported cstate change: term changed from 0 to 1, leader changed from to a382b1ef929b411b825a324ad47ecaa9 (127.5.240.193). New cstate: current_term: 1 leader_uuid: "a382b1ef929b411b825a324ad47ecaa9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a382b1ef929b411b825a324ad47ecaa9" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 48335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37b8cbcdace54a91a27cd7bbb1eefe17" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 47502 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "937ef3ea804c4775a2a090e89e55f0fd" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 47698 } health_report { overall_health: UNKNOWN } } } I0703 01:39:47.836869 6083 external_mini_cluster.cc:490] 8 TS(s) registered with all masters I0703 01:39:47.838068 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver a382b1ef929b411b825a324ad47ecaa9 to finish bootstrapping I0703 01:39:47.842634 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 08ad9c3e352f492f9acb05219b72d0e0 to finish bootstrapping I0703 01:39:47.848179 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 937ef3ea804c4775a2a090e89e55f0fd to finish bootstrapping W0703 01:39:47.851418 9733 tablet.cc:1678] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:39:47.853827 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 37b8cbcdace54a91a27cd7bbb1eefe17 to finish bootstrapping I0703 01:39:47.858633 6083 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 3254ace7bc954d2ab9972a2723f45b0c to finish bootstrapping I0703 01:39:47.865057 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 7bb94c92677448c39c50efc79e39459a to finish bootstrapping I0703 01:39:47.871070 6083 ts_itest-base.cc:233] Waiting for 0 tablets on tserver add911cdebcf44b8a64256c228929d30 to finish bootstrapping I0703 01:39:47.877769 6083 ts_itest-base.cc:233] Waiting for 0 tablets on tserver d33993c1f2c148d9ae2504a60ed04173 to finish bootstrapping I0703 01:39:47.885272 6083 kudu-admin-test.cc:661] Waiting for Master to see the current replicas... I0703 01:39:47.886624 6083 kudu-admin-test.cc:668] Tablet locations: tablet_id: "a87a2a21d927456d930023d9775aec80" replicas { ts_info { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" rpc_addresses { host: "127.5.240.198" port: 39947 } } role: FOLLOWER } replicas { ts_info { permanent_uuid: "a382b1ef929b411b825a324ad47ecaa9" rpc_addresses { host: "127.5.240.193" port: 48335 } } role: LEADER } replicas { ts_info { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" rpc_addresses { host: "127.5.240.194" port: 42244 } } role: FOLLOWER } replicas { ts_info { permanent_uuid: "37b8cbcdace54a91a27cd7bbb1eefe17" rpc_addresses { host: "127.5.240.196" port: 47502 } } role: FOLLOWER } replicas { ts_info { permanent_uuid: "937ef3ea804c4775a2a090e89e55f0fd" rpc_addresses { host: "127.5.240.195" port: 47698 } } role: FOLLOWER } DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } W0703 01:39:47.954794 9855 tablet.cc:1678] T a87a2a21d927456d930023d9775aec80 P 937ef3ea804c4775a2a090e89e55f0fd: Can't schedule compaction. Clean time has not been advanced past its initial value. W0703 01:39:47.959291 10221 tablet.cc:1678] T a87a2a21d927456d930023d9775aec80 P 7bb94c92677448c39c50efc79e39459a: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:39:48.185015 9689 raft_consensus.cc:1081] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 [term 1 FOLLOWER]: Refusing update from remote peer a382b1ef929b411b825a324ad47ecaa9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0703 01:39:48.185860 10493 consensus_queue.cc:962] T a87a2a21d927456d930023d9775aec80 P a382b1ef929b411b825a324ad47ecaa9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0703 01:39:48.206043 10177 raft_consensus.cc:1081] T a87a2a21d927456d930023d9775aec80 P 7bb94c92677448c39c50efc79e39459a [term 1 FOLLOWER]: Refusing update from remote peer a382b1ef929b411b825a324ad47ecaa9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0703 01:39:48.206732 10493 consensus_queue.cc:962] T a87a2a21d927456d930023d9775aec80 P a382b1ef929b411b825a324ad47ecaa9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0703 01:39:48.212550 9811 raft_consensus.cc:1081] T a87a2a21d927456d930023d9775aec80 P 937ef3ea804c4775a2a090e89e55f0fd [term 1 FOLLOWER]: Refusing update from remote peer a382b1ef929b411b825a324ad47ecaa9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0703 01:39:48.216153 10516 consensus_queue.cc:962] T a87a2a21d927456d930023d9775aec80 P a382b1ef929b411b825a324ad47ecaa9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "937ef3ea804c4775a2a090e89e55f0fd" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 47698 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s I0703 01:39:48.219435 9933 raft_consensus.cc:1081] T a87a2a21d927456d930023d9775aec80 P 37b8cbcdace54a91a27cd7bbb1eefe17 [term 1 FOLLOWER]: Refusing update from remote peer a382b1ef929b411b825a324ad47ecaa9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0703 01:39:48.221127 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 9857 I0703 01:39:48.220924 10517 consensus_queue.cc:962] T a87a2a21d927456d930023d9775aec80 P a382b1ef929b411b825a324ad47ecaa9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37b8cbcdace54a91a27cd7bbb1eefe17" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 47502 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0703 01:39:48.221801 9973 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:39:48.268707 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 9857 W0703 01:39:48.276398 9497 connection.cc:512] client connection to 127.5.240.196:47502 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.196:47502): Connection reset by peer (error 104) W0703 01:39:48.276408 9429 connection.cc:512] server connection from 127.5.240.196:37344 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.196:37344): Connection reset by peer (error 104) I0703 01:39:48.276657 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 9735 W0703 01:39:48.276588 9497 consensus_peers.cc:433] T a87a2a21d927456d930023d9775aec80 P a382b1ef929b411b825a324ad47ecaa9 -> Peer 37b8cbcdace54a91a27cd7bbb1eefe17 (127.5.240.196:47502): Couldn't send request to peer 37b8cbcdace54a91a27cd7bbb1eefe17 for tablet a87a2a21d927456d930023d9775aec80. Status: Network error: failed to read from TLS socket (remote: 127.5.240.196:47502): Connection reset by peer (error 104). Retrying in the next heartbeat period. Already tried 1 times. I0703 01:39:48.277401 9851 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:39:48.320879 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 9735 I0703 01:39:48.322150 10342 heartbeater.cc:467] Master 127.5.240.254:54359 was elected leader, sending a full tablet report... W0703 01:39:48.329097 9427 connection.cc:512] server connection from 127.5.240.195:34496 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.195:34496): Connection reset by peer (error 104) W0703 01:39:48.329090 9498 connection.cc:512] client connection to 127.5.240.195:47698 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.195:47698): Connection reset by peer (error 104) I0703 01:39:48.329381 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 9480 I0703 01:39:48.330063 9606 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:39:48.375351 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 9480 W0703 01:39:48.383903 9429 connection.cc:512] server connection from 127.5.240.193:49586 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:49586): Connection reset by peer (error 104) W0703 01:39:48.383965 10111 connection.cc:512] server connection from 127.5.240.193:46995 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:46995): Connection reset by peer (error 104) W0703 01:39:48.384503 9620 connection.cc:512] server connection from 127.5.240.193:53400 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:53400): Connection reset by peer (error 104) I0703 01:39:48.384647 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master pid 9420 I0703 01:39:48.385351 9476 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:39:48.449291 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master with pid 9420 W0703 01:39:48.462524 9620 connection.cc:512] client connection to 127.5.240.254:54359 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.254:54359): Connection reset by peer (error 104) W0703 01:39:48.462554 10108 connection.cc:512] client connection to 127.5.240.254:54359 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.254:54359): Connection reset by peer (error 104) W0703 01:39:48.462612 9986 connection.cc:512] client connection to 127.5.240.254:54359 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.254:54359): Connection reset by peer (error 104) W0703 01:39:48.463125 10230 connection.cc:512] client connection to 127.5.240.254:54359 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.254:54359): Connection reset by peer (error 104) W0703 01:39:48.463160 10352 connection.cc:512] client connection to 127.5.240.254:54359 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.254:54359): Connection reset by peer (error 104) I0703 01:39:48.464114 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:54359 --webserver_interface=127.5.240.254 --webserver_port=34888 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --rpc_reuseport=true with env {} W0703 01:39:48.627032 10098 heartbeater.cc:587] Failed to heartbeat to 127.5.240.254:54359: Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.240.254:54359: connect: Connection refused (error 111) I0703 01:39:48.636262 10464 heartbeater.cc:467] Master 127.5.240.254:54359 was elected leader, sending a full tablet report... W0703 01:39:48.637854 10464 heartbeater.cc:587] Failed to heartbeat to 127.5.240.254:54359: Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.240.254:54359: connect: Connection refused (error 111) WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:39:48.647156 10523 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:39:48.647368 10523 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:39:48.649593 10523 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0703 01:39:48.649641 10523 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:39:48.649685 10523 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0703 01:39:48.649736 10523 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0703 01:39:48.657685 10523 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:54359 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.5.240.254 --webserver_port=34888 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:39:48.663223 10523 master_main.cc:80] Initializing master server... I0703 01:39:48.664325 10523 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 147708us I0703 01:39:48.665383 10523 fs_manager.cc:261] Metadata directory not provided I0703 01:39:48.665449 10523 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:39:48.672420 10523 fs_manager.cc:397] Time spent opening directory manager: real 0.005s user 0.001s sys 0.003s I0703 01:39:48.673262 10523 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:39:48.673715 10523 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:39:48.677270 10523 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0703 01:39:48.677340 10523 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal uuid: "43575583f1234fdd9a8d9f6431ac9e65" format_stamp: "Formatted at 2018-07-03 08:39:44 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:48.678205 10523 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:39:48.727303 10523 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:39:48.728233 10523 master_main.cc:83] Starting Master server... I0703 01:39:48.764961 10523 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.254:54359 I0703 01:39:48.765462 10523 webserver.cc:175] Starting webserver on 127.5.240.254:34888 I0703 01:39:48.765509 10523 webserver.cc:186] Document root disabled I0703 01:39:48.766693 10523 webserver.cc:313] Webserver started. Bound to: http://127.5.240.254:34888/ I0703 01:39:48.767351 10523 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb I0703 01:39:48.775586 10584 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 43575583f1234fdd9a8d9f6431ac9e65: Bootstrap starting. I0703 01:39:48.775760 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master as pid 10523 I0703 01:39:48.775918 6083 kudu-admin-test.cc:686] Forcing unsafe config change on tserver 08ad9c3e352f492f9acb05219b72d0e0 I0703 01:39:48.788048 10584 log.cc:525] T 00000000000000000000000000000000 P 43575583f1234fdd9a8d9f6431ac9e65: Log is configured to *not* fsync() on all Append() calls I0703 01:39:48.796586 10584 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 43575583f1234fdd9a8d9f6431ac9e65: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=1} inserts{seen=4 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:39:48.797029 10584 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 43575583f1234fdd9a8d9f6431ac9e65: Bootstrap complete. I0703 01:39:48.806231 10584 raft_consensus.cc:325] T 00000000000000000000000000000000 P 43575583f1234fdd9a8d9f6431ac9e65 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43575583f1234fdd9a8d9f6431ac9e65" member_type: VOTER } I0703 01:39:48.806960 10584 raft_consensus.cc:628] T 00000000000000000000000000000000 P 43575583f1234fdd9a8d9f6431ac9e65 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43575583f1234fdd9a8d9f6431ac9e65, State: Initialized, Role: FOLLOWER I0703 01:39:48.807423 10584 consensus_queue.cc:227] T 00000000000000000000000000000000 P 43575583f1234fdd9a8d9f6431ac9e65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43575583f1234fdd9a8d9f6431ac9e65" member_type: VOTER } I0703 01:39:48.807561 10584 raft_consensus.cc:365] T 00000000000000000000000000000000 P 43575583f1234fdd9a8d9f6431ac9e65 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:39:48.807657 10584 raft_consensus.cc:457] T 00000000000000000000000000000000 P 43575583f1234fdd9a8d9f6431ac9e65 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:39:48.807739 10584 raft_consensus.cc:2730] T 00000000000000000000000000000000 P 43575583f1234fdd9a8d9f6431ac9e65: Snoozing failure detection for 1.591s (starting election) I0703 01:39:48.807823 10584 raft_consensus.cc:2782] T 00000000000000000000000000000000 P 43575583f1234fdd9a8d9f6431ac9e65 [term 1 FOLLOWER]: Advancing to term 2 I0703 01:39:48.811285 10584 raft_consensus.cc:479] T 00000000000000000000000000000000 P 43575583f1234fdd9a8d9f6431ac9e65 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43575583f1234fdd9a8d9f6431ac9e65" member_type: VOTER } I0703 01:39:48.811667 10584 leader_election.cc:270] T 00000000000000000000000000000000 P 43575583f1234fdd9a8d9f6431ac9e65 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0703 01:39:48.812768 10587 raft_consensus.cc:2730] T 00000000000000000000000000000000 P 43575583f1234fdd9a8d9f6431ac9e65: Snoozing failure detection for 1.778s (election complete) I0703 01:39:48.813019 10587 raft_consensus.cc:2537] T 00000000000000000000000000000000 P 43575583f1234fdd9a8d9f6431ac9e65 [term 2 FOLLOWER]: Leader election won for term 2 I0703 01:39:48.815034 10587 raft_consensus.cc:593] T 00000000000000000000000000000000 P 43575583f1234fdd9a8d9f6431ac9e65 [term 2 LEADER]: Becoming Leader. State: Replica: 43575583f1234fdd9a8d9f6431ac9e65, State: Running, Role: LEADER I0703 01:39:48.815505 10587 consensus_queue.cc:204] T 00000000000000000000000000000000 P 43575583f1234fdd9a8d9f6431ac9e65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43575583f1234fdd9a8d9f6431ac9e65" member_type: VOTER } I0703 01:39:48.817380 10584 sys_catalog.cc:337] T 00000000000000000000000000000000 P 43575583f1234fdd9a8d9f6431ac9e65 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 2 leader_uuid: "43575583f1234fdd9a8d9f6431ac9e65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43575583f1234fdd9a8d9f6431ac9e65" member_type: VOTER } } I0703 01:39:48.817589 10584 sys_catalog.cc:340] T 00000000000000000000000000000000 P 43575583f1234fdd9a8d9f6431ac9e65 [sys.catalog]: This master's current role is: LEADER I0703 01:39:48.817466 10588 sys_catalog.cc:337] T 00000000000000000000000000000000 P 43575583f1234fdd9a8d9f6431ac9e65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "43575583f1234fdd9a8d9f6431ac9e65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43575583f1234fdd9a8d9f6431ac9e65" member_type: VOTER } } I0703 01:39:48.818307 10588 sys_catalog.cc:340] T 00000000000000000000000000000000 P 43575583f1234fdd9a8d9f6431ac9e65 [sys.catalog]: This master's current role is: LEADER I0703 01:39:48.819360 10589 sys_catalog.cc:337] T 00000000000000000000000000000000 P 43575583f1234fdd9a8d9f6431ac9e65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43575583f1234fdd9a8d9f6431ac9e65. Latest consensus state: current_term: 2 leader_uuid: "43575583f1234fdd9a8d9f6431ac9e65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43575583f1234fdd9a8d9f6431ac9e65" member_type: VOTER } } I0703 01:39:48.819974 10589 sys_catalog.cc:340] T 00000000000000000000000000000000 P 43575583f1234fdd9a8d9f6431ac9e65 [sys.catalog]: This master's current role is: LEADER I0703 01:39:48.820427 10584 sys_catalog.cc:424] T 00000000000000000000000000000000 P 43575583f1234fdd9a8d9f6431ac9e65 [sys.catalog]: configured and running, proceeding with master startup. I0703 01:39:48.820756 10591 catalog_manager.cc:1006] Loading table and tablet metadata into memory... I0703 01:39:48.830265 10592 catalog_manager.cc:505] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting I0703 01:39:48.830747 10523 master_main.cc:86] Master server successfully started. I0703 01:39:48.835834 10591 catalog_manager.cc:323] Loaded metadata for table TestTable [id=d45069b9f7d0418dbe6b2bc688216e4f] I0703 01:39:48.838048 10591 catalog_manager.cc:377] Loaded metadata for tablet a87a2a21d927456d930023d9775aec80 (table TestTable [id=d45069b9f7d0418dbe6b2bc688216e4f]) I0703 01:39:48.838291 10591 catalog_manager.cc:1017] Initializing Kudu internal certificate authority... I0703 01:39:48.842890 10591 catalog_manager.cc:1026] Loading token signing keys... I0703 01:39:48.844854 10591 catalog_manager.cc:4113] T 00000000000000000000000000000000 P 43575583f1234fdd9a8d9f6431ac9e65: Loaded TSK: 0 W0703 01:39:48.999568 10585 flags.cc:406] Enabled unsafe flag: --never_fsync=true I0703 01:39:49.050701 9689 tablet_service.cc:1077] Received UnsafeChangeConfig RPC: dest_uuid: "08ad9c3e352f492f9acb05219b72d0e0" tablet_id: "a87a2a21d927456d930023d9775aec80" caller_id: "kudu-tools" new_config { peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" } } from {username='jenkins'} at 127.5.240.194:51039 W0703 01:39:49.051148 9689 raft_consensus.cc:2017] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 [term 1 FOLLOWER]: PROCEEDING WITH UNSAFE CONFIG CHANGE ON THIS SERVER, COMMITTED CONFIG: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } } peers { permanent_uuid: "a382b1ef929b411b825a324ad47ecaa9" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 48335 } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } } peers { permanent_uuid: "37b8cbcdace54a91a27cd7bbb1eefe17" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 47502 } } peers { permanent_uuid: "937ef3ea804c4775a2a090e89e55f0fd" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 47698 } }NEW CONFIG: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } } unsafe_config_change: true I0703 01:39:49.051442 9689 raft_consensus.cc:2782] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 [term 1 FOLLOWER]: Advancing to term 2 I0703 01:39:49.069818 10551 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" instance_seqno: 1530607185578621) as {username='jenkins'} at 127.5.240.194:43161; Asking this server to re-register. I0703 01:39:49.071017 9732 heartbeater.cc:426] Registering TS with master... I0703 01:39:49.071152 9732 heartbeater.cc:475] Master 127.5.240.254:54359 requested a full tablet report, sending... I0703 01:39:49.072414 10551 ts_manager.cc:98] Registered new tserver with Master: 08ad9c3e352f492f9acb05219b72d0e0 (127.5.240.194:42244) I0703 01:39:49.218955 10551 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "7bb94c92677448c39c50efc79e39459a" instance_seqno: 1530607186937767) as {username='jenkins'} at 127.5.240.198:37752; Asking this server to re-register. I0703 01:39:49.219499 10220 heartbeater.cc:426] Registering TS with master... I0703 01:39:49.219646 10220 heartbeater.cc:475] Master 127.5.240.254:54359 requested a full tablet report, sending... I0703 01:39:49.220444 10551 ts_manager.cc:98] Registered new tserver with Master: 7bb94c92677448c39c50efc79e39459a (127.5.240.198:39947) I0703 01:39:49.328547 10551 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "add911cdebcf44b8a64256c228929d30" instance_seqno: 1530607187282947) as {username='jenkins'} at 127.5.240.199:51962; Asking this server to re-register. I0703 01:39:49.329089 10342 heartbeater.cc:426] Registering TS with master... I0703 01:39:49.329202 10342 heartbeater.cc:475] Master 127.5.240.254:54359 requested a full tablet report, sending... I0703 01:39:49.329871 10551 ts_manager.cc:98] Registered new tserver with Master: add911cdebcf44b8a64256c228929d30 (127.5.240.199:59090) I0703 01:39:49.632149 10098 heartbeater.cc:346] Connected to a master server at 127.5.240.254:54359 I0703 01:39:49.632867 10551 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "3254ace7bc954d2ab9972a2723f45b0c" instance_seqno: 1530607186582423) as {username='jenkins'} at 127.5.240.197:50542; Asking this server to re-register. I0703 01:39:49.633411 10098 heartbeater.cc:426] Registering TS with master... I0703 01:39:49.633540 10098 heartbeater.cc:475] Master 127.5.240.254:54359 requested a full tablet report, sending... I0703 01:39:49.634251 10551 ts_manager.cc:98] Registered new tserver with Master: 3254ace7bc954d2ab9972a2723f45b0c (127.5.240.197:54985) I0703 01:39:49.642534 10464 heartbeater.cc:346] Connected to a master server at 127.5.240.254:54359 I0703 01:39:49.642647 10464 heartbeater.cc:467] Master 127.5.240.254:54359 was elected leader, sending a full tablet report... I0703 01:39:49.643302 10551 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "d33993c1f2c148d9ae2504a60ed04173" instance_seqno: 1530607187596524) as {username='jenkins'} at 127.5.240.200:34330; Asking this server to re-register. I0703 01:39:49.643852 10464 heartbeater.cc:426] Registering TS with master... I0703 01:39:49.643975 10464 heartbeater.cc:475] Master 127.5.240.254:54359 requested a full tablet report, sending... I0703 01:39:49.644619 10551 ts_manager.cc:98] Registered new tserver with Master: d33993c1f2c148d9ae2504a60ed04173 (127.5.240.200:54337) I0703 01:39:49.712738 10618 raft_consensus.cc:457] T a87a2a21d927456d930023d9775aec80 P 7bb94c92677448c39c50efc79e39459a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a382b1ef929b411b825a324ad47ecaa9) I0703 01:39:49.712817 10618 raft_consensus.cc:2730] T a87a2a21d927456d930023d9775aec80 P 7bb94c92677448c39c50efc79e39459a: Snoozing failure detection for 1.723s (starting election) I0703 01:39:49.712909 10618 raft_consensus.cc:479] T a87a2a21d927456d930023d9775aec80 P 7bb94c92677448c39c50efc79e39459a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } } peers { permanent_uuid: "a382b1ef929b411b825a324ad47ecaa9" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 48335 } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } } peers { permanent_uuid: "37b8cbcdace54a91a27cd7bbb1eefe17" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 47502 } } peers { permanent_uuid: "937ef3ea804c4775a2a090e89e55f0fd" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 47698 } } I0703 01:39:49.713893 10618 leader_election.cc:243] T a87a2a21d927456d930023d9775aec80 P 7bb94c92677448c39c50efc79e39459a [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer a382b1ef929b411b825a324ad47ecaa9 (127.5.240.193:48335) I0703 01:39:49.714174 10618 leader_election.cc:243] T a87a2a21d927456d930023d9775aec80 P 7bb94c92677448c39c50efc79e39459a [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 08ad9c3e352f492f9acb05219b72d0e0 (127.5.240.194:42244) I0703 01:39:49.714329 10618 leader_election.cc:243] T a87a2a21d927456d930023d9775aec80 P 7bb94c92677448c39c50efc79e39459a [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 37b8cbcdace54a91a27cd7bbb1eefe17 (127.5.240.196:47502) I0703 01:39:49.714517 10618 leader_election.cc:243] T a87a2a21d927456d930023d9775aec80 P 7bb94c92677448c39c50efc79e39459a [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 937ef3ea804c4775a2a090e89e55f0fd (127.5.240.195:47698) W0703 01:39:49.719976 10108 leader_election.cc:299] T a87a2a21d927456d930023d9775aec80 P 7bb94c92677448c39c50efc79e39459a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a382b1ef929b411b825a324ad47ecaa9 (127.5.240.193:48335): Network error: Client connection negotiation failed: client connection to 127.5.240.193:48335: connect: Connection refused (error 111) W0703 01:39:49.720422 10108 leader_election.cc:299] T a87a2a21d927456d930023d9775aec80 P 7bb94c92677448c39c50efc79e39459a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 37b8cbcdace54a91a27cd7bbb1eefe17 (127.5.240.196:47502): Network error: Client connection negotiation failed: client connection to 127.5.240.196:47502: connect: Connection refused (error 111) W0703 01:39:49.721987 10109 leader_election.cc:299] T a87a2a21d927456d930023d9775aec80 P 7bb94c92677448c39c50efc79e39459a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 937ef3ea804c4775a2a090e89e55f0fd (127.5.240.195:47698): Network error: Client connection negotiation failed: client connection to 127.5.240.195:47698: connect: Connection refused (error 111) I0703 01:39:49.722178 10109 leader_election.cc:270] T a87a2a21d927456d930023d9775aec80 P 7bb94c92677448c39c50efc79e39459a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0703 01:39:49.722493 10618 raft_consensus.cc:2730] T a87a2a21d927456d930023d9775aec80 P 7bb94c92677448c39c50efc79e39459a: Snoozing failure detection for 2.156s (election complete) I0703 01:39:49.722610 10618 raft_consensus.cc:2485] T a87a2a21d927456d930023d9775aec80 P 7bb94c92677448c39c50efc79e39459a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0703 01:39:49.724831 9689 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "a87a2a21d927456d930023d9775aec80" candidate_uuid: "7bb94c92677448c39c50efc79e39459a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "08ad9c3e352f492f9acb05219b72d0e0" is_pre_election: true I0703 01:39:49.724990 9689 raft_consensus.cc:2204] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7bb94c92677448c39c50efc79e39459a for term 2 because replica is either leader or believes a valid leader to be alive. I0703 01:39:49.725505 10110 leader_election.cc:393] T a87a2a21d927456d930023d9775aec80 P 7bb94c92677448c39c50efc79e39459a [CANDIDATE]: Term 2 pre-election: Vote denied by peer 08ad9c3e352f492f9acb05219b72d0e0 (127.5.240.194:42244). Message: Invalid argument: T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7bb94c92677448c39c50efc79e39459a for term 2 because replica is either leader or believes a valid leader to be alive. I0703 01:39:50.554193 10622 raft_consensus.cc:457] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader kudu-tools) I0703 01:39:50.554270 10622 raft_consensus.cc:2730] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0: Snoozing failure detection for 1.645s (starting election) I0703 01:39:50.554374 10622 raft_consensus.cc:479] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } } unsafe_config_change: true I0703 01:39:50.554999 10622 leader_election.cc:243] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 7bb94c92677448c39c50efc79e39459a (127.5.240.198:39947) I0703 01:39:50.559345 10177 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "a87a2a21d927456d930023d9775aec80" candidate_uuid: "08ad9c3e352f492f9acb05219b72d0e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7bb94c92677448c39c50efc79e39459a" is_pre_election: true I0703 01:39:50.559489 10177 raft_consensus.cc:2730] T a87a2a21d927456d930023d9775aec80 P 7bb94c92677448c39c50efc79e39459a: Snoozing failure detection for 2.955s (vote granted) I0703 01:39:50.559579 10177 raft_consensus.cc:2243] T a87a2a21d927456d930023d9775aec80 P 7bb94c92677448c39c50efc79e39459a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08ad9c3e352f492f9acb05219b72d0e0 in term 1. I0703 01:39:50.560034 9622 leader_election.cc:379] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 [CANDIDATE]: Term 3 pre-election: Vote granted by peer 7bb94c92677448c39c50efc79e39459a (127.5.240.198:39947) I0703 01:39:50.560104 9622 leader_election.cc:270] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. I0703 01:39:50.560418 10622 raft_consensus.cc:2730] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0: Snoozing failure detection for 1.775s (election complete) I0703 01:39:50.560508 10622 raft_consensus.cc:2537] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 [term 2 FOLLOWER]: Leader pre-election won for term 3 I0703 01:39:50.560556 10622 raft_consensus.cc:457] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader kudu-tools) I0703 01:39:50.560590 10622 raft_consensus.cc:2730] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0: Snoozing failure detection for 2.098s (starting election) I0703 01:39:50.560645 10622 raft_consensus.cc:2782] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 [term 2 FOLLOWER]: Advancing to term 3 I0703 01:39:50.561960 10622 raft_consensus.cc:479] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } } unsafe_config_change: true I0703 01:39:50.562193 10622 leader_election.cc:243] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 [CANDIDATE]: Term 3 election: Requesting vote from peer 7bb94c92677448c39c50efc79e39459a (127.5.240.198:39947) I0703 01:39:50.562737 10177 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "a87a2a21d927456d930023d9775aec80" candidate_uuid: "08ad9c3e352f492f9acb05219b72d0e0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7bb94c92677448c39c50efc79e39459a" I0703 01:39:50.562842 10177 raft_consensus.cc:2782] T a87a2a21d927456d930023d9775aec80 P 7bb94c92677448c39c50efc79e39459a [term 1 FOLLOWER]: Advancing to term 3 I0703 01:39:50.562923 10177 raft_consensus.cc:2730] T a87a2a21d927456d930023d9775aec80 P 7bb94c92677448c39c50efc79e39459a: Snoozing failure detection for 1.771s (vote granted) I0703 01:39:50.564205 10177 raft_consensus.cc:2243] T a87a2a21d927456d930023d9775aec80 P 7bb94c92677448c39c50efc79e39459a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08ad9c3e352f492f9acb05219b72d0e0 in term 3. I0703 01:39:50.564561 9622 leader_election.cc:379] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 [CANDIDATE]: Term 3 election: Vote granted by peer 7bb94c92677448c39c50efc79e39459a (127.5.240.198:39947) I0703 01:39:50.564616 9622 leader_election.cc:270] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. I0703 01:39:50.564796 10622 raft_consensus.cc:2730] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0: Snoozing failure detection for 1.843s (election complete) I0703 01:39:50.564863 10622 raft_consensus.cc:2537] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 [term 3 FOLLOWER]: Leader election won for term 3 I0703 01:39:50.565131 10622 raft_consensus.cc:593] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 [term 3 LEADER]: Becoming Leader. State: Replica: 08ad9c3e352f492f9acb05219b72d0e0, State: Running, Role: LEADER I0703 01:39:50.565500 10622 consensus_queue.cc:204] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 0, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } } unsafe_config_change: true I0703 01:39:50.567903 10551 catalog_manager.cc:3798] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 reported cstate change: term changed from 1 to 3, leader changed from a382b1ef929b411b825a324ad47ecaa9 (127.5.240.193) to 08ad9c3e352f492f9acb05219b72d0e0 (127.5.240.194), now has a pending config: VOTER 7bb94c92677448c39c50efc79e39459a (127.5.240.198), VOTER 08ad9c3e352f492f9acb05219b72d0e0 (127.5.240.194). New cstate: current_term: 3 leader_uuid: "08ad9c3e352f492f9acb05219b72d0e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a382b1ef929b411b825a324ad47ecaa9" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 48335 } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37b8cbcdace54a91a27cd7bbb1eefe17" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 47502 } } peers { permanent_uuid: "937ef3ea804c4775a2a090e89e55f0fd" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 47698 } } } pending_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } } unsafe_config_change: true } I0703 01:39:51.135185 10177 raft_consensus.cc:1081] T a87a2a21d927456d930023d9775aec80 P 7bb94c92677448c39c50efc79e39459a [term 3 FOLLOWER]: Refusing update from remote peer 08ad9c3e352f492f9acb05219b72d0e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch) I0703 01:39:51.135866 10631 consensus_queue.cc:962] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s I0703 01:39:51.139209 10634 raft_consensus.cc:2684] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 [term 3 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, VOTER 37b8cbcdace54a91a27cd7bbb1eefe17 (127.5.240.196) evicted, VOTER 937ef3ea804c4775a2a090e89e55f0fd (127.5.240.195) evicted, VOTER a382b1ef929b411b825a324ad47ecaa9 (127.5.240.193) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } } unsafe_config_change: true } I0703 01:39:51.139986 10177 raft_consensus.cc:2684] T a87a2a21d927456d930023d9775aec80 P 7bb94c92677448c39c50efc79e39459a [term 3 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, VOTER 37b8cbcdace54a91a27cd7bbb1eefe17 (127.5.240.196) evicted, VOTER 937ef3ea804c4775a2a090e89e55f0fd (127.5.240.195) evicted, VOTER a382b1ef929b411b825a324ad47ecaa9 (127.5.240.193) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } } unsafe_config_change: true } I0703 01:39:51.142700 10551 catalog_manager.cc:3798] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 reported cstate change: config changed from index -1 to 2, VOTER 37b8cbcdace54a91a27cd7bbb1eefe17 (127.5.240.196) evicted, VOTER 937ef3ea804c4775a2a090e89e55f0fd (127.5.240.195) evicted, VOTER a382b1ef929b411b825a324ad47ecaa9 (127.5.240.193) evicted, no longer has a pending config: VOTER 7bb94c92677448c39c50efc79e39459a (127.5.240.198), VOTER 08ad9c3e352f492f9acb05219b72d0e0 (127.5.240.194). New cstate: current_term: 3 leader_uuid: "08ad9c3e352f492f9acb05219b72d0e0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } health_report { overall_health: HEALTHY } } unsafe_config_change: true } I0703 01:39:51.146131 10551 catalog_manager.cc:2903] Scheduling retry of a87a2a21d927456d930023d9775aec80 Delete Tablet RPC for TS=a382b1ef929b411b825a324ad47ecaa9 with a delay of 26 ms (attempt = 1) W0703 01:39:51.146337 10551 catalog_manager.cc:3957] Failed to send a87a2a21d927456d930023d9775aec80 Delete Tablet RPC for TS=a382b1ef929b411b825a324ad47ecaa9: Not found: failed to reset TS proxy: Could not find TS for UUID a382b1ef929b411b825a324ad47ecaa9 I0703 01:39:51.146450 10551 catalog_manager.cc:2903] Scheduling retry of a87a2a21d927456d930023d9775aec80 Delete Tablet RPC for TS=37b8cbcdace54a91a27cd7bbb1eefe17 with a delay of 21 ms (attempt = 1) W0703 01:39:51.146508 10551 catalog_manager.cc:3957] Failed to send a87a2a21d927456d930023d9775aec80 Delete Tablet RPC for TS=37b8cbcdace54a91a27cd7bbb1eefe17: Not found: failed to reset TS proxy: Could not find TS for UUID 37b8cbcdace54a91a27cd7bbb1eefe17 I0703 01:39:51.146627 10551 catalog_manager.cc:2903] Scheduling retry of a87a2a21d927456d930023d9775aec80 Delete Tablet RPC for TS=937ef3ea804c4775a2a090e89e55f0fd with a delay of 34 ms (attempt = 1) W0703 01:39:51.146689 10551 catalog_manager.cc:3957] Failed to send a87a2a21d927456d930023d9775aec80 Delete Tablet RPC for TS=937ef3ea804c4775a2a090e89e55f0fd: Not found: failed to reset TS proxy: Could not find TS for UUID 937ef3ea804c4775a2a090e89e55f0fd I0703 01:39:51.147100 10551 catalog_manager.cc:3437] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet a87a2a21d927456d930023d9775aec80 (attempt 1) I0703 01:39:51.153858 9689 consensus_peers.cc:455] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 -> Peer 7bb94c92677448c39c50efc79e39459a (127.5.240.198:39947): Closing peer: 7bb94c92677448c39c50efc79e39459a I0703 01:39:51.154086 9689 consensus_queue.cc:204] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 0, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } } peers { permanent_uuid: "d33993c1f2c148d9ae2504a60ed04173" member_type: NON_VOTER last_known_addr { host: "127.5.240.200" port: 54337 } attrs { promote: true } } unsafe_config_change: true I0703 01:39:51.155750 10177 raft_consensus.cc:1081] T a87a2a21d927456d930023d9775aec80 P 7bb94c92677448c39c50efc79e39459a [term 3 FOLLOWER]: Refusing update from remote peer 08ad9c3e352f492f9acb05219b72d0e0: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch) I0703 01:39:51.156370 10634 consensus_queue.cc:962] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I0703 01:39:51.158401 10631 raft_consensus.cc:2684] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index 2 to 4, NON_VOTER d33993c1f2c148d9ae2504a60ed04173 (127.5.240.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } } peers { permanent_uuid: "d33993c1f2c148d9ae2504a60ed04173" member_type: NON_VOTER last_known_addr { host: "127.5.240.200" port: 54337 } attrs { promote: true } } unsafe_config_change: true } I0703 01:39:51.159456 10177 raft_consensus.cc:2684] T a87a2a21d927456d930023d9775aec80 P 7bb94c92677448c39c50efc79e39459a [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 2 to 4, NON_VOTER d33993c1f2c148d9ae2504a60ed04173 (127.5.240.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } } peers { permanent_uuid: "d33993c1f2c148d9ae2504a60ed04173" member_type: NON_VOTER last_known_addr { host: "127.5.240.200" port: 54337 } attrs { promote: true } } unsafe_config_change: true } I0703 01:39:51.162714 10534 catalog_manager.cc:3352] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a87a2a21d927456d930023d9775aec80 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0703 01:39:51.163667 10551 catalog_manager.cc:3798] T a87a2a21d927456d930023d9775aec80 P 7bb94c92677448c39c50efc79e39459a reported cstate change: config changed from index 2 to 4, NON_VOTER d33993c1f2c148d9ae2504a60ed04173 (127.5.240.200) added. New cstate: current_term: 3 leader_uuid: "08ad9c3e352f492f9acb05219b72d0e0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } } peers { permanent_uuid: "d33993c1f2c148d9ae2504a60ed04173" member_type: NON_VOTER last_known_addr { host: "127.5.240.200" port: 54337 } attrs { promote: true } } unsafe_config_change: true } W0703 01:39:51.164307 9621 consensus_peers.cc:433] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 -> Peer d33993c1f2c148d9ae2504a60ed04173 (127.5.240.200:54337): Couldn't send request to peer d33993c1f2c148d9ae2504a60ed04173 for tablet a87a2a21d927456d930023d9775aec80. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a87a2a21d927456d930023d9775aec80. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:39:51.166198 10550 catalog_manager.cc:3437] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet a87a2a21d927456d930023d9775aec80 (attempt 1) I0703 01:39:51.166995 9689 consensus_peers.cc:455] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 -> Peer d33993c1f2c148d9ae2504a60ed04173 (127.5.240.200:54337): Closing peer: d33993c1f2c148d9ae2504a60ed04173 I0703 01:39:51.167063 9689 consensus_peers.cc:455] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 -> Peer 7bb94c92677448c39c50efc79e39459a (127.5.240.198:39947): Closing peer: 7bb94c92677448c39c50efc79e39459a I0703 01:39:51.167244 9689 consensus_queue.cc:204] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 0, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } } peers { permanent_uuid: "d33993c1f2c148d9ae2504a60ed04173" member_type: NON_VOTER last_known_addr { host: "127.5.240.200" port: 54337 } attrs { promote: true } } peers { permanent_uuid: "add911cdebcf44b8a64256c228929d30" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 59090 } attrs { promote: true } } unsafe_config_change: true I0703 01:39:51.167825 10534 catalog_manager.cc:2903] Scheduling retry of a87a2a21d927456d930023d9775aec80 Delete Tablet RPC for TS=37b8cbcdace54a91a27cd7bbb1eefe17 with a delay of 74 ms (attempt = 2) W0703 01:39:51.167906 10534 catalog_manager.cc:2922] Async tablet task a87a2a21d927456d930023d9775aec80 Delete Tablet RPC for TS=37b8cbcdace54a91a27cd7bbb1eefe17 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 37b8cbcdace54a91a27cd7bbb1eefe17 W0703 01:39:51.169484 9621 consensus_peers.cc:433] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 -> Peer d33993c1f2c148d9ae2504a60ed04173 (127.5.240.200:54337): Couldn't send request to peer d33993c1f2c148d9ae2504a60ed04173 for tablet a87a2a21d927456d930023d9775aec80. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a87a2a21d927456d930023d9775aec80. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:39:51.171901 10177 raft_consensus.cc:1081] T a87a2a21d927456d930023d9775aec80 P 7bb94c92677448c39c50efc79e39459a [term 3 FOLLOWER]: Refusing update from remote peer 08ad9c3e352f492f9acb05219b72d0e0: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch) I0703 01:39:51.172885 10532 catalog_manager.cc:2903] Scheduling retry of a87a2a21d927456d930023d9775aec80 Delete Tablet RPC for TS=a382b1ef929b411b825a324ad47ecaa9 with a delay of 64 ms (attempt = 2) W0703 01:39:51.172962 10532 catalog_manager.cc:2922] Async tablet task a87a2a21d927456d930023d9775aec80 Delete Tablet RPC for TS=a382b1ef929b411b825a324ad47ecaa9 failed: Not found: failed to reset TS proxy: Could not find TS for UUID a382b1ef929b411b825a324ad47ecaa9 I0703 01:39:51.172610 10642 consensus_queue.cc:962] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I0703 01:39:51.174861 10631 raft_consensus.cc:2684] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, NON_VOTER add911cdebcf44b8a64256c228929d30 (127.5.240.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } } peers { permanent_uuid: "d33993c1f2c148d9ae2504a60ed04173" member_type: NON_VOTER last_known_addr { host: "127.5.240.200" port: 54337 } attrs { promote: true } } peers { permanent_uuid: "add911cdebcf44b8a64256c228929d30" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 59090 } attrs { promote: true } } unsafe_config_change: true } W0703 01:39:51.175992 9620 consensus_peers.cc:433] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 -> Peer add911cdebcf44b8a64256c228929d30 (127.5.240.199:59090): Couldn't send request to peer add911cdebcf44b8a64256c228929d30 for tablet a87a2a21d927456d930023d9775aec80. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a87a2a21d927456d930023d9775aec80. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:39:51.175693 10177 raft_consensus.cc:2684] T a87a2a21d927456d930023d9775aec80 P 7bb94c92677448c39c50efc79e39459a [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, NON_VOTER add911cdebcf44b8a64256c228929d30 (127.5.240.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } } peers { permanent_uuid: "d33993c1f2c148d9ae2504a60ed04173" member_type: NON_VOTER last_known_addr { host: "127.5.240.200" port: 54337 } attrs { promote: true } } peers { permanent_uuid: "add911cdebcf44b8a64256c228929d30" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 59090 } attrs { promote: true } } unsafe_config_change: true } I0703 01:39:51.177634 10534 catalog_manager.cc:3352] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a87a2a21d927456d930023d9775aec80 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0703 01:39:51.178494 10550 catalog_manager.cc:3798] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 reported cstate change: config changed from index 4 to 5, NON_VOTER add911cdebcf44b8a64256c228929d30 (127.5.240.199) added. New cstate: current_term: 3 leader_uuid: "08ad9c3e352f492f9acb05219b72d0e0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d33993c1f2c148d9ae2504a60ed04173" member_type: NON_VOTER last_known_addr { host: "127.5.240.200" port: 54337 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "add911cdebcf44b8a64256c228929d30" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 59090 } attrs { promote: true } health_report { overall_health: UNKNOWN } } unsafe_config_change: true } I0703 01:39:51.180622 10550 catalog_manager.cc:3437] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet a87a2a21d927456d930023d9775aec80 (attempt 1) I0703 01:39:51.181243 10535 catalog_manager.cc:2903] Scheduling retry of a87a2a21d927456d930023d9775aec80 Delete Tablet RPC for TS=937ef3ea804c4775a2a090e89e55f0fd with a delay of 78 ms (attempt = 2) W0703 01:39:51.181320 10535 catalog_manager.cc:2922] Async tablet task a87a2a21d927456d930023d9775aec80 Delete Tablet RPC for TS=937ef3ea804c4775a2a090e89e55f0fd failed: Not found: failed to reset TS proxy: Could not find TS for UUID 937ef3ea804c4775a2a090e89e55f0fd I0703 01:39:51.181426 9689 consensus_peers.cc:455] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 -> Peer add911cdebcf44b8a64256c228929d30 (127.5.240.199:59090): Closing peer: add911cdebcf44b8a64256c228929d30 I0703 01:39:51.181527 9689 consensus_peers.cc:455] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 -> Peer d33993c1f2c148d9ae2504a60ed04173 (127.5.240.200:54337): Closing peer: d33993c1f2c148d9ae2504a60ed04173 I0703 01:39:51.181617 9689 consensus_peers.cc:455] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 -> Peer 7bb94c92677448c39c50efc79e39459a (127.5.240.198:39947): Closing peer: 7bb94c92677448c39c50efc79e39459a I0703 01:39:51.181833 9689 consensus_queue.cc:204] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 0, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } } peers { permanent_uuid: "d33993c1f2c148d9ae2504a60ed04173" member_type: NON_VOTER last_known_addr { host: "127.5.240.200" port: 54337 } attrs { promote: true } } peers { permanent_uuid: "add911cdebcf44b8a64256c228929d30" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 59090 } attrs { promote: true } } peers { permanent_uuid: "3254ace7bc954d2ab9972a2723f45b0c" member_type: NON_VOTER last_known_addr { host: "127.5.240.197" port: 54985 } attrs { promote: true } } unsafe_config_change: true W0703 01:39:51.184690 9621 consensus_peers.cc:433] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 -> Peer d33993c1f2c148d9ae2504a60ed04173 (127.5.240.200:54337): Couldn't send request to peer d33993c1f2c148d9ae2504a60ed04173 for tablet a87a2a21d927456d930023d9775aec80. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a87a2a21d927456d930023d9775aec80. Retrying in the next heartbeat period. Already tried 1 times. W0703 01:39:51.185526 9620 consensus_peers.cc:433] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 -> Peer add911cdebcf44b8a64256c228929d30 (127.5.240.199:59090): Couldn't send request to peer add911cdebcf44b8a64256c228929d30 for tablet a87a2a21d927456d930023d9775aec80. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a87a2a21d927456d930023d9775aec80. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:39:51.187201 10177 raft_consensus.cc:1081] T a87a2a21d927456d930023d9775aec80 P 7bb94c92677448c39c50efc79e39459a [term 3 FOLLOWER]: Refusing update from remote peer 08ad9c3e352f492f9acb05219b72d0e0: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch) I0703 01:39:51.187891 10634 consensus_queue.cc:962] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I0703 01:39:51.189968 10644 raft_consensus.cc:2684] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 [term 3 LEADER]: Committing config change with OpId 3.6: config changed from index 5 to 6, NON_VOTER 3254ace7bc954d2ab9972a2723f45b0c (127.5.240.197) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } } peers { permanent_uuid: "d33993c1f2c148d9ae2504a60ed04173" member_type: NON_VOTER last_known_addr { host: "127.5.240.200" port: 54337 } attrs { promote: true } } peers { permanent_uuid: "add911cdebcf44b8a64256c228929d30" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 59090 } attrs { promote: true } } peers { permanent_uuid: "3254ace7bc954d2ab9972a2723f45b0c" member_type: NON_VOTER last_known_addr { host: "127.5.240.197" port: 54985 } attrs { promote: true } } unsafe_config_change: true } W0703 01:39:51.191628 9623 consensus_peers.cc:433] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 -> Peer 3254ace7bc954d2ab9972a2723f45b0c (127.5.240.197:54985): Couldn't send request to peer 3254ace7bc954d2ab9972a2723f45b0c for tablet a87a2a21d927456d930023d9775aec80. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a87a2a21d927456d930023d9775aec80. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:39:51.191872 10177 raft_consensus.cc:2684] T a87a2a21d927456d930023d9775aec80 P 7bb94c92677448c39c50efc79e39459a [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, NON_VOTER 3254ace7bc954d2ab9972a2723f45b0c (127.5.240.197) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } } peers { permanent_uuid: "d33993c1f2c148d9ae2504a60ed04173" member_type: NON_VOTER last_known_addr { host: "127.5.240.200" port: 54337 } attrs { promote: true } } peers { permanent_uuid: "add911cdebcf44b8a64256c228929d30" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 59090 } attrs { promote: true } } peers { permanent_uuid: "3254ace7bc954d2ab9972a2723f45b0c" member_type: NON_VOTER last_known_addr { host: "127.5.240.197" port: 54985 } attrs { promote: true } } unsafe_config_change: true } I0703 01:39:51.192497 10534 catalog_manager.cc:3352] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a87a2a21d927456d930023d9775aec80 with cas_config_opid_index 5: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0703 01:39:51.193533 10550 catalog_manager.cc:3798] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 reported cstate change: config changed from index 5 to 6, NON_VOTER 3254ace7bc954d2ab9972a2723f45b0c (127.5.240.197) added. New cstate: current_term: 3 leader_uuid: "08ad9c3e352f492f9acb05219b72d0e0" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d33993c1f2c148d9ae2504a60ed04173" member_type: NON_VOTER last_known_addr { host: "127.5.240.200" port: 54337 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "add911cdebcf44b8a64256c228929d30" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 59090 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3254ace7bc954d2ab9972a2723f45b0c" member_type: NON_VOTER last_known_addr { host: "127.5.240.197" port: 54985 } attrs { promote: true } health_report { overall_health: UNKNOWN } } unsafe_config_change: true } I0703 01:39:51.237663 10532 catalog_manager.cc:2903] Scheduling retry of a87a2a21d927456d930023d9775aec80 Delete Tablet RPC for TS=a382b1ef929b411b825a324ad47ecaa9 with a delay of 93 ms (attempt = 3) W0703 01:39:51.237718 10532 catalog_manager.cc:2922] Async tablet task a87a2a21d927456d930023d9775aec80 Delete Tablet RPC for TS=a382b1ef929b411b825a324ad47ecaa9 failed: Not found: failed to reset TS proxy: Could not find TS for UUID a382b1ef929b411b825a324ad47ecaa9 I0703 01:39:51.242522 10534 catalog_manager.cc:2903] Scheduling retry of a87a2a21d927456d930023d9775aec80 Delete Tablet RPC for TS=37b8cbcdace54a91a27cd7bbb1eefe17 with a delay of 85 ms (attempt = 3) W0703 01:39:51.242589 10534 catalog_manager.cc:2922] Async tablet task a87a2a21d927456d930023d9775aec80 Delete Tablet RPC for TS=37b8cbcdace54a91a27cd7bbb1eefe17 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 37b8cbcdace54a91a27cd7bbb1eefe17 I0703 01:39:51.260030 10535 catalog_manager.cc:2903] Scheduling retry of a87a2a21d927456d930023d9775aec80 Delete Tablet RPC for TS=937ef3ea804c4775a2a090e89e55f0fd with a delay of 84 ms (attempt = 3) W0703 01:39:51.260085 10535 catalog_manager.cc:2922] Async tablet task a87a2a21d927456d930023d9775aec80 Delete Tablet RPC for TS=937ef3ea804c4775a2a090e89e55f0fd failed: Not found: failed to reset TS proxy: Could not find TS for UUID 937ef3ea804c4775a2a090e89e55f0fd I0703 01:39:51.328212 10534 catalog_manager.cc:2903] Scheduling retry of a87a2a21d927456d930023d9775aec80 Delete Tablet RPC for TS=37b8cbcdace54a91a27cd7bbb1eefe17 with a delay of 162 ms (attempt = 4) W0703 01:39:51.328267 10534 catalog_manager.cc:2922] Async tablet task a87a2a21d927456d930023d9775aec80 Delete Tablet RPC for TS=37b8cbcdace54a91a27cd7bbb1eefe17 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 37b8cbcdace54a91a27cd7bbb1eefe17 I0703 01:39:51.331362 10532 catalog_manager.cc:2903] Scheduling retry of a87a2a21d927456d930023d9775aec80 Delete Tablet RPC for TS=a382b1ef929b411b825a324ad47ecaa9 with a delay of 150 ms (attempt = 4) W0703 01:39:51.331408 10532 catalog_manager.cc:2922] Async tablet task a87a2a21d927456d930023d9775aec80 Delete Tablet RPC for TS=a382b1ef929b411b825a324ad47ecaa9 failed: Not found: failed to reset TS proxy: Could not find TS for UUID a382b1ef929b411b825a324ad47ecaa9 I0703 01:39:51.344712 10535 catalog_manager.cc:2903] Scheduling retry of a87a2a21d927456d930023d9775aec80 Delete Tablet RPC for TS=937ef3ea804c4775a2a090e89e55f0fd with a delay of 155 ms (attempt = 4) W0703 01:39:51.344769 10535 catalog_manager.cc:2922] Async tablet task a87a2a21d927456d930023d9775aec80 Delete Tablet RPC for TS=937ef3ea804c4775a2a090e89e55f0fd failed: Not found: failed to reset TS proxy: Could not find TS for UUID 937ef3ea804c4775a2a090e89e55f0fd I0703 01:39:51.481811 10532 catalog_manager.cc:2903] Scheduling retry of a87a2a21d927456d930023d9775aec80 Delete Tablet RPC for TS=a382b1ef929b411b825a324ad47ecaa9 with a delay of 290 ms (attempt = 5) W0703 01:39:51.481869 10532 catalog_manager.cc:2922] Async tablet task a87a2a21d927456d930023d9775aec80 Delete Tablet RPC for TS=a382b1ef929b411b825a324ad47ecaa9 failed: Not found: failed to reset TS proxy: Could not find TS for UUID a382b1ef929b411b825a324ad47ecaa9 I0703 01:39:51.491211 10534 catalog_manager.cc:2903] Scheduling retry of a87a2a21d927456d930023d9775aec80 Delete Tablet RPC for TS=37b8cbcdace54a91a27cd7bbb1eefe17 with a delay of 296 ms (attempt = 5) W0703 01:39:51.491271 10534 catalog_manager.cc:2922] Async tablet task a87a2a21d927456d930023d9775aec80 Delete Tablet RPC for TS=37b8cbcdace54a91a27cd7bbb1eefe17 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 37b8cbcdace54a91a27cd7bbb1eefe17 I0703 01:39:51.500596 10535 catalog_manager.cc:2903] Scheduling retry of a87a2a21d927456d930023d9775aec80 Delete Tablet RPC for TS=937ef3ea804c4775a2a090e89e55f0fd with a delay of 289 ms (attempt = 5) W0703 01:39:51.500651 10535 catalog_manager.cc:2922] Async tablet task a87a2a21d927456d930023d9775aec80 Delete Tablet RPC for TS=937ef3ea804c4775a2a090e89e55f0fd failed: Not found: failed to reset TS proxy: Could not find TS for UUID 937ef3ea804c4775a2a090e89e55f0fd I0703 01:39:51.565610 10646 ts_tablet_manager.cc:669] T a87a2a21d927456d930023d9775aec80 P d33993c1f2c148d9ae2504a60ed04173: Initiating tablet copy from peer 08ad9c3e352f492f9acb05219b72d0e0 (127.5.240.194:42244) I0703 01:39:51.565873 10646 tablet_copy_client.cc:239] T a87a2a21d927456d930023d9775aec80 P d33993c1f2c148d9ae2504a60ed04173: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.194:42244 I0703 01:39:51.571650 9709 tablet_copy_service.cc:135] P 08ad9c3e352f492f9acb05219b72d0e0: Received BeginTabletCopySession request for tablet a87a2a21d927456d930023d9775aec80 from peer d33993c1f2c148d9ae2504a60ed04173 ({username='jenkins'} at 127.5.240.200:34570) I0703 01:39:51.571781 9709 tablet_copy_service.cc:156] P 08ad9c3e352f492f9acb05219b72d0e0: Beginning new tablet copy session on tablet a87a2a21d927456d930023d9775aec80 from peer d33993c1f2c148d9ae2504a60ed04173 at {username='jenkins'} at 127.5.240.200:34570: session id = d33993c1f2c148d9ae2504a60ed04173-a87a2a21d927456d930023d9775aec80 I0703 01:39:51.573770 9709 tablet_copy_source_session.cc:202] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:39:51.576838 10646 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet a87a2a21d927456d930023d9775aec80. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:39:51.586910 10646 ts_tablet_manager.cc:1173] T a87a2a21d927456d930023d9775aec80 P d33993c1f2c148d9ae2504a60ed04173: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:39:51.587198 10646 tablet_copy_client.cc:566] T a87a2a21d927456d930023d9775aec80 P d33993c1f2c148d9ae2504a60ed04173: tablet copy: Starting download of 0 data blocks... I0703 01:39:51.587388 10646 tablet_copy_client.cc:529] T a87a2a21d927456d930023d9775aec80 P d33993c1f2c148d9ae2504a60ed04173: tablet copy: Starting download of 1 WAL segments... I0703 01:39:51.589088 10646 tablet_copy_client.cc:405] T a87a2a21d927456d930023d9775aec80 P d33993c1f2c148d9ae2504a60ed04173: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:39:51.590816 10646 ts_tablet_manager.cc:1028] T a87a2a21d927456d930023d9775aec80 P d33993c1f2c148d9ae2504a60ed04173: Bootstrapping tablet I0703 01:39:51.591037 10646 tablet_bootstrap.cc:427] T a87a2a21d927456d930023d9775aec80 P d33993c1f2c148d9ae2504a60ed04173: Bootstrap starting. I0703 01:39:51.595516 10646 log.cc:525] T a87a2a21d927456d930023d9775aec80 P d33993c1f2c148d9ae2504a60ed04173: Log is configured to *not* fsync() on all Append() calls I0703 01:39:51.600132 10646 tablet_bootstrap.cc:427] T a87a2a21d927456d930023d9775aec80 P d33993c1f2c148d9ae2504a60ed04173: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:39:51.600472 10646 tablet_bootstrap.cc:427] T a87a2a21d927456d930023d9775aec80 P d33993c1f2c148d9ae2504a60ed04173: Bootstrap complete. I0703 01:39:51.600713 10646 ts_tablet_manager.cc:1045] T a87a2a21d927456d930023d9775aec80 P d33993c1f2c148d9ae2504a60ed04173: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.003s I0703 01:39:51.605576 10646 raft_consensus.cc:325] T a87a2a21d927456d930023d9775aec80 P d33993c1f2c148d9ae2504a60ed04173 [term 3 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } } peers { permanent_uuid: "d33993c1f2c148d9ae2504a60ed04173" member_type: NON_VOTER last_known_addr { host: "127.5.240.200" port: 54337 } attrs { promote: true } } peers { permanent_uuid: "add911cdebcf44b8a64256c228929d30" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 59090 } attrs { promote: true } } peers { permanent_uuid: "3254ace7bc954d2ab9972a2723f45b0c" member_type: NON_VOTER last_known_addr { host: "127.5.240.197" port: 54985 } attrs { promote: true } } unsafe_config_change: true I0703 01:39:51.605908 10646 raft_consensus.cc:628] T a87a2a21d927456d930023d9775aec80 P d33993c1f2c148d9ae2504a60ed04173 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: d33993c1f2c148d9ae2504a60ed04173, State: Initialized, Role: LEARNER I0703 01:39:51.606161 10646 consensus_queue.cc:227] T a87a2a21d927456d930023d9775aec80 P d33993c1f2c148d9ae2504a60ed04173 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } } peers { permanent_uuid: "d33993c1f2c148d9ae2504a60ed04173" member_type: NON_VOTER last_known_addr { host: "127.5.240.200" port: 54337 } attrs { promote: true } } peers { permanent_uuid: "add911cdebcf44b8a64256c228929d30" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 59090 } attrs { promote: true } } peers { permanent_uuid: "3254ace7bc954d2ab9972a2723f45b0c" member_type: NON_VOTER last_known_addr { host: "127.5.240.197" port: 54985 } attrs { promote: true } } unsafe_config_change: true I0703 01:39:51.608009 10646 ts_tablet_manager.cc:1073] T a87a2a21d927456d930023d9775aec80 P d33993c1f2c148d9ae2504a60ed04173: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s I0703 01:39:51.608767 9709 tablet_copy_service.cc:325] P 08ad9c3e352f492f9acb05219b72d0e0: Request end of tablet copy session d33993c1f2c148d9ae2504a60ed04173-a87a2a21d927456d930023d9775aec80 received from {username='jenkins'} at 127.5.240.200:34570 I0703 01:39:51.608860 9709 tablet_copy_service.cc:413] P 08ad9c3e352f492f9acb05219b72d0e0: ending tablet copy session d33993c1f2c148d9ae2504a60ed04173-a87a2a21d927456d930023d9775aec80 on tablet a87a2a21d927456d930023d9775aec80 with peer d33993c1f2c148d9ae2504a60ed04173 W0703 01:39:51.620539 10465 tablet.cc:1678] T a87a2a21d927456d930023d9775aec80 P d33993c1f2c148d9ae2504a60ed04173: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:39:51.693491 10650 ts_tablet_manager.cc:669] T a87a2a21d927456d930023d9775aec80 P 3254ace7bc954d2ab9972a2723f45b0c: Initiating tablet copy from peer 08ad9c3e352f492f9acb05219b72d0e0 (127.5.240.194:42244) I0703 01:39:51.693756 10650 tablet_copy_client.cc:239] T a87a2a21d927456d930023d9775aec80 P 3254ace7bc954d2ab9972a2723f45b0c: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.194:42244 I0703 01:39:51.698778 9709 tablet_copy_service.cc:135] P 08ad9c3e352f492f9acb05219b72d0e0: Received BeginTabletCopySession request for tablet a87a2a21d927456d930023d9775aec80 from peer 3254ace7bc954d2ab9972a2723f45b0c ({username='jenkins'} at 127.5.240.197:47275) I0703 01:39:51.698868 9709 tablet_copy_service.cc:156] P 08ad9c3e352f492f9acb05219b72d0e0: Beginning new tablet copy session on tablet a87a2a21d927456d930023d9775aec80 from peer 3254ace7bc954d2ab9972a2723f45b0c at {username='jenkins'} at 127.5.240.197:47275: session id = 3254ace7bc954d2ab9972a2723f45b0c-a87a2a21d927456d930023d9775aec80 I0703 01:39:51.700404 9709 tablet_copy_source_session.cc:202] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:39:51.703363 10650 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet a87a2a21d927456d930023d9775aec80. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:39:51.713770 10650 ts_tablet_manager.cc:1173] T a87a2a21d927456d930023d9775aec80 P 3254ace7bc954d2ab9972a2723f45b0c: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:39:51.714049 10650 tablet_copy_client.cc:566] T a87a2a21d927456d930023d9775aec80 P 3254ace7bc954d2ab9972a2723f45b0c: tablet copy: Starting download of 0 data blocks... I0703 01:39:51.714213 10650 tablet_copy_client.cc:529] T a87a2a21d927456d930023d9775aec80 P 3254ace7bc954d2ab9972a2723f45b0c: tablet copy: Starting download of 1 WAL segments... I0703 01:39:51.716195 10650 tablet_copy_client.cc:405] T a87a2a21d927456d930023d9775aec80 P 3254ace7bc954d2ab9972a2723f45b0c: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:39:51.717942 10650 ts_tablet_manager.cc:1028] T a87a2a21d927456d930023d9775aec80 P 3254ace7bc954d2ab9972a2723f45b0c: Bootstrapping tablet I0703 01:39:51.718171 10650 tablet_bootstrap.cc:427] T a87a2a21d927456d930023d9775aec80 P 3254ace7bc954d2ab9972a2723f45b0c: Bootstrap starting. I0703 01:39:51.722256 10650 log.cc:525] T a87a2a21d927456d930023d9775aec80 P 3254ace7bc954d2ab9972a2723f45b0c: Log is configured to *not* fsync() on all Append() calls I0703 01:39:51.727047 10650 tablet_bootstrap.cc:427] T a87a2a21d927456d930023d9775aec80 P 3254ace7bc954d2ab9972a2723f45b0c: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:39:51.727358 10650 tablet_bootstrap.cc:427] T a87a2a21d927456d930023d9775aec80 P 3254ace7bc954d2ab9972a2723f45b0c: Bootstrap complete. I0703 01:39:51.727596 10650 ts_tablet_manager.cc:1045] T a87a2a21d927456d930023d9775aec80 P 3254ace7bc954d2ab9972a2723f45b0c: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.004s I0703 01:39:51.732426 10650 raft_consensus.cc:325] T a87a2a21d927456d930023d9775aec80 P 3254ace7bc954d2ab9972a2723f45b0c [term 3 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } } peers { permanent_uuid: "d33993c1f2c148d9ae2504a60ed04173" member_type: NON_VOTER last_known_addr { host: "127.5.240.200" port: 54337 } attrs { promote: true } } peers { permanent_uuid: "add911cdebcf44b8a64256c228929d30" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 59090 } attrs { promote: true } } peers { permanent_uuid: "3254ace7bc954d2ab9972a2723f45b0c" member_type: NON_VOTER last_known_addr { host: "127.5.240.197" port: 54985 } attrs { promote: true } } unsafe_config_change: true I0703 01:39:51.732759 10650 raft_consensus.cc:628] T a87a2a21d927456d930023d9775aec80 P 3254ace7bc954d2ab9972a2723f45b0c [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 3254ace7bc954d2ab9972a2723f45b0c, State: Initialized, Role: LEARNER I0703 01:39:51.733028 10650 consensus_queue.cc:227] T a87a2a21d927456d930023d9775aec80 P 3254ace7bc954d2ab9972a2723f45b0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } } peers { permanent_uuid: "d33993c1f2c148d9ae2504a60ed04173" member_type: NON_VOTER last_known_addr { host: "127.5.240.200" port: 54337 } attrs { promote: true } } peers { permanent_uuid: "add911cdebcf44b8a64256c228929d30" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 59090 } attrs { promote: true } } peers { permanent_uuid: "3254ace7bc954d2ab9972a2723f45b0c" member_type: NON_VOTER last_known_addr { host: "127.5.240.197" port: 54985 } attrs { promote: true } } unsafe_config_change: true I0703 01:39:51.735159 10650 ts_tablet_manager.cc:1073] T a87a2a21d927456d930023d9775aec80 P 3254ace7bc954d2ab9972a2723f45b0c: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s I0703 01:39:51.735833 9709 tablet_copy_service.cc:325] P 08ad9c3e352f492f9acb05219b72d0e0: Request end of tablet copy session 3254ace7bc954d2ab9972a2723f45b0c-a87a2a21d927456d930023d9775aec80 received from {username='jenkins'} at 127.5.240.197:47275 I0703 01:39:51.735915 9709 tablet_copy_service.cc:413] P 08ad9c3e352f492f9acb05219b72d0e0: ending tablet copy session 3254ace7bc954d2ab9972a2723f45b0c-a87a2a21d927456d930023d9775aec80 on tablet a87a2a21d927456d930023d9775aec80 with peer 3254ace7bc954d2ab9972a2723f45b0c I0703 01:39:51.747073 10654 ts_tablet_manager.cc:669] T a87a2a21d927456d930023d9775aec80 P add911cdebcf44b8a64256c228929d30: Initiating tablet copy from peer 08ad9c3e352f492f9acb05219b72d0e0 (127.5.240.194:42244) I0703 01:39:51.747318 10654 tablet_copy_client.cc:239] T a87a2a21d927456d930023d9775aec80 P add911cdebcf44b8a64256c228929d30: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.194:42244 I0703 01:39:51.751585 9709 tablet_copy_service.cc:135] P 08ad9c3e352f492f9acb05219b72d0e0: Received BeginTabletCopySession request for tablet a87a2a21d927456d930023d9775aec80 from peer add911cdebcf44b8a64256c228929d30 ({username='jenkins'} at 127.5.240.199:58987) I0703 01:39:51.751678 9709 tablet_copy_service.cc:156] P 08ad9c3e352f492f9acb05219b72d0e0: Beginning new tablet copy session on tablet a87a2a21d927456d930023d9775aec80 from peer add911cdebcf44b8a64256c228929d30 at {username='jenkins'} at 127.5.240.199:58987: session id = add911cdebcf44b8a64256c228929d30-a87a2a21d927456d930023d9775aec80 I0703 01:39:51.753247 9709 tablet_copy_source_session.cc:202] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:39:51.756075 10654 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet a87a2a21d927456d930023d9775aec80. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:39:51.765993 10654 ts_tablet_manager.cc:1173] T a87a2a21d927456d930023d9775aec80 P add911cdebcf44b8a64256c228929d30: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:39:51.766248 10654 tablet_copy_client.cc:566] T a87a2a21d927456d930023d9775aec80 P add911cdebcf44b8a64256c228929d30: tablet copy: Starting download of 0 data blocks... I0703 01:39:51.766364 10654 tablet_copy_client.cc:529] T a87a2a21d927456d930023d9775aec80 P add911cdebcf44b8a64256c228929d30: tablet copy: Starting download of 1 WAL segments... I0703 01:39:51.768146 10654 tablet_copy_client.cc:405] T a87a2a21d927456d930023d9775aec80 P add911cdebcf44b8a64256c228929d30: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:39:51.769862 10654 ts_tablet_manager.cc:1028] T a87a2a21d927456d930023d9775aec80 P add911cdebcf44b8a64256c228929d30: Bootstrapping tablet I0703 01:39:51.770087 10654 tablet_bootstrap.cc:427] T a87a2a21d927456d930023d9775aec80 P add911cdebcf44b8a64256c228929d30: Bootstrap starting. I0703 01:39:51.772164 10532 catalog_manager.cc:2903] Scheduling retry of a87a2a21d927456d930023d9775aec80 Delete Tablet RPC for TS=a382b1ef929b411b825a324ad47ecaa9 with a delay of 534 ms (attempt = 6) W0703 01:39:51.772243 10532 catalog_manager.cc:2922] Async tablet task a87a2a21d927456d930023d9775aec80 Delete Tablet RPC for TS=a382b1ef929b411b825a324ad47ecaa9 failed: Not found: failed to reset TS proxy: Could not find TS for UUID a382b1ef929b411b825a324ad47ecaa9 I0703 01:39:51.774204 10654 log.cc:525] T a87a2a21d927456d930023d9775aec80 P add911cdebcf44b8a64256c228929d30: Log is configured to *not* fsync() on all Append() calls I0703 01:39:51.778453 10654 tablet_bootstrap.cc:427] T a87a2a21d927456d930023d9775aec80 P add911cdebcf44b8a64256c228929d30: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:39:51.779175 10654 tablet_bootstrap.cc:427] T a87a2a21d927456d930023d9775aec80 P add911cdebcf44b8a64256c228929d30: Bootstrap complete. I0703 01:39:51.779392 10654 ts_tablet_manager.cc:1045] T a87a2a21d927456d930023d9775aec80 P add911cdebcf44b8a64256c228929d30: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s I0703 01:39:51.784624 10654 raft_consensus.cc:325] T a87a2a21d927456d930023d9775aec80 P add911cdebcf44b8a64256c228929d30 [term 3 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } } peers { permanent_uuid: "d33993c1f2c148d9ae2504a60ed04173" member_type: NON_VOTER last_known_addr { host: "127.5.240.200" port: 54337 } attrs { promote: true } } peers { permanent_uuid: "add911cdebcf44b8a64256c228929d30" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 59090 } attrs { promote: true } } peers { permanent_uuid: "3254ace7bc954d2ab9972a2723f45b0c" member_type: NON_VOTER last_known_addr { host: "127.5.240.197" port: 54985 } attrs { promote: true } } unsafe_config_change: true I0703 01:39:51.784948 10654 raft_consensus.cc:628] T a87a2a21d927456d930023d9775aec80 P add911cdebcf44b8a64256c228929d30 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: add911cdebcf44b8a64256c228929d30, State: Initialized, Role: LEARNER I0703 01:39:51.785224 10654 consensus_queue.cc:227] T a87a2a21d927456d930023d9775aec80 P add911cdebcf44b8a64256c228929d30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } } peers { permanent_uuid: "d33993c1f2c148d9ae2504a60ed04173" member_type: NON_VOTER last_known_addr { host: "127.5.240.200" port: 54337 } attrs { promote: true } } peers { permanent_uuid: "add911cdebcf44b8a64256c228929d30" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 59090 } attrs { promote: true } } peers { permanent_uuid: "3254ace7bc954d2ab9972a2723f45b0c" member_type: NON_VOTER last_known_addr { host: "127.5.240.197" port: 54985 } attrs { promote: true } } unsafe_config_change: true I0703 01:39:51.786460 10654 ts_tablet_manager.cc:1073] T a87a2a21d927456d930023d9775aec80 P add911cdebcf44b8a64256c228929d30: Time spent starting tablet: real 0.007s user 0.005s sys 0.003s I0703 01:39:51.787263 9709 tablet_copy_service.cc:325] P 08ad9c3e352f492f9acb05219b72d0e0: Request end of tablet copy session add911cdebcf44b8a64256c228929d30-a87a2a21d927456d930023d9775aec80 received from {username='jenkins'} at 127.5.240.199:58987 I0703 01:39:51.787343 9709 tablet_copy_service.cc:413] P 08ad9c3e352f492f9acb05219b72d0e0: ending tablet copy session add911cdebcf44b8a64256c228929d30-a87a2a21d927456d930023d9775aec80 on tablet a87a2a21d927456d930023d9775aec80 with peer add911cdebcf44b8a64256c228929d30 I0703 01:39:51.787518 10534 catalog_manager.cc:2903] Scheduling retry of a87a2a21d927456d930023d9775aec80 Delete Tablet RPC for TS=37b8cbcdace54a91a27cd7bbb1eefe17 with a delay of 560 ms (attempt = 6) W0703 01:39:51.787668 10534 catalog_manager.cc:2922] Async tablet task a87a2a21d927456d930023d9775aec80 Delete Tablet RPC for TS=37b8cbcdace54a91a27cd7bbb1eefe17 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 37b8cbcdace54a91a27cd7bbb1eefe17 I0703 01:39:51.790035 10535 catalog_manager.cc:2903] Scheduling retry of a87a2a21d927456d930023d9775aec80 Delete Tablet RPC for TS=937ef3ea804c4775a2a090e89e55f0fd with a delay of 542 ms (attempt = 6) W0703 01:39:51.790086 10535 catalog_manager.cc:2922] Async tablet task a87a2a21d927456d930023d9775aec80 Delete Tablet RPC for TS=937ef3ea804c4775a2a090e89e55f0fd failed: Not found: failed to reset TS proxy: Could not find TS for UUID 937ef3ea804c4775a2a090e89e55f0fd W0703 01:39:51.812110 10343 tablet.cc:1678] T a87a2a21d927456d930023d9775aec80 P add911cdebcf44b8a64256c228929d30: Can't schedule compaction. Clean time has not been advanced past its initial value. W0703 01:39:51.857159 10099 tablet.cc:1678] T a87a2a21d927456d930023d9775aec80 P 3254ace7bc954d2ab9972a2723f45b0c: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:39:52.077745 10054 raft_consensus.cc:1026] T a87a2a21d927456d930023d9775aec80 P 3254ace7bc954d2ab9972a2723f45b0c [term 3 LEARNER]: Deduplicated request from leader. Original: 3.5->[3.6-3.6] Dedup: 3.6->[] I0703 01:39:52.138635 10421 raft_consensus.cc:1026] T a87a2a21d927456d930023d9775aec80 P d33993c1f2c148d9ae2504a60ed04173 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.5->[3.6-3.6] Dedup: 3.6->[] I0703 01:39:52.303234 10299 raft_consensus.cc:1026] T a87a2a21d927456d930023d9775aec80 P add911cdebcf44b8a64256c228929d30 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.5->[3.6-3.6] Dedup: 3.6->[] I0703 01:39:52.306571 10532 catalog_manager.cc:2903] Scheduling retry of a87a2a21d927456d930023d9775aec80 Delete Tablet RPC for TS=a382b1ef929b411b825a324ad47ecaa9 with a delay of 1037 ms (attempt = 7) W0703 01:39:52.306627 10532 catalog_manager.cc:2922] Async tablet task a87a2a21d927456d930023d9775aec80 Delete Tablet RPC for TS=a382b1ef929b411b825a324ad47ecaa9 failed: Not found: failed to reset TS proxy: Could not find TS for UUID a382b1ef929b411b825a324ad47ecaa9 I0703 01:39:52.332813 10535 catalog_manager.cc:2903] Scheduling retry of a87a2a21d927456d930023d9775aec80 Delete Tablet RPC for TS=937ef3ea804c4775a2a090e89e55f0fd with a delay of 1071 ms (attempt = 7) W0703 01:39:52.332871 10535 catalog_manager.cc:2922] Async tablet task a87a2a21d927456d930023d9775aec80 Delete Tablet RPC for TS=937ef3ea804c4775a2a090e89e55f0fd failed: Not found: failed to reset TS proxy: Could not find TS for UUID 937ef3ea804c4775a2a090e89e55f0fd I0703 01:39:52.348166 10534 catalog_manager.cc:2903] Scheduling retry of a87a2a21d927456d930023d9775aec80 Delete Tablet RPC for TS=37b8cbcdace54a91a27cd7bbb1eefe17 with a delay of 1024 ms (attempt = 7) W0703 01:39:52.348222 10534 catalog_manager.cc:2922] Async tablet task a87a2a21d927456d930023d9775aec80 Delete Tablet RPC for TS=37b8cbcdace54a91a27cd7bbb1eefe17 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 37b8cbcdace54a91a27cd7bbb1eefe17 I0703 01:39:52.482883 10658 raft_consensus.cc:890] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0: attempting to promote NON_VOTER 3254ace7bc954d2ab9972a2723f45b0c to VOTER I0703 01:39:52.483204 10658 consensus_peers.cc:455] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 -> Peer add911cdebcf44b8a64256c228929d30 (127.5.240.199:59090): Closing peer: add911cdebcf44b8a64256c228929d30 I0703 01:39:52.483273 10658 consensus_peers.cc:455] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 -> Peer 3254ace7bc954d2ab9972a2723f45b0c (127.5.240.197:54985): Closing peer: 3254ace7bc954d2ab9972a2723f45b0c I0703 01:39:52.483325 10658 consensus_peers.cc:455] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 -> Peer d33993c1f2c148d9ae2504a60ed04173 (127.5.240.200:54337): Closing peer: d33993c1f2c148d9ae2504a60ed04173 I0703 01:39:52.483371 10658 consensus_peers.cc:455] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 -> Peer 7bb94c92677448c39c50efc79e39459a (127.5.240.198:39947): Closing peer: 7bb94c92677448c39c50efc79e39459a I0703 01:39:52.483639 10658 consensus_queue.cc:204] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 0, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } } peers { permanent_uuid: "d33993c1f2c148d9ae2504a60ed04173" member_type: NON_VOTER last_known_addr { host: "127.5.240.200" port: 54337 } attrs { promote: true } } peers { permanent_uuid: "add911cdebcf44b8a64256c228929d30" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 59090 } attrs { promote: true } } peers { permanent_uuid: "3254ace7bc954d2ab9972a2723f45b0c" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54985 } attrs { promote: false } } unsafe_config_change: true I0703 01:39:52.486085 10054 raft_consensus.cc:1081] T a87a2a21d927456d930023d9775aec80 P 3254ace7bc954d2ab9972a2723f45b0c [term 3 LEARNER]: Refusing update from remote peer 08ad9c3e352f492f9acb05219b72d0e0: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch) I0703 01:39:52.486193 10299 raft_consensus.cc:1081] T a87a2a21d927456d930023d9775aec80 P add911cdebcf44b8a64256c228929d30 [term 3 LEARNER]: Refusing update from remote peer 08ad9c3e352f492f9acb05219b72d0e0: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch) I0703 01:39:52.487084 10177 raft_consensus.cc:1081] T a87a2a21d927456d930023d9775aec80 P 7bb94c92677448c39c50efc79e39459a [term 3 FOLLOWER]: Refusing update from remote peer 08ad9c3e352f492f9acb05219b72d0e0: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch) I0703 01:39:52.487279 10642 consensus_queue.cc:962] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3254ace7bc954d2ab9972a2723f45b0c" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54985 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I0703 01:39:52.487819 10421 raft_consensus.cc:1081] T a87a2a21d927456d930023d9775aec80 P d33993c1f2c148d9ae2504a60ed04173 [term 3 LEARNER]: Refusing update from remote peer 08ad9c3e352f492f9acb05219b72d0e0: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch) I0703 01:39:52.487702 10659 consensus_queue.cc:962] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "add911cdebcf44b8a64256c228929d30" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 59090 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I0703 01:39:52.488226 10659 consensus_queue.cc:962] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I0703 01:39:52.488943 10642 consensus_queue.cc:962] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d33993c1f2c148d9ae2504a60ed04173" member_type: NON_VOTER last_known_addr { host: "127.5.240.200" port: 54337 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I0703 01:39:52.491233 10658 raft_consensus.cc:2684] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 [term 3 LEADER]: Committing config change with OpId 3.7: config changed from index 6 to 7, 3254ace7bc954d2ab9972a2723f45b0c (127.5.240.197) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } } peers { permanent_uuid: "d33993c1f2c148d9ae2504a60ed04173" member_type: NON_VOTER last_known_addr { host: "127.5.240.200" port: 54337 } attrs { promote: true } } peers { permanent_uuid: "add911cdebcf44b8a64256c228929d30" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 59090 } attrs { promote: true } } peers { permanent_uuid: "3254ace7bc954d2ab9972a2723f45b0c" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54985 } attrs { promote: false } } unsafe_config_change: true } I0703 01:39:52.491964 10054 raft_consensus.cc:2684] T a87a2a21d927456d930023d9775aec80 P 3254ace7bc954d2ab9972a2723f45b0c [term 3 FOLLOWER]: Committing config change with OpId 3.7: config changed from index 6 to 7, 3254ace7bc954d2ab9972a2723f45b0c (127.5.240.197) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } } peers { permanent_uuid: "d33993c1f2c148d9ae2504a60ed04173" member_type: NON_VOTER last_known_addr { host: "127.5.240.200" port: 54337 } attrs { promote: true } } peers { permanent_uuid: "add911cdebcf44b8a64256c228929d30" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 59090 } attrs { promote: true } } peers { permanent_uuid: "3254ace7bc954d2ab9972a2723f45b0c" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54985 } attrs { promote: false } } unsafe_config_change: true } I0703 01:39:52.493350 10177 raft_consensus.cc:2684] T a87a2a21d927456d930023d9775aec80 P 7bb94c92677448c39c50efc79e39459a [term 3 FOLLOWER]: Committing config change with OpId 3.7: config changed from index 6 to 7, 3254ace7bc954d2ab9972a2723f45b0c (127.5.240.197) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } } peers { permanent_uuid: "d33993c1f2c148d9ae2504a60ed04173" member_type: NON_VOTER last_known_addr { host: "127.5.240.200" port: 54337 } attrs { promote: true } } peers { permanent_uuid: "add911cdebcf44b8a64256c228929d30" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 59090 } attrs { promote: true } } peers { permanent_uuid: "3254ace7bc954d2ab9972a2723f45b0c" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54985 } attrs { promote: false } } unsafe_config_change: true } I0703 01:39:52.494940 10659 raft_consensus.cc:890] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0: attempting to promote NON_VOTER d33993c1f2c148d9ae2504a60ed04173 to VOTER I0703 01:39:52.495318 10659 consensus_peers.cc:455] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 -> Peer add911cdebcf44b8a64256c228929d30 (127.5.240.199:59090): Closing peer: add911cdebcf44b8a64256c228929d30 I0703 01:39:52.495383 10659 consensus_peers.cc:455] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 -> Peer 3254ace7bc954d2ab9972a2723f45b0c (127.5.240.197:54985): Closing peer: 3254ace7bc954d2ab9972a2723f45b0c I0703 01:39:52.495434 10659 consensus_peers.cc:455] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 -> Peer d33993c1f2c148d9ae2504a60ed04173 (127.5.240.200:54337): Closing peer: d33993c1f2c148d9ae2504a60ed04173 I0703 01:39:52.495488 10659 consensus_peers.cc:455] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 -> Peer 7bb94c92677448c39c50efc79e39459a (127.5.240.198:39947): Closing peer: 7bb94c92677448c39c50efc79e39459a I0703 01:39:52.495728 10659 consensus_queue.cc:204] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 3.7, Last appended by leader: 0, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } } peers { permanent_uuid: "d33993c1f2c148d9ae2504a60ed04173" member_type: VOTER last_known_addr { host: "127.5.240.200" port: 54337 } attrs { promote: false } } peers { permanent_uuid: "add911cdebcf44b8a64256c228929d30" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 59090 } attrs { promote: true } } peers { permanent_uuid: "3254ace7bc954d2ab9972a2723f45b0c" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54985 } attrs { promote: false } } unsafe_config_change: true I0703 01:39:52.496043 10551 catalog_manager.cc:3798] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 reported cstate change: config changed from index 6 to 7, 3254ace7bc954d2ab9972a2723f45b0c (127.5.240.197) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "08ad9c3e352f492f9acb05219b72d0e0" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d33993c1f2c148d9ae2504a60ed04173" member_type: NON_VOTER last_known_addr { host: "127.5.240.200" port: 54337 } attrs { promote: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "add911cdebcf44b8a64256c228929d30" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 59090 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3254ace7bc954d2ab9972a2723f45b0c" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54985 } attrs { promote: false } health_report { overall_health: HEALTHY } } unsafe_config_change: true } I0703 01:39:52.496794 10421 raft_consensus.cc:2684] T a87a2a21d927456d930023d9775aec80 P d33993c1f2c148d9ae2504a60ed04173 [term 3 LEARNER]: Committing config change with OpId 3.7: config changed from index 6 to 7, 3254ace7bc954d2ab9972a2723f45b0c (127.5.240.197) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } } peers { permanent_uuid: "d33993c1f2c148d9ae2504a60ed04173" member_type: NON_VOTER last_known_addr { host: "127.5.240.200" port: 54337 } attrs { promote: true } } peers { permanent_uuid: "add911cdebcf44b8a64256c228929d30" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 59090 } attrs { promote: true } } peers { permanent_uuid: "3254ace7bc954d2ab9972a2723f45b0c" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54985 } attrs { promote: false } } unsafe_config_change: true } I0703 01:39:52.499250 10299 raft_consensus.cc:1081] T a87a2a21d927456d930023d9775aec80 P add911cdebcf44b8a64256c228929d30 [term 3 LEARNER]: Refusing update from remote peer 08ad9c3e352f492f9acb05219b72d0e0: Log matching property violated. Preceding OpId in replica: term: 3 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch) I0703 01:39:52.499392 10177 raft_consensus.cc:1081] T a87a2a21d927456d930023d9775aec80 P 7bb94c92677448c39c50efc79e39459a [term 3 FOLLOWER]: Refusing update from remote peer 08ad9c3e352f492f9acb05219b72d0e0: Log matching property violated. Preceding OpId in replica: term: 3 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch) I0703 01:39:52.499420 10054 raft_consensus.cc:1081] T a87a2a21d927456d930023d9775aec80 P 3254ace7bc954d2ab9972a2723f45b0c [term 3 FOLLOWER]: Refusing update from remote peer 08ad9c3e352f492f9acb05219b72d0e0: Log matching property violated. Preceding OpId in replica: term: 3 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch) I0703 01:39:52.500274 10663 consensus_queue.cc:962] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I0703 01:39:52.500763 10642 consensus_queue.cc:962] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "add911cdebcf44b8a64256c228929d30" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 59090 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s I0703 01:39:52.500998 10658 consensus_queue.cc:962] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3254ace7bc954d2ab9972a2723f45b0c" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54985 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I0703 01:39:52.501854 10420 raft_consensus.cc:1081] T a87a2a21d927456d930023d9775aec80 P d33993c1f2c148d9ae2504a60ed04173 [term 3 LEARNER]: Refusing update from remote peer 08ad9c3e352f492f9acb05219b72d0e0: Log matching property violated. Preceding OpId in replica: term: 3 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch) I0703 01:39:52.503038 10658 consensus_queue.cc:962] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d33993c1f2c148d9ae2504a60ed04173" member_type: VOTER last_known_addr { host: "127.5.240.200" port: 54337 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I0703 01:39:52.502702 10299 raft_consensus.cc:2684] T a87a2a21d927456d930023d9775aec80 P add911cdebcf44b8a64256c228929d30 [term 3 LEARNER]: Committing config change with OpId 3.7: config changed from index 6 to 7, 3254ace7bc954d2ab9972a2723f45b0c (127.5.240.197) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } } peers { permanent_uuid: "d33993c1f2c148d9ae2504a60ed04173" member_type: NON_VOTER last_known_addr { host: "127.5.240.200" port: 54337 } attrs { promote: true } } peers { permanent_uuid: "add911cdebcf44b8a64256c228929d30" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 59090 } attrs { promote: true } } peers { permanent_uuid: "3254ace7bc954d2ab9972a2723f45b0c" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54985 } attrs { promote: false } } unsafe_config_change: true } I0703 01:39:52.504869 10658 raft_consensus.cc:2684] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 [term 3 LEADER]: Committing config change with OpId 3.8: config changed from index 7 to 8, d33993c1f2c148d9ae2504a60ed04173 (127.5.240.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } } peers { permanent_uuid: "d33993c1f2c148d9ae2504a60ed04173" member_type: VOTER last_known_addr { host: "127.5.240.200" port: 54337 } attrs { promote: false } } peers { permanent_uuid: "add911cdebcf44b8a64256c228929d30" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 59090 } attrs { promote: true } } peers { permanent_uuid: "3254ace7bc954d2ab9972a2723f45b0c" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54985 } attrs { promote: false } } unsafe_config_change: true } I0703 01:39:52.505458 10054 raft_consensus.cc:2684] T a87a2a21d927456d930023d9775aec80 P 3254ace7bc954d2ab9972a2723f45b0c [term 3 FOLLOWER]: Committing config change with OpId 3.8: config changed from index 7 to 8, d33993c1f2c148d9ae2504a60ed04173 (127.5.240.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } } peers { permanent_uuid: "d33993c1f2c148d9ae2504a60ed04173" member_type: VOTER last_known_addr { host: "127.5.240.200" port: 54337 } attrs { promote: false } } peers { permanent_uuid: "add911cdebcf44b8a64256c228929d30" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 59090 } attrs { promote: true } } peers { permanent_uuid: "3254ace7bc954d2ab9972a2723f45b0c" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54985 } attrs { promote: false } } unsafe_config_change: true } I0703 01:39:52.508112 10663 raft_consensus.cc:890] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0: attempting to promote NON_VOTER add911cdebcf44b8a64256c228929d30 to VOTER I0703 01:39:52.508517 10663 consensus_peers.cc:455] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 -> Peer add911cdebcf44b8a64256c228929d30 (127.5.240.199:59090): Closing peer: add911cdebcf44b8a64256c228929d30 I0703 01:39:52.508581 10663 consensus_peers.cc:455] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 -> Peer 3254ace7bc954d2ab9972a2723f45b0c (127.5.240.197:54985): Closing peer: 3254ace7bc954d2ab9972a2723f45b0c I0703 01:39:52.508632 10663 consensus_peers.cc:455] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 -> Peer d33993c1f2c148d9ae2504a60ed04173 (127.5.240.200:54337): Closing peer: d33993c1f2c148d9ae2504a60ed04173 I0703 01:39:52.508673 10663 consensus_peers.cc:455] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 -> Peer 7bb94c92677448c39c50efc79e39459a (127.5.240.198:39947): Closing peer: 7bb94c92677448c39c50efc79e39459a I0703 01:39:52.508543 10299 raft_consensus.cc:2684] T a87a2a21d927456d930023d9775aec80 P add911cdebcf44b8a64256c228929d30 [term 3 LEARNER]: Committing config change with OpId 3.8: config changed from index 7 to 8, d33993c1f2c148d9ae2504a60ed04173 (127.5.240.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } } peers { permanent_uuid: "d33993c1f2c148d9ae2504a60ed04173" member_type: VOTER last_known_addr { host: "127.5.240.200" port: 54337 } attrs { promote: false } } peers { permanent_uuid: "add911cdebcf44b8a64256c228929d30" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 59090 } attrs { promote: true } } peers { permanent_uuid: "3254ace7bc954d2ab9972a2723f45b0c" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54985 } attrs { promote: false } } unsafe_config_change: true } I0703 01:39:52.508848 10663 consensus_queue.cc:204] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 3.8, Last appended by leader: 0, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } } peers { permanent_uuid: "d33993c1f2c148d9ae2504a60ed04173" member_type: VOTER last_known_addr { host: "127.5.240.200" port: 54337 } attrs { promote: false } } peers { permanent_uuid: "add911cdebcf44b8a64256c228929d30" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 59090 } attrs { promote: false } } peers { permanent_uuid: "3254ace7bc954d2ab9972a2723f45b0c" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54985 } attrs { promote: false } } unsafe_config_change: true I0703 01:39:52.508780 10177 raft_consensus.cc:2684] T a87a2a21d927456d930023d9775aec80 P 7bb94c92677448c39c50efc79e39459a [term 3 FOLLOWER]: Committing config change with OpId 3.8: config changed from index 7 to 8, d33993c1f2c148d9ae2504a60ed04173 (127.5.240.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } } peers { permanent_uuid: "d33993c1f2c148d9ae2504a60ed04173" member_type: VOTER last_known_addr { host: "127.5.240.200" port: 54337 } attrs { promote: false } } peers { permanent_uuid: "add911cdebcf44b8a64256c228929d30" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 59090 } attrs { promote: true } } peers { permanent_uuid: "3254ace7bc954d2ab9972a2723f45b0c" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54985 } attrs { promote: false } } unsafe_config_change: true } I0703 01:39:52.511270 10055 raft_consensus.cc:1081] T a87a2a21d927456d930023d9775aec80 P 3254ace7bc954d2ab9972a2723f45b0c [term 3 FOLLOWER]: Refusing update from remote peer 08ad9c3e352f492f9acb05219b72d0e0: Log matching property violated. Preceding OpId in replica: term: 3 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch) I0703 01:39:52.511729 10298 raft_consensus.cc:1081] T a87a2a21d927456d930023d9775aec80 P add911cdebcf44b8a64256c228929d30 [term 3 LEARNER]: Refusing update from remote peer 08ad9c3e352f492f9acb05219b72d0e0: Log matching property violated. Preceding OpId in replica: term: 3 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch) I0703 01:39:52.512615 10642 consensus_queue.cc:962] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3254ace7bc954d2ab9972a2723f45b0c" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54985 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I0703 01:39:52.513734 10642 consensus_queue.cc:962] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "add911cdebcf44b8a64256c228929d30" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 59090 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I0703 01:39:52.520929 10658 raft_consensus.cc:2684] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 [term 3 LEADER]: Committing config change with OpId 3.9: config changed from index 8 to 9, add911cdebcf44b8a64256c228929d30 (127.5.240.199) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } } peers { permanent_uuid: "d33993c1f2c148d9ae2504a60ed04173" member_type: VOTER last_known_addr { host: "127.5.240.200" port: 54337 } attrs { promote: false } } peers { permanent_uuid: "add911cdebcf44b8a64256c228929d30" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 59090 } attrs { promote: false } } peers { permanent_uuid: "3254ace7bc954d2ab9972a2723f45b0c" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54985 } attrs { promote: false } } unsafe_config_change: true } I0703 01:39:52.524772 10177 raft_consensus.cc:1081] T a87a2a21d927456d930023d9775aec80 P 7bb94c92677448c39c50efc79e39459a [term 3 FOLLOWER]: Refusing update from remote peer 08ad9c3e352f492f9acb05219b72d0e0: Log matching property violated. Preceding OpId in replica: term: 3 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch) I0703 01:39:52.524893 10298 raft_consensus.cc:2684] T a87a2a21d927456d930023d9775aec80 P add911cdebcf44b8a64256c228929d30 [term 3 FOLLOWER]: Committing config change with OpId 3.9: config changed from index 8 to 9, add911cdebcf44b8a64256c228929d30 (127.5.240.199) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } } peers { permanent_uuid: "d33993c1f2c148d9ae2504a60ed04173" member_type: VOTER last_known_addr { host: "127.5.240.200" port: 54337 } attrs { promote: false } } peers { permanent_uuid: "add911cdebcf44b8a64256c228929d30" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 59090 } attrs { promote: false } } peers { permanent_uuid: "3254ace7bc954d2ab9972a2723f45b0c" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54985 } attrs { promote: false } } unsafe_config_change: true } I0703 01:39:52.525372 10673 consensus_queue.cc:962] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s I0703 01:39:52.525849 10054 raft_consensus.cc:2684] T a87a2a21d927456d930023d9775aec80 P 3254ace7bc954d2ab9972a2723f45b0c [term 3 FOLLOWER]: Committing config change with OpId 3.9: config changed from index 8 to 9, add911cdebcf44b8a64256c228929d30 (127.5.240.199) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } } peers { permanent_uuid: "d33993c1f2c148d9ae2504a60ed04173" member_type: VOTER last_known_addr { host: "127.5.240.200" port: 54337 } attrs { promote: false } } peers { permanent_uuid: "add911cdebcf44b8a64256c228929d30" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 59090 } attrs { promote: false } } peers { permanent_uuid: "3254ace7bc954d2ab9972a2723f45b0c" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54985 } attrs { promote: false } } unsafe_config_change: true } I0703 01:39:52.526823 10177 raft_consensus.cc:2684] T a87a2a21d927456d930023d9775aec80 P 7bb94c92677448c39c50efc79e39459a [term 3 FOLLOWER]: Committing config change with OpId 3.9: config changed from index 8 to 9, add911cdebcf44b8a64256c228929d30 (127.5.240.199) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } } peers { permanent_uuid: "d33993c1f2c148d9ae2504a60ed04173" member_type: VOTER last_known_addr { host: "127.5.240.200" port: 54337 } attrs { promote: false } } peers { permanent_uuid: "add911cdebcf44b8a64256c228929d30" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 59090 } attrs { promote: false } } peers { permanent_uuid: "3254ace7bc954d2ab9972a2723f45b0c" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54985 } attrs { promote: false } } unsafe_config_change: true } I0703 01:39:52.529187 10421 raft_consensus.cc:1081] T a87a2a21d927456d930023d9775aec80 P d33993c1f2c148d9ae2504a60ed04173 [term 3 FOLLOWER]: Refusing update from remote peer 08ad9c3e352f492f9acb05219b72d0e0: Log matching property violated. Preceding OpId in replica: term: 3 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch) I0703 01:39:52.529866 10659 consensus_queue.cc:962] T a87a2a21d927456d930023d9775aec80 P 08ad9c3e352f492f9acb05219b72d0e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d33993c1f2c148d9ae2504a60ed04173" member_type: VOTER last_known_addr { host: "127.5.240.200" port: 54337 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s I0703 01:39:52.530854 10421 raft_consensus.cc:2684] T a87a2a21d927456d930023d9775aec80 P d33993c1f2c148d9ae2504a60ed04173 [term 3 FOLLOWER]: Committing config change with OpId 3.8: config changed from index 7 to 8, d33993c1f2c148d9ae2504a60ed04173 (127.5.240.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } } peers { permanent_uuid: "d33993c1f2c148d9ae2504a60ed04173" member_type: VOTER last_known_addr { host: "127.5.240.200" port: 54337 } attrs { promote: false } } peers { permanent_uuid: "add911cdebcf44b8a64256c228929d30" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 59090 } attrs { promote: true } } peers { permanent_uuid: "3254ace7bc954d2ab9972a2723f45b0c" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54985 } attrs { promote: false } } unsafe_config_change: true } I0703 01:39:52.533946 10547 catalog_manager.cc:3798] T a87a2a21d927456d930023d9775aec80 P 3254ace7bc954d2ab9972a2723f45b0c reported cstate change: config changed from index 7 to 9, add911cdebcf44b8a64256c228929d30 (127.5.240.199) changed from NON_VOTER to VOTER, d33993c1f2c148d9ae2504a60ed04173 (127.5.240.200) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "08ad9c3e352f492f9acb05219b72d0e0" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } } peers { permanent_uuid: "d33993c1f2c148d9ae2504a60ed04173" member_type: VOTER last_known_addr { host: "127.5.240.200" port: 54337 } attrs { promote: false } } peers { permanent_uuid: "add911cdebcf44b8a64256c228929d30" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 59090 } attrs { promote: false } } peers { permanent_uuid: "3254ace7bc954d2ab9972a2723f45b0c" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54985 } attrs { promote: false } } unsafe_config_change: true } I0703 01:39:52.536540 10421 raft_consensus.cc:2684] T a87a2a21d927456d930023d9775aec80 P d33993c1f2c148d9ae2504a60ed04173 [term 3 FOLLOWER]: Committing config change with OpId 3.9: config changed from index 8 to 9, add911cdebcf44b8a64256c228929d30 (127.5.240.199) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39947 } } peers { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 42244 } } peers { permanent_uuid: "d33993c1f2c148d9ae2504a60ed04173" member_type: VOTER last_known_addr { host: "127.5.240.200" port: 54337 } attrs { promote: false } } peers { permanent_uuid: "add911cdebcf44b8a64256c228929d30" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 59090 } attrs { promote: false } } peers { permanent_uuid: "3254ace7bc954d2ab9972a2723f45b0c" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54985 } attrs { promote: false } } unsafe_config_change: true } I0703 01:39:52.597720 6083 kudu-admin-test.cc:711] Waiting for Master to see new config... I0703 01:39:52.602844 6083 kudu-admin-test.cc:716] Tablet locations: tablet_id: "a87a2a21d927456d930023d9775aec80" replicas { ts_info { permanent_uuid: "7bb94c92677448c39c50efc79e39459a" rpc_addresses { host: "127.5.240.198" port: 39947 } } role: FOLLOWER } replicas { ts_info { permanent_uuid: "08ad9c3e352f492f9acb05219b72d0e0" rpc_addresses { host: "127.5.240.194" port: 42244 } } role: LEADER } replicas { ts_info { permanent_uuid: "d33993c1f2c148d9ae2504a60ed04173" rpc_addresses { host: "127.5.240.200" port: 54337 } } role: FOLLOWER } replicas { ts_info { permanent_uuid: "add911cdebcf44b8a64256c228929d30" rpc_addresses { host: "127.5.240.199" port: 59090 } } role: FOLLOWER } replicas { ts_info { permanent_uuid: "3254ace7bc954d2ab9972a2723f45b0c" rpc_addresses { host: "127.5.240.197" port: 54985 } } role: FOLLOWER } DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } I0703 01:39:52.603883 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 9613 I0703 01:39:52.604959 9729 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:39:52.655401 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 9613 W0703 01:39:52.665109 10354 connection.cc:512] client connection to 127.5.240.194:42244 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:42244): Connection reset by peer (error 104) W0703 01:39:52.665130 10109 connection.cc:512] server connection from 127.5.240.194:43050 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:43050): Connection reset by peer (error 104) W0703 01:39:52.665307 10110 connection.cc:512] client connection to 127.5.240.194:42244 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:42244): Connection reset by peer (error 104) W0703 01:39:52.665436 10534 connection.cc:512] server connection from 127.5.240.194:43161 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:43161): Connection reset by peer (error 104) W0703 01:39:52.665684 10232 connection.cc:512] server connection from 127.5.240.194:49726 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:49726): Connection reset by peer (error 104) W0703 01:39:52.665720 10355 connection.cc:512] server connection from 127.5.240.194:59517 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:59517): Connection reset by peer (error 104) W0703 01:39:52.666023 10232 connection.cc:512] client connection to 127.5.240.194:42244 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:42244): Connection reset by peer (error 104) W0703 01:39:52.666147 9989 connection.cc:512] server connection from 127.5.240.194:46965 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:46965): Connection reset by peer (error 104) W0703 01:39:52.666146 9988 connection.cc:512] client connection to 127.5.240.194:42244 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:42244): Connection reset by peer (error 104) I0703 01:39:52.666466 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 9979 I0703 01:39:52.667192 10095 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:39:52.711472 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 9979 W0703 01:39:52.719442 10535 connection.cc:512] server connection from 127.5.240.197:50542 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.197:50542): Connection reset by peer (error 104) I0703 01:39:52.719554 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 10101 I0703 01:39:52.720285 10217 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:39:52.765571 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 10101 W0703 01:39:52.773952 10533 connection.cc:512] server connection from 127.5.240.198:37752 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.198:37752): Connection reset by peer (error 104) I0703 01:39:52.774066 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 10223 I0703 01:39:52.774839 10339 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:39:52.819222 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 10223 W0703 01:39:52.827246 10532 connection.cc:512] server connection from 127.5.240.199:51962 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.199:51962): Connection reset by peer (error 104) I0703 01:39:52.827327 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 10345 I0703 01:39:52.828114 10461 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:39:52.871307 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 10345 W0703 01:39:52.879243 10532 connection.cc:512] server connection from 127.5.240.200:34330 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.200:34330): Connection reset by peer (error 104) I0703 01:39:52.879343 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master pid 10523 I0703 01:39:52.880148 10581 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:39:52.942509 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master with pid 10523 [ OK ] AdminCliTest.TestUnsafeChangeConfigWithFiveReplicaConfig (8390 ms) [ RUN ] AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig I0703 01:39:52.968318 6083 test_util.cc:208] Using random seed: 1107758186 I0703 01:39:52.969821 6083 ts_itest-base.cc:113] Starting cluster with: I0703 01:39:52.969851 6083 ts_itest-base.cc:114] -------------- I0703 01:39:52.969866 6083 ts_itest-base.cc:115] 5 tablet servers I0703 01:39:52.969883 6083 ts_itest-base.cc:116] 3 replicas per TS I0703 01:39:52.969897 6083 ts_itest-base.cc:117] -------------- I0703 01:39:52.971101 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:51485 --webserver_interface=127.5.240.254 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:39:53.150492 10687 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:39:53.150722 10687 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:39:53.152937 10687 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0703 01:39:53.152974 10687 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:39:53.153002 10687 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0703 01:39:53.153028 10687 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0703 01:39:53.160850 10687 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:51485 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.5.240.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:39:53.166389 10687 master_main.cc:80] Initializing master server... I0703 01:39:53.167434 10687 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 150208us I0703 01:39:53.168506 10687 fs_manager.cc:261] Metadata directory not provided I0703 01:39:53.168562 10687 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:39:53.168812 10687 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:39:53.168869 10687 server_base.cc:433] Attempting to create new FS layout instead I0703 01:39:53.172996 10687 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/instance: uuid: "bf659d42f827436f97ca8c0826afc8b7" format_stamp: "Formatted at 2018-07-03 08:39:53 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:53.173480 10687 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "bf659d42f827436f97ca8c0826afc8b7" format_stamp: "Formatted at 2018-07-03 08:39:53 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:53.181846 10687 fs_manager.cc:501] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s I0703 01:39:53.183032 10687 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:39:53.183817 10687 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:39:53.187434 10687 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s I0703 01:39:53.187485 10687 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal uuid: "bf659d42f827436f97ca8c0826afc8b7" format_stamp: "Formatted at 2018-07-03 08:39:53 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:53.187599 10687 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:39:53.239728 10687 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:39:53.241240 10687 master_main.cc:83] Starting Master server... I0703 01:39:53.278105 10687 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.254:51485 I0703 01:39:53.278596 10687 webserver.cc:175] Starting webserver on 127.5.240.254:0 I0703 01:39:53.278643 10687 webserver.cc:186] Document root disabled I0703 01:39:53.279831 10687 webserver.cc:313] Webserver started. Bound to: http://127.5.240.254:57609/ I0703 01:39:53.280418 10687 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb I0703 01:39:53.281193 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master as pid 10687 I0703 01:39:53.281846 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.5.240.193:0 --local_ip_for_outbound_sockets=127.5.240.193 --webserver_interface=127.5.240.193 --webserver_port=0 --tserver_master_addrs=127.5.240.254:51485 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0703 01:39:53.283587 10746 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:39:53.294906 10746 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P bf659d42f827436f97ca8c0826afc8b7: Bootstrap starting. I0703 01:39:53.297150 10746 tablet_bootstrap.cc:578] T 00000000000000000000000000000000 P bf659d42f827436f97ca8c0826afc8b7: No blocks or log segments found. Creating new log. I0703 01:39:53.297897 10746 log.cc:525] T 00000000000000000000000000000000 P bf659d42f827436f97ca8c0826afc8b7: Log is configured to *not* fsync() on all Append() calls I0703 01:39:53.300175 10746 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P bf659d42f827436f97ca8c0826afc8b7: No bootstrap required, opened a new log I0703 01:39:53.306363 10746 raft_consensus.cc:325] T 00000000000000000000000000000000 P bf659d42f827436f97ca8c0826afc8b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf659d42f827436f97ca8c0826afc8b7" member_type: VOTER } I0703 01:39:53.306560 10746 raft_consensus.cc:351] T 00000000000000000000000000000000 P bf659d42f827436f97ca8c0826afc8b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:39:53.306645 10746 raft_consensus.cc:628] T 00000000000000000000000000000000 P bf659d42f827436f97ca8c0826afc8b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf659d42f827436f97ca8c0826afc8b7, State: Initialized, Role: FOLLOWER I0703 01:39:53.307147 10746 consensus_queue.cc:227] T 00000000000000000000000000000000 P bf659d42f827436f97ca8c0826afc8b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf659d42f827436f97ca8c0826afc8b7" member_type: VOTER } I0703 01:39:53.307296 10746 raft_consensus.cc:365] T 00000000000000000000000000000000 P bf659d42f827436f97ca8c0826afc8b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:39:53.307374 10746 raft_consensus.cc:457] T 00000000000000000000000000000000 P bf659d42f827436f97ca8c0826afc8b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:39:53.307453 10746 raft_consensus.cc:2730] T 00000000000000000000000000000000 P bf659d42f827436f97ca8c0826afc8b7: Snoozing failure detection for 2.041s (starting election) I0703 01:39:53.307538 10746 raft_consensus.cc:2782] T 00000000000000000000000000000000 P bf659d42f827436f97ca8c0826afc8b7 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:39:53.308902 10746 raft_consensus.cc:479] T 00000000000000000000000000000000 P bf659d42f827436f97ca8c0826afc8b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf659d42f827436f97ca8c0826afc8b7" member_type: VOTER } I0703 01:39:53.309262 10746 leader_election.cc:270] T 00000000000000000000000000000000 P bf659d42f827436f97ca8c0826afc8b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:39:53.310725 10749 raft_consensus.cc:2730] T 00000000000000000000000000000000 P bf659d42f827436f97ca8c0826afc8b7: Snoozing failure detection for 2.068s (election complete) I0703 01:39:53.310982 10749 raft_consensus.cc:2537] T 00000000000000000000000000000000 P bf659d42f827436f97ca8c0826afc8b7 [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:39:53.312458 10749 raft_consensus.cc:593] T 00000000000000000000000000000000 P bf659d42f827436f97ca8c0826afc8b7 [term 1 LEADER]: Becoming Leader. State: Replica: bf659d42f827436f97ca8c0826afc8b7, State: Running, Role: LEADER I0703 01:39:53.312846 10749 consensus_queue.cc:204] T 00000000000000000000000000000000 P bf659d42f827436f97ca8c0826afc8b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf659d42f827436f97ca8c0826afc8b7" member_type: VOTER } I0703 01:39:53.315170 10746 sys_catalog.cc:337] T 00000000000000000000000000000000 P bf659d42f827436f97ca8c0826afc8b7 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "bf659d42f827436f97ca8c0826afc8b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf659d42f827436f97ca8c0826afc8b7" member_type: VOTER } } I0703 01:39:53.315467 10746 sys_catalog.cc:340] T 00000000000000000000000000000000 P bf659d42f827436f97ca8c0826afc8b7 [sys.catalog]: This master's current role is: LEADER I0703 01:39:53.315466 10750 sys_catalog.cc:337] T 00000000000000000000000000000000 P bf659d42f827436f97ca8c0826afc8b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf659d42f827436f97ca8c0826afc8b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf659d42f827436f97ca8c0826afc8b7" member_type: VOTER } } I0703 01:39:53.315623 10751 sys_catalog.cc:337] T 00000000000000000000000000000000 P bf659d42f827436f97ca8c0826afc8b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf659d42f827436f97ca8c0826afc8b7. Latest consensus state: current_term: 1 leader_uuid: "bf659d42f827436f97ca8c0826afc8b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf659d42f827436f97ca8c0826afc8b7" member_type: VOTER } } I0703 01:39:53.317477 10750 sys_catalog.cc:340] T 00000000000000000000000000000000 P bf659d42f827436f97ca8c0826afc8b7 [sys.catalog]: This master's current role is: LEADER I0703 01:39:53.317477 10751 sys_catalog.cc:340] T 00000000000000000000000000000000 P bf659d42f827436f97ca8c0826afc8b7 [sys.catalog]: This master's current role is: LEADER I0703 01:39:53.318634 10746 sys_catalog.cc:424] T 00000000000000000000000000000000 P bf659d42f827436f97ca8c0826afc8b7 [sys.catalog]: configured and running, proceeding with master startup. I0703 01:39:53.319330 10753 catalog_manager.cc:1006] Loading table and tablet metadata into memory... I0703 01:39:53.325933 10687 master_main.cc:86] Master server successfully started. I0703 01:39:53.331164 10753 catalog_manager.cc:1017] Initializing Kudu internal certificate authority... I0703 01:39:53.371778 10753 catalog_manager.cc:904] Generated new certificate authority record I0703 01:39:53.373708 10753 catalog_manager.cc:1026] Loading token signing keys... I0703 01:39:53.384238 10753 catalog_manager.cc:4102] T 00000000000000000000000000000000 P bf659d42f827436f97ca8c0826afc8b7: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:39:53.453763 10747 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:39:53.453995 10747 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:39:53.456032 10747 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:39:53.456136 10747 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.193 I0703 01:39:53.463012 10747 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.5.240.193 --webserver_port=0 --tserver_master_addrs=127.5.240.254:51485 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.193 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:39:53.469645 10747 tablet_server_main.cc:85] Initializing tablet server... I0703 01:39:53.470849 10747 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 150208us I0703 01:39:53.471823 10747 fs_manager.cc:261] Metadata directory not provided I0703 01:39:53.471886 10747 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:39:53.472484 10747 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:39:53.472550 10747 server_base.cc:433] Attempting to create new FS layout instead I0703 01:39:53.476686 10747 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "200046fa8cd344f4a557b6feab568415" format_stamp: "Formatted at 2018-07-03 08:39:53 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:53.477144 10747 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "200046fa8cd344f4a557b6feab568415" format_stamp: "Formatted at 2018-07-03 08:39:53 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:53.485584 10747 fs_manager.cc:501] Time spent creating directory manager: real 0.008s user 0.004s sys 0.003s I0703 01:39:53.486778 10747 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:39:53.487169 10747 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:39:53.490617 10747 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0703 01:39:53.490677 10747 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal uuid: "200046fa8cd344f4a557b6feab568415" format_stamp: "Formatted at 2018-07-03 08:39:53 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:53.490799 10747 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:39:53.575477 10747 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:39:53.578006 10747 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:39:53.578429 10747 tablet_server_main.cc:90] Starting tablet server... I0703 01:39:53.675509 10747 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.193:48069 I0703 01:39:53.675951 10747 webserver.cc:175] Starting webserver on 127.5.240.193:0 I0703 01:39:53.675987 10747 webserver.cc:186] Document root disabled I0703 01:39:53.677141 10747 webserver.cc:313] Webserver started. Bound to: http://127.5.240.193:33448/ I0703 01:39:53.677695 10747 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb I0703 01:39:53.680016 10747 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:39:53.683840 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 10747 I0703 01:39:53.684789 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.5.240.194:0 --local_ip_for_outbound_sockets=127.5.240.194 --webserver_interface=127.5.240.194 --webserver_port=0 --tserver_master_addrs=127.5.240.254:51485 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0703 01:39:53.695241 10876 heartbeater.cc:346] Connected to a master server at 127.5.240.254:51485 I0703 01:39:53.695407 10876 heartbeater.cc:426] Registering TS with master... I0703 01:39:53.696374 10876 heartbeater.cc:475] Master 127.5.240.254:51485 requested a full tablet report, sending... I0703 01:39:53.697777 10713 ts_manager.cc:98] Registered new tserver with Master: 200046fa8cd344f4a557b6feab568415 (127.5.240.193:48069) I0703 01:39:53.699650 10713 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.193:42143 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:39:53.843027 10880 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:39:53.843235 10880 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:39:53.845232 10880 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:39:53.845320 10880 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.194 I0703 01:39:53.852334 10880 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.194:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.5.240.194 --webserver_port=0 --tserver_master_addrs=127.5.240.254:51485 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.194 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:39:53.858742 10880 tablet_server_main.cc:85] Initializing tablet server... I0703 01:39:53.859851 10880 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 150208us I0703 01:39:53.860782 10880 fs_manager.cc:261] Metadata directory not provided I0703 01:39:53.860837 10880 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:39:53.861047 10880 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:39:53.861104 10880 server_base.cc:433] Attempting to create new FS layout instead I0703 01:39:53.865137 10880 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "20e529e97c57431bb03e326238d1bf8a" format_stamp: "Formatted at 2018-07-03 08:39:53 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:53.865659 10880 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "20e529e97c57431bb03e326238d1bf8a" format_stamp: "Formatted at 2018-07-03 08:39:53 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:53.872920 10880 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.003s sys 0.003s I0703 01:39:53.874078 10880 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:39:53.874436 10880 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:39:53.877996 10880 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0703 01:39:53.878046 10880 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal uuid: "20e529e97c57431bb03e326238d1bf8a" format_stamp: "Formatted at 2018-07-03 08:39:53 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:53.878155 10880 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:39:53.903998 10880 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:39:53.906225 10880 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:39:53.906592 10880 tablet_server_main.cc:90] Starting tablet server... I0703 01:39:53.989037 10880 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.194:58915 I0703 01:39:53.989466 10880 webserver.cc:175] Starting webserver on 127.5.240.194:0 I0703 01:39:53.989500 10880 webserver.cc:186] Document root disabled I0703 01:39:53.990732 10880 webserver.cc:313] Webserver started. Bound to: http://127.5.240.194:41257/ I0703 01:39:53.991271 10880 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb I0703 01:39:53.993058 10880 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:39:53.995944 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 10880 I0703 01:39:53.996878 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.5.240.195:0 --local_ip_for_outbound_sockets=127.5.240.195 --webserver_interface=127.5.240.195 --webserver_port=0 --tserver_master_addrs=127.5.240.254:51485 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0703 01:39:54.006821 10999 heartbeater.cc:346] Connected to a master server at 127.5.240.254:51485 I0703 01:39:54.006978 10999 heartbeater.cc:426] Registering TS with master... I0703 01:39:54.007772 10999 heartbeater.cc:475] Master 127.5.240.254:51485 requested a full tablet report, sending... I0703 01:39:54.008826 10713 ts_manager.cc:98] Registered new tserver with Master: 20e529e97c57431bb03e326238d1bf8a (127.5.240.194:58915) I0703 01:39:54.009655 10713 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.194:33101 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:39:54.153358 11002 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:39:54.153586 11002 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:39:54.155596 11002 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:39:54.155689 11002 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.195 I0703 01:39:54.162256 11002 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.195:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.5.240.195 --webserver_port=0 --tserver_master_addrs=127.5.240.254:51485 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.195 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:39:54.168762 11002 tablet_server_main.cc:85] Initializing tablet server... I0703 01:39:54.169893 11002 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 150708us I0703 01:39:54.170826 11002 fs_manager.cc:261] Metadata directory not provided I0703 01:39:54.170879 11002 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:39:54.171092 11002 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:39:54.171149 11002 server_base.cc:433] Attempting to create new FS layout instead I0703 01:39:54.175163 11002 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "bb2c63b1e9fa4dc3aa89afac6501f1ff" format_stamp: "Formatted at 2018-07-03 08:39:54 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:54.175693 11002 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "bb2c63b1e9fa4dc3aa89afac6501f1ff" format_stamp: "Formatted at 2018-07-03 08:39:54 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:54.182976 11002 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.002s sys 0.004s I0703 01:39:54.184252 11002 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:39:54.184666 11002 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:39:54.188300 11002 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s I0703 01:39:54.188350 11002 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal uuid: "bb2c63b1e9fa4dc3aa89afac6501f1ff" format_stamp: "Formatted at 2018-07-03 08:39:54 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:54.188464 11002 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:39:54.238397 11002 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:39:54.240327 11002 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:39:54.240710 11002 tablet_server_main.cc:90] Starting tablet server... I0703 01:39:54.324333 11002 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.195:34920 I0703 01:39:54.324775 11002 webserver.cc:175] Starting webserver on 127.5.240.195:0 I0703 01:39:54.324813 11002 webserver.cc:186] Document root disabled I0703 01:39:54.325994 11002 webserver.cc:313] Webserver started. Bound to: http://127.5.240.195:42570/ I0703 01:39:54.326515 11002 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb I0703 01:39:54.328327 11002 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:39:54.329478 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 11002 I0703 01:39:54.330116 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.5.240.196:0 --local_ip_for_outbound_sockets=127.5.240.196 --webserver_interface=127.5.240.196 --webserver_port=0 --tserver_master_addrs=127.5.240.254:51485 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0703 01:39:54.340135 11121 heartbeater.cc:346] Connected to a master server at 127.5.240.254:51485 I0703 01:39:54.340315 11121 heartbeater.cc:426] Registering TS with master... I0703 01:39:54.341253 11121 heartbeater.cc:475] Master 127.5.240.254:51485 requested a full tablet report, sending... I0703 01:39:54.342340 10713 ts_manager.cc:98] Registered new tserver with Master: bb2c63b1e9fa4dc3aa89afac6501f1ff (127.5.240.195:34920) I0703 01:39:54.343212 10713 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.195:48242 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:39:54.488256 11124 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:39:54.488459 11124 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:39:54.490443 11124 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:39:54.490535 11124 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.196 I0703 01:39:54.497612 11124 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.196:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb --webserver_interface=127.5.240.196 --webserver_port=0 --tserver_master_addrs=127.5.240.254:51485 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.196 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:39:54.504067 11124 tablet_server_main.cc:85] Initializing tablet server... I0703 01:39:54.505159 11124 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 150708us I0703 01:39:54.506103 11124 fs_manager.cc:261] Metadata directory not provided I0703 01:39:54.506156 11124 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:39:54.506362 11124 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:39:54.506418 11124 server_base.cc:433] Attempting to create new FS layout instead I0703 01:39:54.510397 11124 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/instance: uuid: "6b95eacb907e4467b3633c2cc9ad055b" format_stamp: "Formatted at 2018-07-03 08:39:54 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:54.510903 11124 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal/instance: uuid: "6b95eacb907e4467b3633c2cc9ad055b" format_stamp: "Formatted at 2018-07-03 08:39:54 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:54.518051 11124 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.001s sys 0.005s I0703 01:39:54.519204 11124 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:39:54.519585 11124 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:39:54.523150 11124 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0703 01:39:54.523201 11124 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal uuid: "6b95eacb907e4467b3633c2cc9ad055b" format_stamp: "Formatted at 2018-07-03 08:39:54 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:54.523310 11124 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:39:54.572227 11124 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:39:54.574023 11124 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:39:54.574378 11124 tablet_server_main.cc:90] Starting tablet server... I0703 01:39:54.659283 11124 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.196:48168 I0703 01:39:54.659720 11124 webserver.cc:175] Starting webserver on 127.5.240.196:0 I0703 01:39:54.659752 11124 webserver.cc:186] Document root disabled I0703 01:39:54.660384 11124 webserver.cc:313] Webserver started. Bound to: http://127.5.240.196:36331/ I0703 01:39:54.660921 11124 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb I0703 01:39:54.661197 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 11124 I0703 01:39:54.661857 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data --block_manager=log --rpc_bind_addresses=127.5.240.197:0 --local_ip_for_outbound_sockets=127.5.240.197 --webserver_interface=127.5.240.197 --webserver_port=0 --tserver_master_addrs=127.5.240.254:51485 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0703 01:39:54.663842 11124 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:39:54.676333 11243 heartbeater.cc:346] Connected to a master server at 127.5.240.254:51485 I0703 01:39:54.676473 11243 heartbeater.cc:426] Registering TS with master... I0703 01:39:54.677258 11243 heartbeater.cc:475] Master 127.5.240.254:51485 requested a full tablet report, sending... I0703 01:39:54.678220 10713 ts_manager.cc:98] Registered new tserver with Master: 6b95eacb907e4467b3633c2cc9ad055b (127.5.240.196:48168) I0703 01:39:54.679059 10713 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.196:36458 I0703 01:39:54.702131 10876 heartbeater.cc:467] Master 127.5.240.254:51485 was elected leader, sending a full tablet report... WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:39:54.821162 11246 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:39:54.821377 11246 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:39:54.823396 11246 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:39:54.823487 11246 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.197 I0703 01:39:54.829645 11246 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.197:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb --webserver_interface=127.5.240.197 --webserver_port=0 --tserver_master_addrs=127.5.240.254:51485 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.197 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:39:54.836248 11246 tablet_server_main.cc:85] Initializing tablet server... I0703 01:39:54.837373 11246 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 150708us I0703 01:39:54.838318 11246 fs_manager.cc:261] Metadata directory not provided I0703 01:39:54.838371 11246 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:39:54.838620 11246 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:39:54.838677 11246 server_base.cc:433] Attempting to create new FS layout instead I0703 01:39:54.842803 11246 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/instance: uuid: "e4d277c8cd7248b59ecbaa2db3b09d5c" format_stamp: "Formatted at 2018-07-03 08:39:54 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:54.843293 11246 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal/instance: uuid: "e4d277c8cd7248b59ecbaa2db3b09d5c" format_stamp: "Formatted at 2018-07-03 08:39:54 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:54.851092 11246 fs_manager.cc:501] Time spent creating directory manager: real 0.008s user 0.004s sys 0.002s I0703 01:39:54.852294 11246 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:39:54.852704 11246 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:39:54.855787 11246 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0703 01:39:54.855845 11246 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal uuid: "e4d277c8cd7248b59ecbaa2db3b09d5c" format_stamp: "Formatted at 2018-07-03 08:39:54 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:54.855959 11246 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:39:54.894477 11246 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:39:54.896386 11246 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:39:54.896737 11246 tablet_server_main.cc:90] Starting tablet server... I0703 01:39:54.985690 11246 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.197:37924 I0703 01:39:54.986120 11246 webserver.cc:175] Starting webserver on 127.5.240.197:0 I0703 01:39:54.986156 11246 webserver.cc:186] Document root disabled I0703 01:39:54.987334 11246 webserver.cc:313] Webserver started. Bound to: http://127.5.240.197:34695/ I0703 01:39:54.987869 11246 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb I0703 01:39:54.989794 11246 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:39:54.994485 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 11246 I0703 01:39:55.002068 11365 heartbeater.cc:346] Connected to a master server at 127.5.240.254:51485 I0703 01:39:55.002230 11365 heartbeater.cc:426] Registering TS with master... I0703 01:39:55.003082 11365 heartbeater.cc:475] Master 127.5.240.254:51485 requested a full tablet report, sending... I0703 01:39:55.004103 10713 ts_manager.cc:98] Registered new tserver with Master: e4d277c8cd7248b59ecbaa2db3b09d5c (127.5.240.197:37924) I0703 01:39:55.004736 6083 external_mini_cluster.cc:490] 5 TS(s) registered with all masters I0703 01:39:55.005054 10713 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.197:37129 I0703 01:39:55.011509 10999 heartbeater.cc:467] Master 127.5.240.254:51485 was elected leader, sending a full tablet report... I0703 01:39:55.015532 10712 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.5.240.254:49685: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0703 01:39:55.050047 10936 tablet_service.cc:759] Processing CreateTablet for tablet 88835abd0ea34ff7acd90bb2c84931f5 (table=TestTable [id=20caf1ed0e1d4dd69440a2ed2a5bac36]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:39:55.050747 10936 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 88835abd0ea34ff7acd90bb2c84931f5. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:39:55.050597 11302 tablet_service.cc:759] Processing CreateTablet for tablet 88835abd0ea34ff7acd90bb2c84931f5 (table=TestTable [id=20caf1ed0e1d4dd69440a2ed2a5bac36]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:39:55.051288 11302 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 88835abd0ea34ff7acd90bb2c84931f5. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:39:55.053607 11180 tablet_service.cc:759] Processing CreateTablet for tablet 88835abd0ea34ff7acd90bb2c84931f5 (table=TestTable [id=20caf1ed0e1d4dd69440a2ed2a5bac36]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:39:55.054468 11180 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 88835abd0ea34ff7acd90bb2c84931f5. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:39:55.061271 11302 ts_tablet_manager.cc:1173] T 88835abd0ea34ff7acd90bb2c84931f5 P e4d277c8cd7248b59ecbaa2db3b09d5c: Registered tablet (data state: TABLET_DATA_READY) I0703 01:39:55.062696 11385 ts_tablet_manager.cc:1028] T 88835abd0ea34ff7acd90bb2c84931f5 P e4d277c8cd7248b59ecbaa2db3b09d5c: Bootstrapping tablet I0703 01:39:55.063074 11385 tablet_bootstrap.cc:427] T 88835abd0ea34ff7acd90bb2c84931f5 P e4d277c8cd7248b59ecbaa2db3b09d5c: Bootstrap starting. I0703 01:39:55.064741 10936 ts_tablet_manager.cc:1173] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a: Registered tablet (data state: TABLET_DATA_READY) I0703 01:39:55.066148 11385 tablet_bootstrap.cc:578] T 88835abd0ea34ff7acd90bb2c84931f5 P e4d277c8cd7248b59ecbaa2db3b09d5c: No blocks or log segments found. Creating new log. I0703 01:39:55.066905 11385 log.cc:525] T 88835abd0ea34ff7acd90bb2c84931f5 P e4d277c8cd7248b59ecbaa2db3b09d5c: Log is configured to *not* fsync() on all Append() calls I0703 01:39:55.068027 11180 ts_tablet_manager.cc:1173] T 88835abd0ea34ff7acd90bb2c84931f5 P 6b95eacb907e4467b3633c2cc9ad055b: Registered tablet (data state: TABLET_DATA_READY) I0703 01:39:55.068511 11385 tablet_bootstrap.cc:427] T 88835abd0ea34ff7acd90bb2c84931f5 P e4d277c8cd7248b59ecbaa2db3b09d5c: No bootstrap required, opened a new log I0703 01:39:55.068754 11385 ts_tablet_manager.cc:1045] T 88835abd0ea34ff7acd90bb2c84931f5 P e4d277c8cd7248b59ecbaa2db3b09d5c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s I0703 01:39:55.068922 11388 ts_tablet_manager.cc:1028] T 88835abd0ea34ff7acd90bb2c84931f5 P 6b95eacb907e4467b3633c2cc9ad055b: Bootstrapping tablet I0703 01:39:55.069501 11388 tablet_bootstrap.cc:427] T 88835abd0ea34ff7acd90bb2c84931f5 P 6b95eacb907e4467b3633c2cc9ad055b: Bootstrap starting. I0703 01:39:55.069777 11386 ts_tablet_manager.cc:1028] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a: Bootstrapping tablet I0703 01:39:55.070123 11386 tablet_bootstrap.cc:427] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a: Bootstrap starting. I0703 01:39:55.072106 11386 tablet_bootstrap.cc:578] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a: No blocks or log segments found. Creating new log. I0703 01:39:55.072824 11386 log.cc:525] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a: Log is configured to *not* fsync() on all Append() calls I0703 01:39:55.072912 11388 tablet_bootstrap.cc:578] T 88835abd0ea34ff7acd90bb2c84931f5 P 6b95eacb907e4467b3633c2cc9ad055b: No blocks or log segments found. Creating new log. I0703 01:39:55.074167 11388 log.cc:525] T 88835abd0ea34ff7acd90bb2c84931f5 P 6b95eacb907e4467b3633c2cc9ad055b: Log is configured to *not* fsync() on all Append() calls I0703 01:39:55.075171 11386 tablet_bootstrap.cc:427] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a: No bootstrap required, opened a new log I0703 01:39:55.075338 11386 ts_tablet_manager.cc:1045] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s I0703 01:39:55.078435 11388 tablet_bootstrap.cc:427] T 88835abd0ea34ff7acd90bb2c84931f5 P 6b95eacb907e4467b3633c2cc9ad055b: No bootstrap required, opened a new log I0703 01:39:55.078196 11385 raft_consensus.cc:325] T 88835abd0ea34ff7acd90bb2c84931f5 P e4d277c8cd7248b59ecbaa2db3b09d5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20e529e97c57431bb03e326238d1bf8a" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 58915 } } peers { permanent_uuid: "6b95eacb907e4467b3633c2cc9ad055b" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 48168 } } peers { permanent_uuid: "e4d277c8cd7248b59ecbaa2db3b09d5c" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 37924 } } I0703 01:39:55.078636 11385 raft_consensus.cc:351] T 88835abd0ea34ff7acd90bb2c84931f5 P e4d277c8cd7248b59ecbaa2db3b09d5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:39:55.078657 11388 ts_tablet_manager.cc:1045] T 88835abd0ea34ff7acd90bb2c84931f5 P 6b95eacb907e4467b3633c2cc9ad055b: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.005s I0703 01:39:55.078820 11385 raft_consensus.cc:628] T 88835abd0ea34ff7acd90bb2c84931f5 P e4d277c8cd7248b59ecbaa2db3b09d5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4d277c8cd7248b59ecbaa2db3b09d5c, State: Initialized, Role: FOLLOWER I0703 01:39:55.079428 11385 consensus_queue.cc:227] T 88835abd0ea34ff7acd90bb2c84931f5 P e4d277c8cd7248b59ecbaa2db3b09d5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20e529e97c57431bb03e326238d1bf8a" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 58915 } } peers { permanent_uuid: "6b95eacb907e4467b3633c2cc9ad055b" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 48168 } } peers { permanent_uuid: "e4d277c8cd7248b59ecbaa2db3b09d5c" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 37924 } } I0703 01:39:55.080998 11365 heartbeater.cc:467] Master 127.5.240.254:51485 was elected leader, sending a full tablet report... I0703 01:39:55.082300 11386 raft_consensus.cc:325] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20e529e97c57431bb03e326238d1bf8a" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 58915 } } peers { permanent_uuid: "6b95eacb907e4467b3633c2cc9ad055b" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 48168 } } peers { permanent_uuid: "e4d277c8cd7248b59ecbaa2db3b09d5c" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 37924 } } I0703 01:39:55.082577 11386 raft_consensus.cc:351] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:39:55.082657 11386 raft_consensus.cc:628] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20e529e97c57431bb03e326238d1bf8a, State: Initialized, Role: FOLLOWER I0703 01:39:55.083009 11386 consensus_queue.cc:227] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20e529e97c57431bb03e326238d1bf8a" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 58915 } } peers { permanent_uuid: "6b95eacb907e4467b3633c2cc9ad055b" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 48168 } } peers { permanent_uuid: "e4d277c8cd7248b59ecbaa2db3b09d5c" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 37924 } } I0703 01:39:55.084200 11386 ts_tablet_manager.cc:1073] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a: Time spent starting tablet: real 0.009s user 0.007s sys 0.002s I0703 01:39:55.085688 11385 ts_tablet_manager.cc:1073] T 88835abd0ea34ff7acd90bb2c84931f5 P e4d277c8cd7248b59ecbaa2db3b09d5c: Time spent starting tablet: real 0.017s user 0.009s sys 0.002s I0703 01:39:55.087916 11388 raft_consensus.cc:325] T 88835abd0ea34ff7acd90bb2c84931f5 P 6b95eacb907e4467b3633c2cc9ad055b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20e529e97c57431bb03e326238d1bf8a" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 58915 } } peers { permanent_uuid: "6b95eacb907e4467b3633c2cc9ad055b" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 48168 } } peers { permanent_uuid: "e4d277c8cd7248b59ecbaa2db3b09d5c" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 37924 } } I0703 01:39:55.088176 11388 raft_consensus.cc:351] T 88835abd0ea34ff7acd90bb2c84931f5 P 6b95eacb907e4467b3633c2cc9ad055b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:39:55.088243 11388 raft_consensus.cc:628] T 88835abd0ea34ff7acd90bb2c84931f5 P 6b95eacb907e4467b3633c2cc9ad055b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b95eacb907e4467b3633c2cc9ad055b, State: Initialized, Role: FOLLOWER I0703 01:39:55.088559 11388 consensus_queue.cc:227] T 88835abd0ea34ff7acd90bb2c84931f5 P 6b95eacb907e4467b3633c2cc9ad055b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20e529e97c57431bb03e326238d1bf8a" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 58915 } } peers { permanent_uuid: "6b95eacb907e4467b3633c2cc9ad055b" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 48168 } } peers { permanent_uuid: "e4d277c8cd7248b59ecbaa2db3b09d5c" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 37924 } } I0703 01:39:55.089720 11243 heartbeater.cc:467] Master 127.5.240.254:51485 was elected leader, sending a full tablet report... I0703 01:39:55.090226 11388 ts_tablet_manager.cc:1073] T 88835abd0ea34ff7acd90bb2c84931f5 P 6b95eacb907e4467b3633c2cc9ad055b: Time spent starting tablet: real 0.011s user 0.008s sys 0.002s I0703 01:39:55.128443 11392 raft_consensus.cc:457] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0703 01:39:55.128535 11392 raft_consensus.cc:2730] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a: Snoozing failure detection for 1.932s (starting election) I0703 01:39:55.128636 11392 raft_consensus.cc:479] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20e529e97c57431bb03e326238d1bf8a" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 58915 } } peers { permanent_uuid: "6b95eacb907e4467b3633c2cc9ad055b" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 48168 } } peers { permanent_uuid: "e4d277c8cd7248b59ecbaa2db3b09d5c" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 37924 } } I0703 01:39:55.129287 11392 leader_election.cc:243] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 6b95eacb907e4467b3633c2cc9ad055b (127.5.240.196:48168) I0703 01:39:55.129611 11392 leader_election.cc:243] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer e4d277c8cd7248b59ecbaa2db3b09d5c (127.5.240.197:37924) I0703 01:39:55.138635 11322 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "88835abd0ea34ff7acd90bb2c84931f5" candidate_uuid: "20e529e97c57431bb03e326238d1bf8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4d277c8cd7248b59ecbaa2db3b09d5c" is_pre_election: true I0703 01:39:55.138871 11322 raft_consensus.cc:2730] T 88835abd0ea34ff7acd90bb2c84931f5 P e4d277c8cd7248b59ecbaa2db3b09d5c: Snoozing failure detection for 1.616s (vote granted) I0703 01:39:55.138798 11200 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "88835abd0ea34ff7acd90bb2c84931f5" candidate_uuid: "20e529e97c57431bb03e326238d1bf8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b95eacb907e4467b3633c2cc9ad055b" is_pre_election: true I0703 01:39:55.138959 11322 raft_consensus.cc:2243] T 88835abd0ea34ff7acd90bb2c84931f5 P e4d277c8cd7248b59ecbaa2db3b09d5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20e529e97c57431bb03e326238d1bf8a in term 0. I0703 01:39:55.139020 11200 raft_consensus.cc:2730] T 88835abd0ea34ff7acd90bb2c84931f5 P 6b95eacb907e4467b3633c2cc9ad055b: Snoozing failure detection for 1.587s (vote granted) I0703 01:39:55.139107 11200 raft_consensus.cc:2243] T 88835abd0ea34ff7acd90bb2c84931f5 P 6b95eacb907e4467b3633c2cc9ad055b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20e529e97c57431bb03e326238d1bf8a in term 0. I0703 01:39:55.139554 10888 leader_election.cc:379] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a [CANDIDATE]: Term 1 pre-election: Vote granted by peer e4d277c8cd7248b59ecbaa2db3b09d5c (127.5.240.197:37924) I0703 01:39:55.139664 10888 leader_election.cc:270] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0703 01:39:55.139791 10890 leader_election.cc:379] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a [CANDIDATE]: Term 1 pre-election: Vote granted by peer 6b95eacb907e4467b3633c2cc9ad055b (127.5.240.196:48168) I0703 01:39:55.139978 11392 raft_consensus.cc:2730] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a: Snoozing failure detection for 2.116s (election complete) I0703 01:39:55.140095 11392 raft_consensus.cc:2537] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a [term 0 FOLLOWER]: Leader pre-election won for term 1 I0703 01:39:55.140151 11392 raft_consensus.cc:457] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0703 01:39:55.140206 11392 raft_consensus.cc:2730] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a: Snoozing failure detection for 2.133s (starting election) I0703 01:39:55.140259 11392 raft_consensus.cc:2782] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a [term 0 FOLLOWER]: Advancing to term 1 I0703 01:39:55.141726 11392 raft_consensus.cc:479] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20e529e97c57431bb03e326238d1bf8a" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 58915 } } peers { permanent_uuid: "6b95eacb907e4467b3633c2cc9ad055b" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 48168 } } peers { permanent_uuid: "e4d277c8cd7248b59ecbaa2db3b09d5c" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 37924 } } I0703 01:39:55.142048 11392 leader_election.cc:243] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a [CANDIDATE]: Term 1 election: Requesting vote from peer 6b95eacb907e4467b3633c2cc9ad055b (127.5.240.196:48168) I0703 01:39:55.142201 11392 leader_election.cc:243] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a [CANDIDATE]: Term 1 election: Requesting vote from peer e4d277c8cd7248b59ecbaa2db3b09d5c (127.5.240.197:37924) I0703 01:39:55.142629 11200 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "88835abd0ea34ff7acd90bb2c84931f5" candidate_uuid: "20e529e97c57431bb03e326238d1bf8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b95eacb907e4467b3633c2cc9ad055b" I0703 01:39:55.142745 11200 raft_consensus.cc:2782] T 88835abd0ea34ff7acd90bb2c84931f5 P 6b95eacb907e4467b3633c2cc9ad055b [term 0 FOLLOWER]: Advancing to term 1 I0703 01:39:55.142791 11322 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "88835abd0ea34ff7acd90bb2c84931f5" candidate_uuid: "20e529e97c57431bb03e326238d1bf8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4d277c8cd7248b59ecbaa2db3b09d5c" I0703 01:39:55.142837 11200 raft_consensus.cc:2730] T 88835abd0ea34ff7acd90bb2c84931f5 P 6b95eacb907e4467b3633c2cc9ad055b: Snoozing failure detection for 1.974s (vote granted) I0703 01:39:55.142901 11322 raft_consensus.cc:2782] T 88835abd0ea34ff7acd90bb2c84931f5 P e4d277c8cd7248b59ecbaa2db3b09d5c [term 0 FOLLOWER]: Advancing to term 1 I0703 01:39:55.142992 11322 raft_consensus.cc:2730] T 88835abd0ea34ff7acd90bb2c84931f5 P e4d277c8cd7248b59ecbaa2db3b09d5c: Snoozing failure detection for 1.825s (vote granted) I0703 01:39:55.144148 11200 raft_consensus.cc:2243] T 88835abd0ea34ff7acd90bb2c84931f5 P 6b95eacb907e4467b3633c2cc9ad055b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20e529e97c57431bb03e326238d1bf8a in term 1. I0703 01:39:55.144315 11322 raft_consensus.cc:2243] T 88835abd0ea34ff7acd90bb2c84931f5 P e4d277c8cd7248b59ecbaa2db3b09d5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20e529e97c57431bb03e326238d1bf8a in term 1. I0703 01:39:55.144556 10890 leader_election.cc:379] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a [CANDIDATE]: Term 1 election: Vote granted by peer 6b95eacb907e4467b3633c2cc9ad055b (127.5.240.196:48168) I0703 01:39:55.144649 10890 leader_election.cc:270] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:39:55.144749 10888 leader_election.cc:379] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a [CANDIDATE]: Term 1 election: Vote granted by peer e4d277c8cd7248b59ecbaa2db3b09d5c (127.5.240.197:37924) I0703 01:39:55.144858 11392 raft_consensus.cc:2730] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a: Snoozing failure detection for 2.080s (election complete) I0703 01:39:55.144955 11392 raft_consensus.cc:2537] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:39:55.145280 11392 raft_consensus.cc:593] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a [term 1 LEADER]: Becoming Leader. State: Replica: 20e529e97c57431bb03e326238d1bf8a, State: Running, Role: LEADER I0703 01:39:55.145632 11392 consensus_queue.cc:204] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20e529e97c57431bb03e326238d1bf8a" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 58915 } } peers { permanent_uuid: "6b95eacb907e4467b3633c2cc9ad055b" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 48168 } } peers { permanent_uuid: "e4d277c8cd7248b59ecbaa2db3b09d5c" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 37924 } } I0703 01:39:55.149870 10713 catalog_manager.cc:3798] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a reported cstate change: term changed from 0 to 1, leader changed from to 20e529e97c57431bb03e326238d1bf8a (127.5.240.194). New cstate: current_term: 1 leader_uuid: "20e529e97c57431bb03e326238d1bf8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20e529e97c57431bb03e326238d1bf8a" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 58915 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b95eacb907e4467b3633c2cc9ad055b" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 48168 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4d277c8cd7248b59ecbaa2db3b09d5c" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 37924 } health_report { overall_health: UNKNOWN } } } I0703 01:39:55.157681 6083 external_mini_cluster.cc:490] 5 TS(s) registered with all masters I0703 01:39:55.158746 6083 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 200046fa8cd344f4a557b6feab568415 to finish bootstrapping W0703 01:39:55.164459 11244 tablet.cc:1678] T 88835abd0ea34ff7acd90bb2c84931f5 P 6b95eacb907e4467b3633c2cc9ad055b: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:39:55.165205 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 20e529e97c57431bb03e326238d1bf8a to finish bootstrapping I0703 01:39:55.169440 6083 ts_itest-base.cc:233] Waiting for 0 tablets on tserver bb2c63b1e9fa4dc3aa89afac6501f1ff to finish bootstrapping I0703 01:39:55.175195 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 6b95eacb907e4467b3633c2cc9ad055b to finish bootstrapping I0703 01:39:55.179360 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver e4d277c8cd7248b59ecbaa2db3b09d5c to finish bootstrapping I0703 01:39:55.184111 6083 kudu-admin-test.cc:747] Waiting for Master to see the current replicas... I0703 01:39:55.185261 6083 kudu-admin-test.cc:754] Tablet locations: tablet_id: "88835abd0ea34ff7acd90bb2c84931f5" replicas { ts_info { permanent_uuid: "20e529e97c57431bb03e326238d1bf8a" rpc_addresses { host: "127.5.240.194" port: 58915 } } role: LEADER } replicas { ts_info { permanent_uuid: "6b95eacb907e4467b3633c2cc9ad055b" rpc_addresses { host: "127.5.240.196" port: 48168 } } role: FOLLOWER } replicas { ts_info { permanent_uuid: "e4d277c8cd7248b59ecbaa2db3b09d5c" rpc_addresses { host: "127.5.240.197" port: 37924 } } role: FOLLOWER } DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } W0703 01:39:55.243999 11000 tablet.cc:1678] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a: Can't schedule compaction. Clean time has not been advanced past its initial value. W0703 01:39:55.244191 11366 tablet.cc:1678] T 88835abd0ea34ff7acd90bb2c84931f5 P e4d277c8cd7248b59ecbaa2db3b09d5c: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:39:55.345427 11121 heartbeater.cc:467] Master 127.5.240.254:51485 was elected leader, sending a full tablet report... I0703 01:39:55.586797 11322 raft_consensus.cc:1081] T 88835abd0ea34ff7acd90bb2c84931f5 P e4d277c8cd7248b59ecbaa2db3b09d5c [term 1 FOLLOWER]: Refusing update from remote peer 20e529e97c57431bb03e326238d1bf8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0703 01:39:55.587620 11392 consensus_queue.cc:962] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d277c8cd7248b59ecbaa2db3b09d5c" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 37924 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0703 01:39:55.592929 11200 raft_consensus.cc:1081] T 88835abd0ea34ff7acd90bb2c84931f5 P 6b95eacb907e4467b3633c2cc9ad055b [term 1 FOLLOWER]: Refusing update from remote peer 20e529e97c57431bb03e326238d1bf8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0703 01:39:55.594506 11396 consensus_queue.cc:962] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b95eacb907e4467b3633c2cc9ad055b" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 48168 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0703 01:39:55.600093 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 11124 I0703 01:39:55.600683 11240 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:39:55.643189 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 11124 W0703 01:39:55.651329 10890 connection.cc:512] client connection to 127.5.240.196:48168 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.196:48168): Connection reset by peer (error 104) W0703 01:39:55.651861 10697 connection.cc:512] server connection from 127.5.240.196:36458 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.196:36458): Connection reset by peer (error 104) I0703 01:39:55.652103 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 11246 I0703 01:39:55.652957 11362 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:39:55.695451 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 11246 W0703 01:39:55.703480 10888 connection.cc:512] client connection to 127.5.240.197:37924 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.197:37924): Connection reset by peer (error 104) W0703 01:39:55.703512 10696 connection.cc:512] server connection from 127.5.240.197:37129 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.197:37129): Connection reset by peer (error 104) I0703 01:39:55.705098 10956 consensus_peers.cc:455] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a -> Peer e4d277c8cd7248b59ecbaa2db3b09d5c (127.5.240.197:37924): Closing peer: e4d277c8cd7248b59ecbaa2db3b09d5c I0703 01:39:55.705179 10956 consensus_peers.cc:455] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a -> Peer 6b95eacb907e4467b3633c2cc9ad055b (127.5.240.196:48168): Closing peer: 6b95eacb907e4467b3633c2cc9ad055b I0703 01:39:55.705449 10956 consensus_queue.cc:204] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "20e529e97c57431bb03e326238d1bf8a" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 58915 } } peers { permanent_uuid: "6b95eacb907e4467b3633c2cc9ad055b" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 48168 } } W0703 01:39:55.707185 10890 consensus_peers.cc:433] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a -> Peer 6b95eacb907e4467b3633c2cc9ad055b (127.5.240.196:48168): Couldn't send request to peer 6b95eacb907e4467b3633c2cc9ad055b for tablet 88835abd0ea34ff7acd90bb2c84931f5. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.196:48168: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. W0703 01:39:56.238219 10890 consensus_peers.cc:433] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a -> Peer 6b95eacb907e4467b3633c2cc9ad055b (127.5.240.196:48168): Couldn't send request to peer 6b95eacb907e4467b3633c2cc9ad055b for tablet 88835abd0ea34ff7acd90bb2c84931f5. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.196:48168: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. W0703 01:39:56.648664 10890 consensus_peers.cc:433] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a -> Peer 6b95eacb907e4467b3633c2cc9ad055b (127.5.240.196:48168): Couldn't send request to peer 6b95eacb907e4467b3633c2cc9ad055b for tablet 88835abd0ea34ff7acd90bb2c84931f5. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.196:48168: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times. W0703 01:39:57.263288 10890 consensus_peers.cc:433] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a -> Peer 6b95eacb907e4467b3633c2cc9ad055b (127.5.240.196:48168): Couldn't send request to peer 6b95eacb907e4467b3633c2cc9ad055b for tablet 88835abd0ea34ff7acd90bb2c84931f5. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.196:48168: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 4 times. I0703 01:39:57.705626 6083 kudu-admin-test.cc:776] Change Config Op timed out, Sending a Replace config command when change config op is pending on the leader. W0703 01:39:57.809994 10890 consensus_peers.cc:433] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a -> Peer 6b95eacb907e4467b3633c2cc9ad055b (127.5.240.196:48168): Couldn't send request to peer 6b95eacb907e4467b3633c2cc9ad055b for tablet 88835abd0ea34ff7acd90bb2c84931f5. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.196:48168: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 5 times. W0703 01:39:57.917577 11413 flags.cc:406] Enabled unsafe flag: --never_fsync=true I0703 01:39:57.965971 10956 tablet_service.cc:1077] Received UnsafeChangeConfig RPC: dest_uuid: "20e529e97c57431bb03e326238d1bf8a" tablet_id: "88835abd0ea34ff7acd90bb2c84931f5" caller_id: "kudu-tools" new_config { peers { permanent_uuid: "20e529e97c57431bb03e326238d1bf8a" } } from {username='jenkins'} at 127.5.240.194:33593 W0703 01:39:57.966187 10956 raft_consensus.cc:1918] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a [term 1 LEADER]: Replica has a pending config, but the new config will be unsafely changed anyway. Currently pending config on the node: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "20e529e97c57431bb03e326238d1bf8a" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 58915 } } peers { permanent_uuid: "6b95eacb907e4467b3633c2cc9ad055b" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 48168 } } W0703 01:39:57.966467 10956 raft_consensus.cc:2017] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a [term 1 LEADER]: PROCEEDING WITH UNSAFE CONFIG CHANGE ON THIS SERVER, COMMITTED CONFIG: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20e529e97c57431bb03e326238d1bf8a" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 58915 } } peers { permanent_uuid: "6b95eacb907e4467b3633c2cc9ad055b" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 48168 } } peers { permanent_uuid: "e4d277c8cd7248b59ecbaa2db3b09d5c" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 37924 } }NEW CONFIG: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "20e529e97c57431bb03e326238d1bf8a" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 58915 } } unsafe_config_change: true I0703 01:39:57.966826 10956 raft_consensus.cc:2777] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a [term 1 LEADER]: Stepping down as leader of term 1 I0703 01:39:57.966874 10956 raft_consensus.cc:628] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 20e529e97c57431bb03e326238d1bf8a, State: Running, Role: LEADER I0703 01:39:57.967104 10956 consensus_queue.cc:227] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "20e529e97c57431bb03e326238d1bf8a" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 58915 } } peers { permanent_uuid: "6b95eacb907e4467b3633c2cc9ad055b" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 48168 } } I0703 01:39:57.967224 10956 consensus_peers.cc:455] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a -> Peer 6b95eacb907e4467b3633c2cc9ad055b (127.5.240.196:48168): Closing peer: 6b95eacb907e4467b3633c2cc9ad055b I0703 01:39:57.967334 10956 raft_consensus.cc:2782] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a [term 1 FOLLOWER]: Advancing to term 2 I0703 01:39:57.969475 10956 raft_consensus.cc:2849] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a [term 2 FOLLOWER]: Allowing unsafe config change even though there is a pending config! Existing pending config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "20e529e97c57431bb03e326238d1bf8a" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 58915 } } peers { permanent_uuid: "6b95eacb907e4467b3633c2cc9ad055b" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 48168 } }; New pending config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "20e529e97c57431bb03e326238d1bf8a" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 58915 } } unsafe_config_change: true I0703 01:39:58.012956 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master pid 10687 I0703 01:39:58.013869 10743 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:39:58.075881 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master with pid 10687 W0703 01:39:58.088559 10888 connection.cc:512] client connection to 127.5.240.254:51485 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.254:51485): Connection reset by peer (error 104) W0703 01:39:58.088537 10765 connection.cc:512] client connection to 127.5.240.254:51485 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.254:51485): Connection reset by peer (error 104) W0703 01:39:58.088562 11010 connection.cc:512] client connection to 127.5.240.254:51485 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.254:51485): Connection reset by peer (error 104) I0703 01:39:58.090260 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:51485 --webserver_interface=127.5.240.254 --webserver_port=57609 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:39:58.270741 11431 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:39:58.270968 11431 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:39:58.273186 11431 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0703 01:39:58.273231 11431 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:39:58.273275 11431 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0703 01:39:58.273321 11431 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0703 01:39:58.281751 11431 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:51485 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.5.240.254 --webserver_port=57609 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:39:58.287371 11431 master_main.cc:80] Initializing master server... I0703 01:39:58.288437 11431 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 152708us I0703 01:39:58.289510 11431 fs_manager.cc:261] Metadata directory not provided I0703 01:39:58.289597 11431 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:39:58.297073 11431 fs_manager.cc:397] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s I0703 01:39:58.297906 11431 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:39:58.298296 11431 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:39:58.301980 11431 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s I0703 01:39:58.302052 11431 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal uuid: "bf659d42f827436f97ca8c0826afc8b7" format_stamp: "Formatted at 2018-07-03 08:39:53 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:39:58.302932 11431 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:39:58.349166 11431 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:39:58.350674 11431 master_main.cc:83] Starting Master server... W0703 01:39:58.351716 11121 heartbeater.cc:587] Failed to heartbeat to 127.5.240.254:51485: Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.240.254:51485: connect: Connection refused (error 111) I0703 01:39:58.388538 11431 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.254:51485 I0703 01:39:58.389140 11431 webserver.cc:175] Starting webserver on 127.5.240.254:57609 I0703 01:39:58.389189 11431 webserver.cc:186] Document root disabled I0703 01:39:58.389909 11431 webserver.cc:313] Webserver started. Bound to: http://127.5.240.254:57609/ I0703 01:39:58.390605 11431 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb I0703 01:39:58.391178 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master as pid 11431 I0703 01:39:58.400252 11491 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P bf659d42f827436f97ca8c0826afc8b7: Bootstrap starting. I0703 01:39:58.412866 11491 log.cc:525] T 00000000000000000000000000000000 P bf659d42f827436f97ca8c0826afc8b7: Log is configured to *not* fsync() on all Append() calls I0703 01:39:58.421937 11491 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P bf659d42f827436f97ca8c0826afc8b7: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=1} inserts{seen=4 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:39:58.422338 11491 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P bf659d42f827436f97ca8c0826afc8b7: Bootstrap complete. I0703 01:39:58.431457 11491 raft_consensus.cc:325] T 00000000000000000000000000000000 P bf659d42f827436f97ca8c0826afc8b7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf659d42f827436f97ca8c0826afc8b7" member_type: VOTER } I0703 01:39:58.432188 11491 raft_consensus.cc:628] T 00000000000000000000000000000000 P bf659d42f827436f97ca8c0826afc8b7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf659d42f827436f97ca8c0826afc8b7, State: Initialized, Role: FOLLOWER I0703 01:39:58.432704 11491 consensus_queue.cc:227] T 00000000000000000000000000000000 P bf659d42f827436f97ca8c0826afc8b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf659d42f827436f97ca8c0826afc8b7" member_type: VOTER } I0703 01:39:58.432881 11491 raft_consensus.cc:365] T 00000000000000000000000000000000 P bf659d42f827436f97ca8c0826afc8b7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:39:58.432956 11491 raft_consensus.cc:457] T 00000000000000000000000000000000 P bf659d42f827436f97ca8c0826afc8b7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:39:58.433033 11491 raft_consensus.cc:2730] T 00000000000000000000000000000000 P bf659d42f827436f97ca8c0826afc8b7: Snoozing failure detection for 2.214s (starting election) I0703 01:39:58.433115 11491 raft_consensus.cc:2782] T 00000000000000000000000000000000 P bf659d42f827436f97ca8c0826afc8b7 [term 1 FOLLOWER]: Advancing to term 2 I0703 01:39:58.436486 11491 raft_consensus.cc:479] T 00000000000000000000000000000000 P bf659d42f827436f97ca8c0826afc8b7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf659d42f827436f97ca8c0826afc8b7" member_type: VOTER } I0703 01:39:58.436848 11491 leader_election.cc:270] T 00000000000000000000000000000000 P bf659d42f827436f97ca8c0826afc8b7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0703 01:39:58.438151 11495 raft_consensus.cc:2730] T 00000000000000000000000000000000 P bf659d42f827436f97ca8c0826afc8b7: Snoozing failure detection for 2.198s (election complete) I0703 01:39:58.438555 11495 raft_consensus.cc:2537] T 00000000000000000000000000000000 P bf659d42f827436f97ca8c0826afc8b7 [term 2 FOLLOWER]: Leader election won for term 2 I0703 01:39:58.439946 11495 raft_consensus.cc:593] T 00000000000000000000000000000000 P bf659d42f827436f97ca8c0826afc8b7 [term 2 LEADER]: Becoming Leader. State: Replica: bf659d42f827436f97ca8c0826afc8b7, State: Running, Role: LEADER I0703 01:39:58.440354 11495 consensus_queue.cc:204] T 00000000000000000000000000000000 P bf659d42f827436f97ca8c0826afc8b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf659d42f827436f97ca8c0826afc8b7" member_type: VOTER } I0703 01:39:58.441884 11491 sys_catalog.cc:337] T 00000000000000000000000000000000 P bf659d42f827436f97ca8c0826afc8b7 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 2 leader_uuid: "bf659d42f827436f97ca8c0826afc8b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf659d42f827436f97ca8c0826afc8b7" member_type: VOTER } } I0703 01:39:58.442482 11491 sys_catalog.cc:340] T 00000000000000000000000000000000 P bf659d42f827436f97ca8c0826afc8b7 [sys.catalog]: This master's current role is: LEADER I0703 01:39:58.442162 11497 sys_catalog.cc:337] T 00000000000000000000000000000000 P bf659d42f827436f97ca8c0826afc8b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf659d42f827436f97ca8c0826afc8b7. Latest consensus state: current_term: 2 leader_uuid: "bf659d42f827436f97ca8c0826afc8b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf659d42f827436f97ca8c0826afc8b7" member_type: VOTER } } I0703 01:39:58.442085 11496 sys_catalog.cc:337] T 00000000000000000000000000000000 P bf659d42f827436f97ca8c0826afc8b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "bf659d42f827436f97ca8c0826afc8b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf659d42f827436f97ca8c0826afc8b7" member_type: VOTER } } I0703 01:39:58.443370 11497 sys_catalog.cc:340] T 00000000000000000000000000000000 P bf659d42f827436f97ca8c0826afc8b7 [sys.catalog]: This master's current role is: LEADER I0703 01:39:58.443370 11496 sys_catalog.cc:340] T 00000000000000000000000000000000 P bf659d42f827436f97ca8c0826afc8b7 [sys.catalog]: This master's current role is: LEADER I0703 01:39:58.445317 11491 sys_catalog.cc:424] T 00000000000000000000000000000000 P bf659d42f827436f97ca8c0826afc8b7 [sys.catalog]: configured and running, proceeding with master startup. I0703 01:39:58.445583 11499 catalog_manager.cc:1006] Loading table and tablet metadata into memory... I0703 01:39:58.446815 11431 master_main.cc:86] Master server successfully started. I0703 01:39:58.450623 11499 catalog_manager.cc:323] Loaded metadata for table TestTable [id=20caf1ed0e1d4dd69440a2ed2a5bac36] I0703 01:39:58.452607 11499 catalog_manager.cc:377] Loaded metadata for tablet 88835abd0ea34ff7acd90bb2c84931f5 (table TestTable [id=20caf1ed0e1d4dd69440a2ed2a5bac36]) I0703 01:39:58.452761 11499 catalog_manager.cc:1017] Initializing Kudu internal certificate authority... I0703 01:39:58.455708 11499 catalog_manager.cc:1026] Loading token signing keys... I0703 01:39:58.456830 11499 catalog_manager.cc:4113] T 00000000000000000000000000000000 P bf659d42f827436f97ca8c0826afc8b7: Loaded TSK: 0 I0703 01:39:58.720557 11458 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "200046fa8cd344f4a557b6feab568415" instance_seqno: 1530607193658994) as {username='jenkins'} at 127.5.240.193:55936; Asking this server to re-register. I0703 01:39:58.721648 10876 heartbeater.cc:426] Registering TS with master... I0703 01:39:58.721788 10876 heartbeater.cc:475] Master 127.5.240.254:51485 requested a full tablet report, sending... I0703 01:39:58.722798 11458 ts_manager.cc:98] Registered new tserver with Master: 200046fa8cd344f4a557b6feab568415 (127.5.240.193:48069) I0703 01:39:58.976537 11458 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "20e529e97c57431bb03e326238d1bf8a" instance_seqno: 1530607193971988) as {username='jenkins'} at 127.5.240.194:54612; Asking this server to re-register. I0703 01:39:58.977169 10999 heartbeater.cc:426] Registering TS with master... I0703 01:39:58.977293 10999 heartbeater.cc:475] Master 127.5.240.254:51485 requested a full tablet report, sending... I0703 01:39:58.978206 11458 ts_manager.cc:98] Registered new tserver with Master: 20e529e97c57431bb03e326238d1bf8a (127.5.240.194:58915) I0703 01:39:59.357276 11121 heartbeater.cc:346] Connected to a master server at 127.5.240.254:51485 I0703 01:39:59.357977 11458 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "bb2c63b1e9fa4dc3aa89afac6501f1ff" instance_seqno: 1530607194307802) as {username='jenkins'} at 127.5.240.195:40313; Asking this server to re-register. I0703 01:39:59.358533 11121 heartbeater.cc:426] Registering TS with master... I0703 01:39:59.358649 11121 heartbeater.cc:475] Master 127.5.240.254:51485 requested a full tablet report, sending... I0703 01:39:59.359328 11458 ts_manager.cc:98] Registered new tserver with Master: bb2c63b1e9fa4dc3aa89afac6501f1ff (127.5.240.195:34920) I0703 01:39:59.470518 11508 raft_consensus.cc:457] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader kudu-tools) I0703 01:39:59.470626 11508 raft_consensus.cc:2730] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a: Snoozing failure detection for 1.938s (starting election) I0703 01:39:59.470701 11508 raft_consensus.cc:479] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "20e529e97c57431bb03e326238d1bf8a" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 58915 } } unsafe_config_change: true I0703 01:39:59.470893 11508 leader_election.cc:270] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. I0703 01:39:59.471276 11508 raft_consensus.cc:2730] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a: Snoozing failure detection for 1.586s (election complete) I0703 01:39:59.471328 11508 raft_consensus.cc:2537] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a [term 2 FOLLOWER]: Leader pre-election won for term 3 I0703 01:39:59.471365 11508 raft_consensus.cc:457] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a [term 2 FOLLOWER]: Starting leader election (detected failure of leader kudu-tools) I0703 01:39:59.471397 11508 raft_consensus.cc:2730] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a: Snoozing failure detection for 1.579s (starting election) I0703 01:39:59.471437 11508 raft_consensus.cc:2782] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a [term 2 FOLLOWER]: Advancing to term 3 I0703 01:39:59.472846 11508 raft_consensus.cc:479] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "20e529e97c57431bb03e326238d1bf8a" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 58915 } } unsafe_config_change: true I0703 01:39:59.472987 11508 leader_election.cc:270] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. I0703 01:39:59.473150 11509 raft_consensus.cc:2730] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a: Snoozing failure detection for 1.821s (election complete) I0703 01:39:59.473220 11509 raft_consensus.cc:2537] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a [term 3 FOLLOWER]: Leader election won for term 3 I0703 01:39:59.473361 11509 raft_consensus.cc:593] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a [term 3 LEADER]: Becoming Leader. State: Replica: 20e529e97c57431bb03e326238d1bf8a, State: Running, Role: LEADER I0703 01:39:59.473680 11509 consensus_queue.cc:204] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.3, Last appended by leader: 0, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "20e529e97c57431bb03e326238d1bf8a" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 58915 } } unsafe_config_change: true I0703 01:39:59.475473 11458 catalog_manager.cc:3798] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a reported cstate change: term changed from 1 to 3, now has a pending config: VOTER 20e529e97c57431bb03e326238d1bf8a (127.5.240.194). New cstate: current_term: 3 leader_uuid: "20e529e97c57431bb03e326238d1bf8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20e529e97c57431bb03e326238d1bf8a" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 58915 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b95eacb907e4467b3633c2cc9ad055b" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 48168 } } peers { permanent_uuid: "e4d277c8cd7248b59ecbaa2db3b09d5c" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 37924 } } } pending_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "20e529e97c57431bb03e326238d1bf8a" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 58915 } } unsafe_config_change: true } I0703 01:39:59.474920 11509 raft_consensus.cc:2684] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a [term 3 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e4d277c8cd7248b59ecbaa2db3b09d5c (127.5.240.197) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "20e529e97c57431bb03e326238d1bf8a" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 58915 } } peers { permanent_uuid: "6b95eacb907e4467b3633c2cc9ad055b" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 48168 } } } W0703 01:39:59.479416 11509 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.ChangeConfig from 127.5.240.194:33591 (request call id 42) took 3775ms (client timeout 2000). W0703 01:39:59.480216 11509 rpcz_store.cc:255] Trace: 0703 01:39:55.704287 (+ 0us) service_pool.cc:162] Inserting onto call queue 0703 01:39:55.704337 (+ 50us) service_pool.cc:221] Handling call 0703 01:39:55.706037 (+ 1700us) log.cc:587] Serialized 379 byte log entry 0703 01:39:59.479338 (+3773301us) inbound_call.cc:162] Queueing success response Metrics: {"dns_us":63,"raft.queue_time_us":98,"raft.run_cpu_time_us":117,"raft.run_wall_time_us":118} I0703 01:39:59.480621 11509 raft_consensus.cc:2684] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a [term 3 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER 6b95eacb907e4467b3633c2cc9ad055b (127.5.240.196) evicted, VOTER e4d277c8cd7248b59ecbaa2db3b09d5c (127.5.240.197) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "20e529e97c57431bb03e326238d1bf8a" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 58915 } } unsafe_config_change: true } I0703 01:39:59.486734 11458 catalog_manager.cc:3798] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a reported cstate change: config changed from index -1 to 3, VOTER 6b95eacb907e4467b3633c2cc9ad055b (127.5.240.196) evicted, VOTER e4d277c8cd7248b59ecbaa2db3b09d5c (127.5.240.197) evicted, no longer has a pending config: VOTER 20e529e97c57431bb03e326238d1bf8a (127.5.240.194). New cstate: current_term: 3 leader_uuid: "20e529e97c57431bb03e326238d1bf8a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "20e529e97c57431bb03e326238d1bf8a" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 58915 } health_report { overall_health: HEALTHY } } unsafe_config_change: true } I0703 01:39:59.488726 11458 catalog_manager.cc:2903] Scheduling retry of 88835abd0ea34ff7acd90bb2c84931f5 Delete Tablet RPC for TS=6b95eacb907e4467b3633c2cc9ad055b with a delay of 26 ms (attempt = 1) W0703 01:39:59.488823 11458 catalog_manager.cc:3957] Failed to send 88835abd0ea34ff7acd90bb2c84931f5 Delete Tablet RPC for TS=6b95eacb907e4467b3633c2cc9ad055b: Not found: failed to reset TS proxy: Could not find TS for UUID 6b95eacb907e4467b3633c2cc9ad055b I0703 01:39:59.488943 11458 catalog_manager.cc:2903] Scheduling retry of 88835abd0ea34ff7acd90bb2c84931f5 Delete Tablet RPC for TS=e4d277c8cd7248b59ecbaa2db3b09d5c with a delay of 21 ms (attempt = 1) W0703 01:39:59.489002 11458 catalog_manager.cc:3957] Failed to send 88835abd0ea34ff7acd90bb2c84931f5 Delete Tablet RPC for TS=e4d277c8cd7248b59ecbaa2db3b09d5c: Not found: failed to reset TS proxy: Could not find TS for UUID e4d277c8cd7248b59ecbaa2db3b09d5c I0703 01:39:59.489373 11458 catalog_manager.cc:3437] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 88835abd0ea34ff7acd90bb2c84931f5 (attempt 1) I0703 01:39:59.494587 10956 consensus_queue.cc:204] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 0, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20e529e97c57431bb03e326238d1bf8a" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 58915 } } peers { permanent_uuid: "200046fa8cd344f4a557b6feab568415" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 48069 } attrs { promote: true } } unsafe_config_change: true I0703 01:39:59.495365 11508 raft_consensus.cc:2684] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 3 to 5, NON_VOTER 200046fa8cd344f4a557b6feab568415 (127.5.240.193) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20e529e97c57431bb03e326238d1bf8a" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 58915 } } peers { permanent_uuid: "200046fa8cd344f4a557b6feab568415" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 48069 } attrs { promote: true } } unsafe_config_change: true } I0703 01:39:59.498603 11440 catalog_manager.cc:3352] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 88835abd0ea34ff7acd90bb2c84931f5 with cas_config_opid_index 3: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0703 01:39:59.499338 11458 catalog_manager.cc:3798] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a reported cstate change: config changed from index 3 to 5, NON_VOTER 200046fa8cd344f4a557b6feab568415 (127.5.240.193) added. New cstate: current_term: 3 leader_uuid: "20e529e97c57431bb03e326238d1bf8a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20e529e97c57431bb03e326238d1bf8a" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 58915 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "200046fa8cd344f4a557b6feab568415" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 48069 } attrs { promote: true } health_report { overall_health: UNKNOWN } } unsafe_config_change: true } I0703 01:39:59.501212 11458 catalog_manager.cc:3437] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 88835abd0ea34ff7acd90bb2c84931f5 (attempt 1) I0703 01:39:59.501889 10956 consensus_peers.cc:455] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a -> Peer 200046fa8cd344f4a557b6feab568415 (127.5.240.193:48069): Closing peer: 200046fa8cd344f4a557b6feab568415 I0703 01:39:59.502050 10956 consensus_queue.cc:204] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 0, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "20e529e97c57431bb03e326238d1bf8a" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 58915 } } peers { permanent_uuid: "200046fa8cd344f4a557b6feab568415" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 48069 } attrs { promote: true } } peers { permanent_uuid: "bb2c63b1e9fa4dc3aa89afac6501f1ff" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 34920 } attrs { promote: true } } unsafe_config_change: true I0703 01:39:59.504098 11508 raft_consensus.cc:2684] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a [term 3 LEADER]: Committing config change with OpId 3.6: config changed from index 5 to 6, NON_VOTER bb2c63b1e9fa4dc3aa89afac6501f1ff (127.5.240.195) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "20e529e97c57431bb03e326238d1bf8a" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 58915 } } peers { permanent_uuid: "200046fa8cd344f4a557b6feab568415" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 48069 } attrs { promote: true } } peers { permanent_uuid: "bb2c63b1e9fa4dc3aa89afac6501f1ff" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 34920 } attrs { promote: true } } unsafe_config_change: true } W0703 01:39:59.505123 10887 consensus_peers.cc:433] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a -> Peer 200046fa8cd344f4a557b6feab568415 (127.5.240.193:48069): Couldn't send request to peer 200046fa8cd344f4a557b6feab568415 for tablet 88835abd0ea34ff7acd90bb2c84931f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 88835abd0ea34ff7acd90bb2c84931f5. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:39:59.506753 11440 catalog_manager.cc:3352] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 88835abd0ea34ff7acd90bb2c84931f5 with cas_config_opid_index 5: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0703 01:39:59.507637 11458 catalog_manager.cc:3798] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a reported cstate change: config changed from index 5 to 6, NON_VOTER bb2c63b1e9fa4dc3aa89afac6501f1ff (127.5.240.195) added. New cstate: current_term: 3 leader_uuid: "20e529e97c57431bb03e326238d1bf8a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "20e529e97c57431bb03e326238d1bf8a" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 58915 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "200046fa8cd344f4a557b6feab568415" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 48069 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb2c63b1e9fa4dc3aa89afac6501f1ff" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 34920 } attrs { promote: true } health_report { overall_health: UNKNOWN } } unsafe_config_change: true } W0703 01:39:59.511193 10890 consensus_peers.cc:433] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a -> Peer bb2c63b1e9fa4dc3aa89afac6501f1ff (127.5.240.195:34920): Couldn't send request to peer bb2c63b1e9fa4dc3aa89afac6501f1ff for tablet 88835abd0ea34ff7acd90bb2c84931f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 88835abd0ea34ff7acd90bb2c84931f5. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:39:59.511241 11440 catalog_manager.cc:2903] Scheduling retry of 88835abd0ea34ff7acd90bb2c84931f5 Delete Tablet RPC for TS=e4d277c8cd7248b59ecbaa2db3b09d5c with a delay of 50 ms (attempt = 2) W0703 01:39:59.511313 11440 catalog_manager.cc:2922] Async tablet task 88835abd0ea34ff7acd90bb2c84931f5 Delete Tablet RPC for TS=e4d277c8cd7248b59ecbaa2db3b09d5c failed: Not found: failed to reset TS proxy: Could not find TS for UUID e4d277c8cd7248b59ecbaa2db3b09d5c I0703 01:39:59.515512 11438 catalog_manager.cc:2903] Scheduling retry of 88835abd0ea34ff7acd90bb2c84931f5 Delete Tablet RPC for TS=6b95eacb907e4467b3633c2cc9ad055b with a delay of 43 ms (attempt = 2) W0703 01:39:59.515574 11438 catalog_manager.cc:2922] Async tablet task 88835abd0ea34ff7acd90bb2c84931f5 Delete Tablet RPC for TS=6b95eacb907e4467b3633c2cc9ad055b failed: Not found: failed to reset TS proxy: Could not find TS for UUID 6b95eacb907e4467b3633c2cc9ad055b I0703 01:39:59.558955 11438 catalog_manager.cc:2903] Scheduling retry of 88835abd0ea34ff7acd90bb2c84931f5 Delete Tablet RPC for TS=6b95eacb907e4467b3633c2cc9ad055b with a delay of 106 ms (attempt = 3) W0703 01:39:59.559001 11438 catalog_manager.cc:2922] Async tablet task 88835abd0ea34ff7acd90bb2c84931f5 Delete Tablet RPC for TS=6b95eacb907e4467b3633c2cc9ad055b failed: Not found: failed to reset TS proxy: Could not find TS for UUID 6b95eacb907e4467b3633c2cc9ad055b I0703 01:39:59.561636 11440 catalog_manager.cc:2903] Scheduling retry of 88835abd0ea34ff7acd90bb2c84931f5 Delete Tablet RPC for TS=e4d277c8cd7248b59ecbaa2db3b09d5c with a delay of 96 ms (attempt = 3) W0703 01:39:59.561683 11440 catalog_manager.cc:2922] Async tablet task 88835abd0ea34ff7acd90bb2c84931f5 Delete Tablet RPC for TS=e4d277c8cd7248b59ecbaa2db3b09d5c failed: Not found: failed to reset TS proxy: Could not find TS for UUID e4d277c8cd7248b59ecbaa2db3b09d5c I0703 01:39:59.658388 11440 catalog_manager.cc:2903] Scheduling retry of 88835abd0ea34ff7acd90bb2c84931f5 Delete Tablet RPC for TS=e4d277c8cd7248b59ecbaa2db3b09d5c with a delay of 174 ms (attempt = 4) W0703 01:39:59.658464 11440 catalog_manager.cc:2922] Async tablet task 88835abd0ea34ff7acd90bb2c84931f5 Delete Tablet RPC for TS=e4d277c8cd7248b59ecbaa2db3b09d5c failed: Not found: failed to reset TS proxy: Could not find TS for UUID e4d277c8cd7248b59ecbaa2db3b09d5c I0703 01:39:59.665614 11438 catalog_manager.cc:2903] Scheduling retry of 88835abd0ea34ff7acd90bb2c84931f5 Delete Tablet RPC for TS=6b95eacb907e4467b3633c2cc9ad055b with a delay of 157 ms (attempt = 4) W0703 01:39:59.665658 11438 catalog_manager.cc:2922] Async tablet task 88835abd0ea34ff7acd90bb2c84931f5 Delete Tablet RPC for TS=6b95eacb907e4467b3633c2cc9ad055b failed: Not found: failed to reset TS proxy: Could not find TS for UUID 6b95eacb907e4467b3633c2cc9ad055b I0703 01:39:59.823711 11438 catalog_manager.cc:2903] Scheduling retry of 88835abd0ea34ff7acd90bb2c84931f5 Delete Tablet RPC for TS=6b95eacb907e4467b3633c2cc9ad055b with a delay of 277 ms (attempt = 5) W0703 01:39:59.823760 11438 catalog_manager.cc:2922] Async tablet task 88835abd0ea34ff7acd90bb2c84931f5 Delete Tablet RPC for TS=6b95eacb907e4467b3633c2cc9ad055b failed: Not found: failed to reset TS proxy: Could not find TS for UUID 6b95eacb907e4467b3633c2cc9ad055b I0703 01:39:59.833437 11440 catalog_manager.cc:2903] Scheduling retry of 88835abd0ea34ff7acd90bb2c84931f5 Delete Tablet RPC for TS=e4d277c8cd7248b59ecbaa2db3b09d5c with a delay of 276 ms (attempt = 5) W0703 01:39:59.833482 11440 catalog_manager.cc:2922] Async tablet task 88835abd0ea34ff7acd90bb2c84931f5 Delete Tablet RPC for TS=e4d277c8cd7248b59ecbaa2db3b09d5c failed: Not found: failed to reset TS proxy: Could not find TS for UUID e4d277c8cd7248b59ecbaa2db3b09d5c I0703 01:39:59.997694 11520 ts_tablet_manager.cc:669] T 88835abd0ea34ff7acd90bb2c84931f5 P 200046fa8cd344f4a557b6feab568415: Initiating tablet copy from peer 20e529e97c57431bb03e326238d1bf8a (127.5.240.194:58915) I0703 01:39:59.997946 11520 tablet_copy_client.cc:239] T 88835abd0ea34ff7acd90bb2c84931f5 P 200046fa8cd344f4a557b6feab568415: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.194:58915 I0703 01:40:00.003975 10976 tablet_copy_service.cc:135] P 20e529e97c57431bb03e326238d1bf8a: Received BeginTabletCopySession request for tablet 88835abd0ea34ff7acd90bb2c84931f5 from peer 200046fa8cd344f4a557b6feab568415 ({username='jenkins'} at 127.5.240.193:50478) I0703 01:40:00.004106 10976 tablet_copy_service.cc:156] P 20e529e97c57431bb03e326238d1bf8a: Beginning new tablet copy session on tablet 88835abd0ea34ff7acd90bb2c84931f5 from peer 200046fa8cd344f4a557b6feab568415 at {username='jenkins'} at 127.5.240.193:50478: session id = 200046fa8cd344f4a557b6feab568415-88835abd0ea34ff7acd90bb2c84931f5 I0703 01:40:00.006012 10976 tablet_copy_source_session.cc:202] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:40:00.008766 11520 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 88835abd0ea34ff7acd90bb2c84931f5. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:40:00.018159 11520 ts_tablet_manager.cc:1173] T 88835abd0ea34ff7acd90bb2c84931f5 P 200046fa8cd344f4a557b6feab568415: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:40:00.018375 11520 tablet_copy_client.cc:566] T 88835abd0ea34ff7acd90bb2c84931f5 P 200046fa8cd344f4a557b6feab568415: tablet copy: Starting download of 0 data blocks... I0703 01:40:00.018656 11520 tablet_copy_client.cc:529] T 88835abd0ea34ff7acd90bb2c84931f5 P 200046fa8cd344f4a557b6feab568415: tablet copy: Starting download of 1 WAL segments... I0703 01:40:00.020189 11520 tablet_copy_client.cc:405] T 88835abd0ea34ff7acd90bb2c84931f5 P 200046fa8cd344f4a557b6feab568415: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:40:00.021822 11520 ts_tablet_manager.cc:1028] T 88835abd0ea34ff7acd90bb2c84931f5 P 200046fa8cd344f4a557b6feab568415: Bootstrapping tablet I0703 01:40:00.022027 11520 tablet_bootstrap.cc:427] T 88835abd0ea34ff7acd90bb2c84931f5 P 200046fa8cd344f4a557b6feab568415: Bootstrap starting. I0703 01:40:00.026355 11520 log.cc:525] T 88835abd0ea34ff7acd90bb2c84931f5 P 200046fa8cd344f4a557b6feab568415: Log is configured to *not* fsync() on all Append() calls I0703 01:40:00.030328 11520 tablet_bootstrap.cc:427] T 88835abd0ea34ff7acd90bb2c84931f5 P 200046fa8cd344f4a557b6feab568415: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:40:00.030650 11520 tablet_bootstrap.cc:427] T 88835abd0ea34ff7acd90bb2c84931f5 P 200046fa8cd344f4a557b6feab568415: Bootstrap complete. I0703 01:40:00.030846 11520 ts_tablet_manager.cc:1045] T 88835abd0ea34ff7acd90bb2c84931f5 P 200046fa8cd344f4a557b6feab568415: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s I0703 01:40:00.035552 11520 raft_consensus.cc:325] T 88835abd0ea34ff7acd90bb2c84931f5 P 200046fa8cd344f4a557b6feab568415 [term 3 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "20e529e97c57431bb03e326238d1bf8a" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 58915 } } peers { permanent_uuid: "200046fa8cd344f4a557b6feab568415" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 48069 } attrs { promote: true } } peers { permanent_uuid: "bb2c63b1e9fa4dc3aa89afac6501f1ff" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 34920 } attrs { promote: true } } unsafe_config_change: true I0703 01:40:00.035847 11520 raft_consensus.cc:628] T 88835abd0ea34ff7acd90bb2c84931f5 P 200046fa8cd344f4a557b6feab568415 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 200046fa8cd344f4a557b6feab568415, State: Initialized, Role: LEARNER I0703 01:40:00.036074 11520 consensus_queue.cc:227] T 88835abd0ea34ff7acd90bb2c84931f5 P 200046fa8cd344f4a557b6feab568415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "20e529e97c57431bb03e326238d1bf8a" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 58915 } } peers { permanent_uuid: "200046fa8cd344f4a557b6feab568415" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 48069 } attrs { promote: true } } peers { permanent_uuid: "bb2c63b1e9fa4dc3aa89afac6501f1ff" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 34920 } attrs { promote: true } } unsafe_config_change: true I0703 01:40:00.037762 11520 ts_tablet_manager.cc:1073] T 88835abd0ea34ff7acd90bb2c84931f5 P 200046fa8cd344f4a557b6feab568415: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s I0703 01:40:00.038476 10976 tablet_copy_service.cc:325] P 20e529e97c57431bb03e326238d1bf8a: Request end of tablet copy session 200046fa8cd344f4a557b6feab568415-88835abd0ea34ff7acd90bb2c84931f5 received from {username='jenkins'} at 127.5.240.193:50478 I0703 01:40:00.038564 10976 tablet_copy_service.cc:413] P 20e529e97c57431bb03e326238d1bf8a: ending tablet copy session 200046fa8cd344f4a557b6feab568415-88835abd0ea34ff7acd90bb2c84931f5 on tablet 88835abd0ea34ff7acd90bb2c84931f5 with peer 200046fa8cd344f4a557b6feab568415 I0703 01:40:00.101559 11438 catalog_manager.cc:2903] Scheduling retry of 88835abd0ea34ff7acd90bb2c84931f5 Delete Tablet RPC for TS=6b95eacb907e4467b3633c2cc9ad055b with a delay of 546 ms (attempt = 6) W0703 01:40:00.101622 11438 catalog_manager.cc:2922] Async tablet task 88835abd0ea34ff7acd90bb2c84931f5 Delete Tablet RPC for TS=6b95eacb907e4467b3633c2cc9ad055b failed: Not found: failed to reset TS proxy: Could not find TS for UUID 6b95eacb907e4467b3633c2cc9ad055b I0703 01:40:00.110510 11440 catalog_manager.cc:2903] Scheduling retry of 88835abd0ea34ff7acd90bb2c84931f5 Delete Tablet RPC for TS=e4d277c8cd7248b59ecbaa2db3b09d5c with a delay of 534 ms (attempt = 6) W0703 01:40:00.110574 11440 catalog_manager.cc:2922] Async tablet task 88835abd0ea34ff7acd90bb2c84931f5 Delete Tablet RPC for TS=e4d277c8cd7248b59ecbaa2db3b09d5c failed: Not found: failed to reset TS proxy: Could not find TS for UUID e4d277c8cd7248b59ecbaa2db3b09d5c I0703 01:40:00.122288 11525 ts_tablet_manager.cc:669] T 88835abd0ea34ff7acd90bb2c84931f5 P bb2c63b1e9fa4dc3aa89afac6501f1ff: Initiating tablet copy from peer 20e529e97c57431bb03e326238d1bf8a (127.5.240.194:58915) I0703 01:40:00.122525 11525 tablet_copy_client.cc:239] T 88835abd0ea34ff7acd90bb2c84931f5 P bb2c63b1e9fa4dc3aa89afac6501f1ff: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.194:58915 I0703 01:40:00.127207 10976 tablet_copy_service.cc:135] P 20e529e97c57431bb03e326238d1bf8a: Received BeginTabletCopySession request for tablet 88835abd0ea34ff7acd90bb2c84931f5 from peer bb2c63b1e9fa4dc3aa89afac6501f1ff ({username='jenkins'} at 127.5.240.195:60127) I0703 01:40:00.127296 10976 tablet_copy_service.cc:156] P 20e529e97c57431bb03e326238d1bf8a: Beginning new tablet copy session on tablet 88835abd0ea34ff7acd90bb2c84931f5 from peer bb2c63b1e9fa4dc3aa89afac6501f1ff at {username='jenkins'} at 127.5.240.195:60127: session id = bb2c63b1e9fa4dc3aa89afac6501f1ff-88835abd0ea34ff7acd90bb2c84931f5 I0703 01:40:00.128886 10976 tablet_copy_source_session.cc:202] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:40:00.131310 11525 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 88835abd0ea34ff7acd90bb2c84931f5. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:40:00.140682 11525 ts_tablet_manager.cc:1173] T 88835abd0ea34ff7acd90bb2c84931f5 P bb2c63b1e9fa4dc3aa89afac6501f1ff: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:40:00.140944 11525 tablet_copy_client.cc:566] T 88835abd0ea34ff7acd90bb2c84931f5 P bb2c63b1e9fa4dc3aa89afac6501f1ff: tablet copy: Starting download of 0 data blocks... I0703 01:40:00.141182 11525 tablet_copy_client.cc:529] T 88835abd0ea34ff7acd90bb2c84931f5 P bb2c63b1e9fa4dc3aa89afac6501f1ff: tablet copy: Starting download of 1 WAL segments... I0703 01:40:00.142590 11525 tablet_copy_client.cc:405] T 88835abd0ea34ff7acd90bb2c84931f5 P bb2c63b1e9fa4dc3aa89afac6501f1ff: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:40:00.144217 11525 ts_tablet_manager.cc:1028] T 88835abd0ea34ff7acd90bb2c84931f5 P bb2c63b1e9fa4dc3aa89afac6501f1ff: Bootstrapping tablet I0703 01:40:00.144434 11525 tablet_bootstrap.cc:427] T 88835abd0ea34ff7acd90bb2c84931f5 P bb2c63b1e9fa4dc3aa89afac6501f1ff: Bootstrap starting. I0703 01:40:00.148882 11525 log.cc:525] T 88835abd0ea34ff7acd90bb2c84931f5 P bb2c63b1e9fa4dc3aa89afac6501f1ff: Log is configured to *not* fsync() on all Append() calls I0703 01:40:00.153000 11525 tablet_bootstrap.cc:427] T 88835abd0ea34ff7acd90bb2c84931f5 P bb2c63b1e9fa4dc3aa89afac6501f1ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:40:00.153329 11525 tablet_bootstrap.cc:427] T 88835abd0ea34ff7acd90bb2c84931f5 P bb2c63b1e9fa4dc3aa89afac6501f1ff: Bootstrap complete. I0703 01:40:00.153538 11525 ts_tablet_manager.cc:1045] T 88835abd0ea34ff7acd90bb2c84931f5 P bb2c63b1e9fa4dc3aa89afac6501f1ff: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s I0703 01:40:00.158257 11525 raft_consensus.cc:325] T 88835abd0ea34ff7acd90bb2c84931f5 P bb2c63b1e9fa4dc3aa89afac6501f1ff [term 3 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "20e529e97c57431bb03e326238d1bf8a" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 58915 } } peers { permanent_uuid: "200046fa8cd344f4a557b6feab568415" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 48069 } attrs { promote: true } } peers { permanent_uuid: "bb2c63b1e9fa4dc3aa89afac6501f1ff" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 34920 } attrs { promote: true } } unsafe_config_change: true I0703 01:40:00.158550 11525 raft_consensus.cc:628] T 88835abd0ea34ff7acd90bb2c84931f5 P bb2c63b1e9fa4dc3aa89afac6501f1ff [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: bb2c63b1e9fa4dc3aa89afac6501f1ff, State: Initialized, Role: LEARNER I0703 01:40:00.158820 11525 consensus_queue.cc:227] T 88835abd0ea34ff7acd90bb2c84931f5 P bb2c63b1e9fa4dc3aa89afac6501f1ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "20e529e97c57431bb03e326238d1bf8a" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 58915 } } peers { permanent_uuid: "200046fa8cd344f4a557b6feab568415" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 48069 } attrs { promote: true } } peers { permanent_uuid: "bb2c63b1e9fa4dc3aa89afac6501f1ff" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 34920 } attrs { promote: true } } unsafe_config_change: true I0703 01:40:00.160164 11525 ts_tablet_manager.cc:1073] T 88835abd0ea34ff7acd90bb2c84931f5 P bb2c63b1e9fa4dc3aa89afac6501f1ff: Time spent starting tablet: real 0.007s user 0.005s sys 0.003s I0703 01:40:00.160910 10976 tablet_copy_service.cc:325] P 20e529e97c57431bb03e326238d1bf8a: Request end of tablet copy session bb2c63b1e9fa4dc3aa89afac6501f1ff-88835abd0ea34ff7acd90bb2c84931f5 received from {username='jenkins'} at 127.5.240.195:60127 I0703 01:40:00.160985 10976 tablet_copy_service.cc:413] P 20e529e97c57431bb03e326238d1bf8a: ending tablet copy session bb2c63b1e9fa4dc3aa89afac6501f1ff-88835abd0ea34ff7acd90bb2c84931f5 on tablet 88835abd0ea34ff7acd90bb2c84931f5 with peer bb2c63b1e9fa4dc3aa89afac6501f1ff W0703 01:40:00.183961 10877 tablet.cc:1678] T 88835abd0ea34ff7acd90bb2c84931f5 P 200046fa8cd344f4a557b6feab568415: Can't schedule compaction. Clean time has not been advanced past its initial value. W0703 01:40:00.332026 11122 tablet.cc:1678] T 88835abd0ea34ff7acd90bb2c84931f5 P bb2c63b1e9fa4dc3aa89afac6501f1ff: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:40:00.395290 10833 raft_consensus.cc:1026] T 88835abd0ea34ff7acd90bb2c84931f5 P 200046fa8cd344f4a557b6feab568415 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.5->[3.6-3.6] Dedup: 3.6->[] I0703 01:40:00.595854 11078 raft_consensus.cc:1026] T 88835abd0ea34ff7acd90bb2c84931f5 P bb2c63b1e9fa4dc3aa89afac6501f1ff [term 3 LEARNER]: Deduplicated request from leader. Original: 3.5->[3.6-3.6] Dedup: 3.6->[] I0703 01:40:00.645292 11440 catalog_manager.cc:2903] Scheduling retry of 88835abd0ea34ff7acd90bb2c84931f5 Delete Tablet RPC for TS=e4d277c8cd7248b59ecbaa2db3b09d5c with a delay of 1051 ms (attempt = 7) W0703 01:40:00.645365 11440 catalog_manager.cc:2922] Async tablet task 88835abd0ea34ff7acd90bb2c84931f5 Delete Tablet RPC for TS=e4d277c8cd7248b59ecbaa2db3b09d5c failed: Not found: failed to reset TS proxy: Could not find TS for UUID e4d277c8cd7248b59ecbaa2db3b09d5c I0703 01:40:00.648331 11438 catalog_manager.cc:2903] Scheduling retry of 88835abd0ea34ff7acd90bb2c84931f5 Delete Tablet RPC for TS=6b95eacb907e4467b3633c2cc9ad055b with a delay of 1058 ms (attempt = 7) W0703 01:40:00.648382 11438 catalog_manager.cc:2922] Async tablet task 88835abd0ea34ff7acd90bb2c84931f5 Delete Tablet RPC for TS=6b95eacb907e4467b3633c2cc9ad055b failed: Not found: failed to reset TS proxy: Could not find TS for UUID 6b95eacb907e4467b3633c2cc9ad055b I0703 01:40:00.847755 11530 raft_consensus.cc:890] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a: attempting to promote NON_VOTER 200046fa8cd344f4a557b6feab568415 to VOTER I0703 01:40:00.848078 11530 consensus_peers.cc:455] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a -> Peer bb2c63b1e9fa4dc3aa89afac6501f1ff (127.5.240.195:34920): Closing peer: bb2c63b1e9fa4dc3aa89afac6501f1ff I0703 01:40:00.848146 11530 consensus_peers.cc:455] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a -> Peer 200046fa8cd344f4a557b6feab568415 (127.5.240.193:48069): Closing peer: 200046fa8cd344f4a557b6feab568415 I0703 01:40:00.848359 11530 consensus_queue.cc:204] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 0, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "20e529e97c57431bb03e326238d1bf8a" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 58915 } } peers { permanent_uuid: "200046fa8cd344f4a557b6feab568415" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 48069 } attrs { promote: false } } peers { permanent_uuid: "bb2c63b1e9fa4dc3aa89afac6501f1ff" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 34920 } attrs { promote: true } } unsafe_config_change: true I0703 01:40:00.850356 11078 raft_consensus.cc:1081] T 88835abd0ea34ff7acd90bb2c84931f5 P bb2c63b1e9fa4dc3aa89afac6501f1ff [term 3 LEARNER]: Refusing update from remote peer 20e529e97c57431bb03e326238d1bf8a: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch) I0703 01:40:00.850376 10833 raft_consensus.cc:1081] T 88835abd0ea34ff7acd90bb2c84931f5 P 200046fa8cd344f4a557b6feab568415 [term 3 LEARNER]: Refusing update from remote peer 20e529e97c57431bb03e326238d1bf8a: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch) I0703 01:40:00.851073 11529 consensus_queue.cc:962] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "200046fa8cd344f4a557b6feab568415" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 48069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I0703 01:40:00.851338 11509 consensus_queue.cc:962] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb2c63b1e9fa4dc3aa89afac6501f1ff" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 34920 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I0703 01:40:00.854454 11529 raft_consensus.cc:853] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a [term 3 LEADER]: attempt to promote peer bb2c63b1e9fa4dc3aa89afac6501f1ff: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing. I0703 01:40:00.855171 11509 raft_consensus.cc:2684] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a [term 3 LEADER]: Committing config change with OpId 3.7: config changed from index 6 to 7, 200046fa8cd344f4a557b6feab568415 (127.5.240.193) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "20e529e97c57431bb03e326238d1bf8a" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 58915 } } peers { permanent_uuid: "200046fa8cd344f4a557b6feab568415" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 48069 } attrs { promote: false } } peers { permanent_uuid: "bb2c63b1e9fa4dc3aa89afac6501f1ff" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 34920 } attrs { promote: true } } unsafe_config_change: true } I0703 01:40:00.855901 10832 raft_consensus.cc:2684] T 88835abd0ea34ff7acd90bb2c84931f5 P 200046fa8cd344f4a557b6feab568415 [term 3 FOLLOWER]: Committing config change with OpId 3.7: config changed from index 6 to 7, 200046fa8cd344f4a557b6feab568415 (127.5.240.193) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "20e529e97c57431bb03e326238d1bf8a" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 58915 } } peers { permanent_uuid: "200046fa8cd344f4a557b6feab568415" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 48069 } attrs { promote: false } } peers { permanent_uuid: "bb2c63b1e9fa4dc3aa89afac6501f1ff" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 34920 } attrs { promote: true } } unsafe_config_change: true } I0703 01:40:00.858166 11078 raft_consensus.cc:2684] T 88835abd0ea34ff7acd90bb2c84931f5 P bb2c63b1e9fa4dc3aa89afac6501f1ff [term 3 LEARNER]: Committing config change with OpId 3.7: config changed from index 6 to 7, 200046fa8cd344f4a557b6feab568415 (127.5.240.193) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "20e529e97c57431bb03e326238d1bf8a" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 58915 } } peers { permanent_uuid: "200046fa8cd344f4a557b6feab568415" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 48069 } attrs { promote: false } } peers { permanent_uuid: "bb2c63b1e9fa4dc3aa89afac6501f1ff" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 34920 } attrs { promote: true } } unsafe_config_change: true } I0703 01:40:00.858743 11458 catalog_manager.cc:3798] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a reported cstate change: config changed from index 6 to 7, 200046fa8cd344f4a557b6feab568415 (127.5.240.193) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "20e529e97c57431bb03e326238d1bf8a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "20e529e97c57431bb03e326238d1bf8a" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 58915 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "200046fa8cd344f4a557b6feab568415" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 48069 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb2c63b1e9fa4dc3aa89afac6501f1ff" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 34920 } attrs { promote: true } health_report { overall_health: HEALTHY } } unsafe_config_change: true } I0703 01:40:00.861537 11530 raft_consensus.cc:890] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a: attempting to promote NON_VOTER bb2c63b1e9fa4dc3aa89afac6501f1ff to VOTER I0703 01:40:00.861838 11530 consensus_peers.cc:455] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a -> Peer bb2c63b1e9fa4dc3aa89afac6501f1ff (127.5.240.195:34920): Closing peer: bb2c63b1e9fa4dc3aa89afac6501f1ff I0703 01:40:00.861912 11530 consensus_peers.cc:455] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a -> Peer 200046fa8cd344f4a557b6feab568415 (127.5.240.193:48069): Closing peer: 200046fa8cd344f4a557b6feab568415 I0703 01:40:00.862160 11530 consensus_queue.cc:204] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 3.7, Last appended by leader: 0, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "20e529e97c57431bb03e326238d1bf8a" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 58915 } } peers { permanent_uuid: "200046fa8cd344f4a557b6feab568415" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 48069 } attrs { promote: false } } peers { permanent_uuid: "bb2c63b1e9fa4dc3aa89afac6501f1ff" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 34920 } attrs { promote: false } } unsafe_config_change: true I0703 01:40:00.863694 11078 raft_consensus.cc:1081] T 88835abd0ea34ff7acd90bb2c84931f5 P bb2c63b1e9fa4dc3aa89afac6501f1ff [term 3 LEARNER]: Refusing update from remote peer 20e529e97c57431bb03e326238d1bf8a: Log matching property violated. Preceding OpId in replica: term: 3 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch) I0703 01:40:00.863945 10832 raft_consensus.cc:1081] T 88835abd0ea34ff7acd90bb2c84931f5 P 200046fa8cd344f4a557b6feab568415 [term 3 FOLLOWER]: Refusing update from remote peer 20e529e97c57431bb03e326238d1bf8a: Log matching property violated. Preceding OpId in replica: term: 3 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch) I0703 01:40:00.864276 11529 consensus_queue.cc:962] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb2c63b1e9fa4dc3aa89afac6501f1ff" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 34920 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I0703 01:40:00.864661 11530 consensus_queue.cc:962] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "200046fa8cd344f4a557b6feab568415" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 48069 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I0703 01:40:00.866930 11529 raft_consensus.cc:2684] T 88835abd0ea34ff7acd90bb2c84931f5 P 20e529e97c57431bb03e326238d1bf8a [term 3 LEADER]: Committing config change with OpId 3.8: config changed from index 7 to 8, bb2c63b1e9fa4dc3aa89afac6501f1ff (127.5.240.195) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "20e529e97c57431bb03e326238d1bf8a" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 58915 } } peers { permanent_uuid: "200046fa8cd344f4a557b6feab568415" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 48069 } attrs { promote: false } } peers { permanent_uuid: "bb2c63b1e9fa4dc3aa89afac6501f1ff" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 34920 } attrs { promote: false } } unsafe_config_change: true } I0703 01:40:00.867638 11078 raft_consensus.cc:2684] T 88835abd0ea34ff7acd90bb2c84931f5 P bb2c63b1e9fa4dc3aa89afac6501f1ff [term 3 FOLLOWER]: Committing config change with OpId 3.8: config changed from index 7 to 8, bb2c63b1e9fa4dc3aa89afac6501f1ff (127.5.240.195) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "20e529e97c57431bb03e326238d1bf8a" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 58915 } } peers { permanent_uuid: "200046fa8cd344f4a557b6feab568415" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 48069 } attrs { promote: false } } peers { permanent_uuid: "bb2c63b1e9fa4dc3aa89afac6501f1ff" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 34920 } attrs { promote: false } } unsafe_config_change: true } I0703 01:40:00.868075 10832 raft_consensus.cc:2684] T 88835abd0ea34ff7acd90bb2c84931f5 P 200046fa8cd344f4a557b6feab568415 [term 3 FOLLOWER]: Committing config change with OpId 3.8: config changed from index 7 to 8, bb2c63b1e9fa4dc3aa89afac6501f1ff (127.5.240.195) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "20e529e97c57431bb03e326238d1bf8a" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 58915 } } peers { permanent_uuid: "200046fa8cd344f4a557b6feab568415" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 48069 } attrs { promote: false } } peers { permanent_uuid: "bb2c63b1e9fa4dc3aa89afac6501f1ff" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 34920 } attrs { promote: false } } unsafe_config_change: true } I0703 01:40:00.871301 11458 catalog_manager.cc:3798] T 88835abd0ea34ff7acd90bb2c84931f5 P bb2c63b1e9fa4dc3aa89afac6501f1ff reported cstate change: config changed from index 7 to 8, bb2c63b1e9fa4dc3aa89afac6501f1ff (127.5.240.195) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "20e529e97c57431bb03e326238d1bf8a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "20e529e97c57431bb03e326238d1bf8a" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 58915 } } peers { permanent_uuid: "200046fa8cd344f4a557b6feab568415" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 48069 } attrs { promote: false } } peers { permanent_uuid: "bb2c63b1e9fa4dc3aa89afac6501f1ff" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 34920 } attrs { promote: false } } unsafe_config_change: true } I0703 01:40:00.981963 6083 kudu-admin-test.cc:806] Waiting for Master to see new config... I0703 01:40:00.987244 6083 kudu-admin-test.cc:811] Tablet locations: tablet_id: "88835abd0ea34ff7acd90bb2c84931f5" replicas { ts_info { permanent_uuid: "20e529e97c57431bb03e326238d1bf8a" rpc_addresses { host: "127.5.240.194" port: 58915 } } role: LEADER } replicas { ts_info { permanent_uuid: "200046fa8cd344f4a557b6feab568415" rpc_addresses { host: "127.5.240.193" port: 48069 } } role: FOLLOWER } replicas { ts_info { permanent_uuid: "bb2c63b1e9fa4dc3aa89afac6501f1ff" rpc_addresses { host: "127.5.240.195" port: 34920 } } role: FOLLOWER } DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } I0703 01:40:00.988155 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 10747 I0703 01:40:00.989281 10873 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:40:01.032833 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 10747 W0703 01:40:01.041208 10887 connection.cc:512] client connection to 127.5.240.193:48069 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:48069): Connection reset by peer (error 104) W0703 01:40:01.041270 11438 connection.cc:512] server connection from 127.5.240.193:55936 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:55936): Connection reset by peer (error 104) I0703 01:40:01.041376 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 10880 W0703 01:40:01.041415 10890 connection.cc:512] server connection from 127.5.240.193:50478 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:50478): Connection reset by peer (error 104) I0703 01:40:01.042223 10996 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:40:01.089536 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 10880 W0703 01:40:01.098230 11441 connection.cc:512] server connection from 127.5.240.194:54612 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:54612): Connection reset by peer (error 104) W0703 01:40:01.098474 11012 connection.cc:512] server connection from 127.5.240.194:34638 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:34638): Connection reset by peer (error 104) I0703 01:40:01.098660 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 11002 W0703 01:40:01.098676 11011 connection.cc:512] client connection to 127.5.240.194:58915 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:58915): Connection reset by peer (error 104) I0703 01:40:01.099421 11118 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:40:01.142617 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 11002 W0703 01:40:01.151374 11440 connection.cc:512] server connection from 127.5.240.195:40313 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.195:40313): Connection reset by peer (error 104) I0703 01:40:01.151435 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master pid 11431 I0703 01:40:01.152223 11488 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:40:01.214094 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master with pid 11431 [ OK ] AdminCliTest.TestUnsafeChangeConfigLeaderWithPendingConfig (8269 ms) [ RUN ] AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig I0703 01:40:01.237104 6083 test_util.cc:208] Using random seed: 1116026973 I0703 01:40:01.238639 6083 ts_itest-base.cc:113] Starting cluster with: I0703 01:40:01.238675 6083 ts_itest-base.cc:114] -------------- I0703 01:40:01.238713 6083 ts_itest-base.cc:115] 5 tablet servers I0703 01:40:01.238746 6083 ts_itest-base.cc:116] 3 replicas per TS I0703 01:40:01.238772 6083 ts_itest-base.cc:117] -------------- I0703 01:40:01.240312 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:45595 --webserver_interface=127.5.240.254 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:40:01.420693 11549 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:40:01.420929 11549 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:40:01.423143 11549 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0703 01:40:01.423190 11549 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:40:01.423259 11549 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0703 01:40:01.423307 11549 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0703 01:40:01.431365 11549 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:45595 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.5.240.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:40:01.437034 11549 master_main.cc:80] Initializing master server... I0703 01:40:01.438112 11549 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 154208us I0703 01:40:01.439210 11549 fs_manager.cc:261] Metadata directory not provided I0703 01:40:01.439275 11549 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:40:01.439522 11549 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:40:01.439611 11549 server_base.cc:433] Attempting to create new FS layout instead I0703 01:40:01.443763 11549 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/instance: uuid: "a5ace5d56f5f4273b353e57fa3bfaa3e" format_stamp: "Formatted at 2018-07-03 08:40:01 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:01.444278 11549 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "a5ace5d56f5f4273b353e57fa3bfaa3e" format_stamp: "Formatted at 2018-07-03 08:40:01 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:01.452353 11549 fs_manager.cc:501] Time spent creating directory manager: real 0.008s user 0.004s sys 0.003s I0703 01:40:01.453560 11549 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:40:01.453963 11549 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:40:01.457581 11549 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s I0703 01:40:01.457639 11549 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal uuid: "a5ace5d56f5f4273b353e57fa3bfaa3e" format_stamp: "Formatted at 2018-07-03 08:40:01 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:01.457746 11549 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:40:01.503026 11549 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:40:01.503953 11549 master_main.cc:83] Starting Master server... I0703 01:40:01.540964 11549 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.254:45595 I0703 01:40:01.541424 11549 webserver.cc:175] Starting webserver on 127.5.240.254:0 I0703 01:40:01.541468 11549 webserver.cc:186] Document root disabled I0703 01:40:01.542138 11549 webserver.cc:313] Webserver started. Bound to: http://127.5.240.254:41699/ I0703 01:40:01.542738 11549 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb I0703 01:40:01.546880 11608 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:40:01.550855 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master as pid 11549 I0703 01:40:01.551457 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.5.240.193:0 --local_ip_for_outbound_sockets=127.5.240.193 --webserver_interface=127.5.240.193 --webserver_port=0 --tserver_master_addrs=127.5.240.254:45595 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0703 01:40:01.559474 11608 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P a5ace5d56f5f4273b353e57fa3bfaa3e: Bootstrap starting. I0703 01:40:01.561738 11608 tablet_bootstrap.cc:578] T 00000000000000000000000000000000 P a5ace5d56f5f4273b353e57fa3bfaa3e: No blocks or log segments found. Creating new log. I0703 01:40:01.562479 11608 log.cc:525] T 00000000000000000000000000000000 P a5ace5d56f5f4273b353e57fa3bfaa3e: Log is configured to *not* fsync() on all Append() calls I0703 01:40:01.564239 11608 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P a5ace5d56f5f4273b353e57fa3bfaa3e: No bootstrap required, opened a new log I0703 01:40:01.570672 11608 raft_consensus.cc:325] T 00000000000000000000000000000000 P a5ace5d56f5f4273b353e57fa3bfaa3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5ace5d56f5f4273b353e57fa3bfaa3e" member_type: VOTER } I0703 01:40:01.570873 11608 raft_consensus.cc:351] T 00000000000000000000000000000000 P a5ace5d56f5f4273b353e57fa3bfaa3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:40:01.570936 11608 raft_consensus.cc:628] T 00000000000000000000000000000000 P a5ace5d56f5f4273b353e57fa3bfaa3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5ace5d56f5f4273b353e57fa3bfaa3e, State: Initialized, Role: FOLLOWER I0703 01:40:01.571398 11608 consensus_queue.cc:227] T 00000000000000000000000000000000 P a5ace5d56f5f4273b353e57fa3bfaa3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5ace5d56f5f4273b353e57fa3bfaa3e" member_type: VOTER } I0703 01:40:01.571552 11608 raft_consensus.cc:365] T 00000000000000000000000000000000 P a5ace5d56f5f4273b353e57fa3bfaa3e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:40:01.571655 11608 raft_consensus.cc:457] T 00000000000000000000000000000000 P a5ace5d56f5f4273b353e57fa3bfaa3e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:40:01.571738 11608 raft_consensus.cc:2730] T 00000000000000000000000000000000 P a5ace5d56f5f4273b353e57fa3bfaa3e: Snoozing failure detection for 2.133s (starting election) I0703 01:40:01.571822 11608 raft_consensus.cc:2782] T 00000000000000000000000000000000 P a5ace5d56f5f4273b353e57fa3bfaa3e [term 0 FOLLOWER]: Advancing to term 1 I0703 01:40:01.573220 11608 raft_consensus.cc:479] T 00000000000000000000000000000000 P a5ace5d56f5f4273b353e57fa3bfaa3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5ace5d56f5f4273b353e57fa3bfaa3e" member_type: VOTER } I0703 01:40:01.573601 11608 leader_election.cc:270] T 00000000000000000000000000000000 P a5ace5d56f5f4273b353e57fa3bfaa3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:40:01.575208 11615 raft_consensus.cc:2730] T 00000000000000000000000000000000 P a5ace5d56f5f4273b353e57fa3bfaa3e: Snoozing failure detection for 1.502s (election complete) I0703 01:40:01.575690 11615 raft_consensus.cc:2537] T 00000000000000000000000000000000 P a5ace5d56f5f4273b353e57fa3bfaa3e [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:40:01.577047 11615 raft_consensus.cc:593] T 00000000000000000000000000000000 P a5ace5d56f5f4273b353e57fa3bfaa3e [term 1 LEADER]: Becoming Leader. State: Replica: a5ace5d56f5f4273b353e57fa3bfaa3e, State: Running, Role: LEADER I0703 01:40:01.577447 11615 consensus_queue.cc:204] T 00000000000000000000000000000000 P a5ace5d56f5f4273b353e57fa3bfaa3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5ace5d56f5f4273b353e57fa3bfaa3e" member_type: VOTER } I0703 01:40:01.580067 11608 sys_catalog.cc:337] T 00000000000000000000000000000000 P a5ace5d56f5f4273b353e57fa3bfaa3e [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "a5ace5d56f5f4273b353e57fa3bfaa3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5ace5d56f5f4273b353e57fa3bfaa3e" member_type: VOTER } } I0703 01:40:01.580260 11608 sys_catalog.cc:340] T 00000000000000000000000000000000 P a5ace5d56f5f4273b353e57fa3bfaa3e [sys.catalog]: This master's current role is: LEADER I0703 01:40:01.581126 11616 sys_catalog.cc:337] T 00000000000000000000000000000000 P a5ace5d56f5f4273b353e57fa3bfaa3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5ace5d56f5f4273b353e57fa3bfaa3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5ace5d56f5f4273b353e57fa3bfaa3e" member_type: VOTER } } I0703 01:40:01.581125 11617 sys_catalog.cc:337] T 00000000000000000000000000000000 P a5ace5d56f5f4273b353e57fa3bfaa3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5ace5d56f5f4273b353e57fa3bfaa3e. Latest consensus state: current_term: 1 leader_uuid: "a5ace5d56f5f4273b353e57fa3bfaa3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5ace5d56f5f4273b353e57fa3bfaa3e" member_type: VOTER } } I0703 01:40:01.581969 11616 sys_catalog.cc:340] T 00000000000000000000000000000000 P a5ace5d56f5f4273b353e57fa3bfaa3e [sys.catalog]: This master's current role is: LEADER I0703 01:40:01.582031 11617 sys_catalog.cc:340] T 00000000000000000000000000000000 P a5ace5d56f5f4273b353e57fa3bfaa3e [sys.catalog]: This master's current role is: LEADER I0703 01:40:01.583243 11608 sys_catalog.cc:424] T 00000000000000000000000000000000 P a5ace5d56f5f4273b353e57fa3bfaa3e [sys.catalog]: configured and running, proceeding with master startup. I0703 01:40:01.583711 11619 catalog_manager.cc:1006] Loading table and tablet metadata into memory... I0703 01:40:01.584946 11549 master_main.cc:86] Master server successfully started. I0703 01:40:01.590231 11619 catalog_manager.cc:1017] Initializing Kudu internal certificate authority... I0703 01:40:01.651857 11619 catalog_manager.cc:904] Generated new certificate authority record I0703 01:40:01.653913 11619 catalog_manager.cc:1026] Loading token signing keys... I0703 01:40:01.664052 11619 catalog_manager.cc:4102] T 00000000000000000000000000000000 P a5ace5d56f5f4273b353e57fa3bfaa3e: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:40:01.732264 11610 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:40:01.732496 11610 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:40:01.734578 11610 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:40:01.734690 11610 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.193 I0703 01:40:01.742221 11610 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.5.240.193 --webserver_port=0 --tserver_master_addrs=127.5.240.254:45595 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.193 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:40:01.748854 11610 tablet_server_main.cc:85] Initializing tablet server... I0703 01:40:01.750046 11610 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 154208us I0703 01:40:01.751034 11610 fs_manager.cc:261] Metadata directory not provided I0703 01:40:01.751099 11610 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:40:01.751332 11610 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:40:01.751397 11610 server_base.cc:433] Attempting to create new FS layout instead I0703 01:40:01.755594 11610 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "ee1b80f8727c4321a63259acf818aff8" format_stamp: "Formatted at 2018-07-03 08:40:01 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:01.756085 11610 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "ee1b80f8727c4321a63259acf818aff8" format_stamp: "Formatted at 2018-07-03 08:40:01 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:01.764407 11610 fs_manager.cc:501] Time spent creating directory manager: real 0.008s user 0.004s sys 0.003s I0703 01:40:01.765736 11610 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:40:01.766167 11610 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:40:01.770360 11610 fs_manager.cc:417] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s I0703 01:40:01.770423 11610 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal uuid: "ee1b80f8727c4321a63259acf818aff8" format_stamp: "Formatted at 2018-07-03 08:40:01 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:01.770563 11610 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:40:01.829135 11610 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:40:01.831300 11610 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:40:01.831784 11610 tablet_server_main.cc:90] Starting tablet server... I0703 01:40:01.927708 11610 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.193:46472 I0703 01:40:01.928130 11610 webserver.cc:175] Starting webserver on 127.5.240.193:0 I0703 01:40:01.928167 11610 webserver.cc:186] Document root disabled I0703 01:40:01.929348 11610 webserver.cc:313] Webserver started. Bound to: http://127.5.240.193:34473/ I0703 01:40:01.929913 11610 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb I0703 01:40:01.931715 11610 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:40:01.934355 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 11610 I0703 01:40:01.934947 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.5.240.194:0 --local_ip_for_outbound_sockets=127.5.240.194 --webserver_interface=127.5.240.194 --webserver_port=0 --tserver_master_addrs=127.5.240.254:45595 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0703 01:40:01.947799 11742 heartbeater.cc:346] Connected to a master server at 127.5.240.254:45595 I0703 01:40:01.947947 11742 heartbeater.cc:426] Registering TS with master... I0703 01:40:01.948742 11742 heartbeater.cc:475] Master 127.5.240.254:45595 requested a full tablet report, sending... I0703 01:40:01.950331 11575 ts_manager.cc:98] Registered new tserver with Master: ee1b80f8727c4321a63259acf818aff8 (127.5.240.193:46472) I0703 01:40:01.952913 11575 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.193:47942 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:40:02.096467 11746 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:40:02.096725 11746 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:40:02.098729 11746 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:40:02.098848 11746 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.194 I0703 01:40:02.105741 11746 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.194:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.5.240.194 --webserver_port=0 --tserver_master_addrs=127.5.240.254:45595 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.194 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:40:02.112298 11746 tablet_server_main.cc:85] Initializing tablet server... I0703 01:40:02.113448 11746 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 154708us I0703 01:40:02.114405 11746 fs_manager.cc:261] Metadata directory not provided I0703 01:40:02.114468 11746 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:40:02.114718 11746 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:40:02.114781 11746 server_base.cc:433] Attempting to create new FS layout instead I0703 01:40:02.118847 11746 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "ea5f6a64e88b44c58375146ddaaac0b5" format_stamp: "Formatted at 2018-07-03 08:40:02 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:02.119367 11746 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "ea5f6a64e88b44c58375146ddaaac0b5" format_stamp: "Formatted at 2018-07-03 08:40:02 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:02.126775 11746 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s I0703 01:40:02.127959 11746 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:40:02.128355 11746 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:40:02.131955 11746 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s I0703 01:40:02.132014 11746 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal uuid: "ea5f6a64e88b44c58375146ddaaac0b5" format_stamp: "Formatted at 2018-07-03 08:40:02 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:02.132133 11746 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:40:02.183028 11746 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:40:02.184967 11746 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:40:02.185328 11746 tablet_server_main.cc:90] Starting tablet server... I0703 01:40:02.269220 11746 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.194:37351 I0703 01:40:02.269665 11746 webserver.cc:175] Starting webserver on 127.5.240.194:0 I0703 01:40:02.269702 11746 webserver.cc:186] Document root disabled I0703 01:40:02.270949 11746 webserver.cc:313] Webserver started. Bound to: http://127.5.240.194:35892/ I0703 01:40:02.271471 11746 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb I0703 01:40:02.273247 11746 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:40:02.278489 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 11746 I0703 01:40:02.279309 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.5.240.195:0 --local_ip_for_outbound_sockets=127.5.240.195 --webserver_interface=127.5.240.195 --webserver_port=0 --tserver_master_addrs=127.5.240.254:45595 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0703 01:40:02.286974 11865 heartbeater.cc:346] Connected to a master server at 127.5.240.254:45595 I0703 01:40:02.287117 11865 heartbeater.cc:426] Registering TS with master... I0703 01:40:02.287919 11865 heartbeater.cc:475] Master 127.5.240.254:45595 requested a full tablet report, sending... I0703 01:40:02.289037 11575 ts_manager.cc:98] Registered new tserver with Master: ea5f6a64e88b44c58375146ddaaac0b5 (127.5.240.194:37351) I0703 01:40:02.289896 11575 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.194:44976 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:40:02.440472 11868 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:40:02.440718 11868 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:40:02.442709 11868 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:40:02.442802 11868 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.195 I0703 01:40:02.449339 11868 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.195:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.5.240.195 --webserver_port=0 --tserver_master_addrs=127.5.240.254:45595 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.195 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:40:02.456023 11868 tablet_server_main.cc:85] Initializing tablet server... I0703 01:40:02.457168 11868 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 154708us I0703 01:40:02.458128 11868 fs_manager.cc:261] Metadata directory not provided I0703 01:40:02.458181 11868 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:40:02.458390 11868 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:40:02.458447 11868 server_base.cc:433] Attempting to create new FS layout instead I0703 01:40:02.462541 11868 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "7e8e64942ea3471ab77a706b397f0c4f" format_stamp: "Formatted at 2018-07-03 08:40:02 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:02.463063 11868 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "7e8e64942ea3471ab77a706b397f0c4f" format_stamp: "Formatted at 2018-07-03 08:40:02 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:02.578655 11868 fs_manager.cc:501] Time spent creating directory manager: real 0.115s user 0.005s sys 0.002s I0703 01:40:02.579864 11868 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:40:02.580263 11868 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:40:02.583911 11868 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0703 01:40:02.583962 11868 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal uuid: "7e8e64942ea3471ab77a706b397f0c4f" format_stamp: "Formatted at 2018-07-03 08:40:02 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:02.584084 11868 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:40:02.620040 11868 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:40:02.621405 11868 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:40:02.621709 11868 tablet_server_main.cc:90] Starting tablet server... I0703 01:40:02.706140 11868 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.195:41189 I0703 01:40:02.706576 11868 webserver.cc:175] Starting webserver on 127.5.240.195:0 I0703 01:40:02.706606 11868 webserver.cc:186] Document root disabled I0703 01:40:02.707244 11868 webserver.cc:313] Webserver started. Bound to: http://127.5.240.195:46775/ I0703 01:40:02.707784 11868 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb I0703 01:40:02.711658 11868 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:40:02.714135 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 11868 I0703 01:40:02.714977 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.5.240.196:0 --local_ip_for_outbound_sockets=127.5.240.196 --webserver_interface=127.5.240.196 --webserver_port=0 --tserver_master_addrs=127.5.240.254:45595 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0703 01:40:02.723394 11987 heartbeater.cc:346] Connected to a master server at 127.5.240.254:45595 I0703 01:40:02.723569 11987 heartbeater.cc:426] Registering TS with master... I0703 01:40:02.724501 11987 heartbeater.cc:475] Master 127.5.240.254:45595 requested a full tablet report, sending... I0703 01:40:02.725636 11575 ts_manager.cc:98] Registered new tserver with Master: 7e8e64942ea3471ab77a706b397f0c4f (127.5.240.195:41189) I0703 01:40:02.726960 11575 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.195:43470 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:40:02.879146 11990 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:40:02.879398 11990 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:40:02.881413 11990 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:40:02.881522 11990 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.196 I0703 01:40:02.888103 11990 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.196:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb --webserver_interface=127.5.240.196 --webserver_port=0 --tserver_master_addrs=127.5.240.254:45595 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.196 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:40:02.894718 11990 tablet_server_main.cc:85] Initializing tablet server... I0703 01:40:02.895881 11990 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 154708us I0703 01:40:02.896834 11990 fs_manager.cc:261] Metadata directory not provided I0703 01:40:02.896898 11990 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:40:02.897130 11990 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:40:02.897193 11990 server_base.cc:433] Attempting to create new FS layout instead I0703 01:40:02.901279 11990 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/instance: uuid: "655d2894f7d14417bffae7af9cca3733" format_stamp: "Formatted at 2018-07-03 08:40:02 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:02.901830 11990 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal/instance: uuid: "655d2894f7d14417bffae7af9cca3733" format_stamp: "Formatted at 2018-07-03 08:40:02 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:02.909222 11990 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.003s sys 0.003s I0703 01:40:02.910408 11990 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:40:02.910821 11990 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:40:02.914381 11990 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s I0703 01:40:02.914438 11990 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal uuid: "655d2894f7d14417bffae7af9cca3733" format_stamp: "Formatted at 2018-07-03 08:40:02 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:02.914564 11990 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:40:02.955176 11742 heartbeater.cc:467] Master 127.5.240.254:45595 was elected leader, sending a full tablet report... I0703 01:40:02.983706 11990 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:40:02.985596 11990 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:40:02.985965 11990 tablet_server_main.cc:90] Starting tablet server... I0703 01:40:03.070528 11990 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.196:50417 I0703 01:40:03.070969 11990 webserver.cc:175] Starting webserver on 127.5.240.196:0 I0703 01:40:03.071002 11990 webserver.cc:186] Document root disabled I0703 01:40:03.072165 11990 webserver.cc:313] Webserver started. Bound to: http://127.5.240.196:47347/ I0703 01:40:03.072712 11990 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb I0703 01:40:03.074473 11990 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:40:03.078519 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 11990 I0703 01:40:03.079126 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data --block_manager=log --rpc_bind_addresses=127.5.240.197:0 --local_ip_for_outbound_sockets=127.5.240.197 --webserver_interface=127.5.240.197 --webserver_port=0 --tserver_master_addrs=127.5.240.254:45595 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0703 01:40:03.086676 12109 heartbeater.cc:346] Connected to a master server at 127.5.240.254:45595 I0703 01:40:03.086820 12109 heartbeater.cc:426] Registering TS with master... I0703 01:40:03.087615 12109 heartbeater.cc:475] Master 127.5.240.254:45595 requested a full tablet report, sending... I0703 01:40:03.088688 11575 ts_manager.cc:98] Registered new tserver with Master: 655d2894f7d14417bffae7af9cca3733 (127.5.240.196:50417) I0703 01:40:03.089537 11575 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.196:50595 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:40:03.235746 12112 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:40:03.235955 12112 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:40:03.237956 12112 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:40:03.238047 12112 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.197 I0703 01:40:03.244843 12112 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.197:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb --webserver_interface=127.5.240.197 --webserver_port=0 --tserver_master_addrs=127.5.240.254:45595 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.197 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:40:03.251315 12112 tablet_server_main.cc:85] Initializing tablet server... I0703 01:40:03.252424 12112 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 155208us I0703 01:40:03.253361 12112 fs_manager.cc:261] Metadata directory not provided I0703 01:40:03.253415 12112 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:40:03.253643 12112 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:40:03.253702 12112 server_base.cc:433] Attempting to create new FS layout instead I0703 01:40:03.257905 12112 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/instance: uuid: "01a76da15ef14c80b66d2706c0978eb8" format_stamp: "Formatted at 2018-07-03 08:40:03 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:03.258399 12112 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal/instance: uuid: "01a76da15ef14c80b66d2706c0978eb8" format_stamp: "Formatted at 2018-07-03 08:40:03 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:03.292142 11865 heartbeater.cc:467] Master 127.5.240.254:45595 was elected leader, sending a full tablet report... I0703 01:40:03.359504 12112 fs_manager.cc:501] Time spent creating directory manager: real 0.101s user 0.003s sys 0.003s I0703 01:40:03.360724 12112 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:40:03.361109 12112 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:40:03.364778 12112 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s I0703 01:40:03.364830 12112 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal uuid: "01a76da15ef14c80b66d2706c0978eb8" format_stamp: "Formatted at 2018-07-03 08:40:03 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:03.364953 12112 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:40:03.429565 12112 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:40:03.431416 12112 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:40:03.431779 12112 tablet_server_main.cc:90] Starting tablet server... I0703 01:40:03.519366 12112 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.197:43693 I0703 01:40:03.519809 12112 webserver.cc:175] Starting webserver on 127.5.240.197:0 I0703 01:40:03.519845 12112 webserver.cc:186] Document root disabled I0703 01:40:03.521010 12112 webserver.cc:313] Webserver started. Bound to: http://127.5.240.197:38683/ I0703 01:40:03.570328 12112 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb I0703 01:40:03.572161 12112 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:40:03.572752 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 12112 I0703 01:40:03.584518 12231 heartbeater.cc:346] Connected to a master server at 127.5.240.254:45595 I0703 01:40:03.584666 12231 heartbeater.cc:426] Registering TS with master... I0703 01:40:03.585431 12231 heartbeater.cc:475] Master 127.5.240.254:45595 requested a full tablet report, sending... I0703 01:40:03.586408 11575 ts_manager.cc:98] Registered new tserver with Master: 01a76da15ef14c80b66d2706c0978eb8 (127.5.240.197:43693) I0703 01:40:03.587468 11575 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.197:36749 I0703 01:40:03.588030 6083 external_mini_cluster.cc:490] 5 TS(s) registered with all masters I0703 01:40:03.598284 11574 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.5.240.254:54334: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0703 01:40:03.625433 12046 tablet_service.cc:759] Processing CreateTablet for tablet aa3ae539f4c6418496bbb47ff00cb47e (table=TestTable [id=eade82a1ab6a4ddbb599737edcdea82f]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:40:03.625758 11802 tablet_service.cc:759] Processing CreateTablet for tablet aa3ae539f4c6418496bbb47ff00cb47e (table=TestTable [id=eade82a1ab6a4ddbb599737edcdea82f]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:40:03.626368 12046 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet aa3ae539f4c6418496bbb47ff00cb47e. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:40:03.626441 11802 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet aa3ae539f4c6418496bbb47ff00cb47e. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:40:03.629076 12168 tablet_service.cc:759] Processing CreateTablet for tablet aa3ae539f4c6418496bbb47ff00cb47e (table=TestTable [id=eade82a1ab6a4ddbb599737edcdea82f]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:40:03.629667 12168 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet aa3ae539f4c6418496bbb47ff00cb47e. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:40:03.638960 12168 ts_tablet_manager.cc:1173] T aa3ae539f4c6418496bbb47ff00cb47e P 01a76da15ef14c80b66d2706c0978eb8: Registered tablet (data state: TABLET_DATA_READY) I0703 01:40:03.639865 12251 ts_tablet_manager.cc:1028] T aa3ae539f4c6418496bbb47ff00cb47e P 01a76da15ef14c80b66d2706c0978eb8: Bootstrapping tablet I0703 01:40:03.640470 12251 tablet_bootstrap.cc:427] T aa3ae539f4c6418496bbb47ff00cb47e P 01a76da15ef14c80b66d2706c0978eb8: Bootstrap starting. I0703 01:40:03.641369 12046 ts_tablet_manager.cc:1173] T aa3ae539f4c6418496bbb47ff00cb47e P 655d2894f7d14417bffae7af9cca3733: Registered tablet (data state: TABLET_DATA_READY) I0703 01:40:03.642217 12252 ts_tablet_manager.cc:1028] T aa3ae539f4c6418496bbb47ff00cb47e P 655d2894f7d14417bffae7af9cca3733: Bootstrapping tablet I0703 01:40:03.642297 11802 ts_tablet_manager.cc:1173] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5: Registered tablet (data state: TABLET_DATA_READY) I0703 01:40:03.642769 12252 tablet_bootstrap.cc:427] T aa3ae539f4c6418496bbb47ff00cb47e P 655d2894f7d14417bffae7af9cca3733: Bootstrap starting. I0703 01:40:03.643703 12251 tablet_bootstrap.cc:578] T aa3ae539f4c6418496bbb47ff00cb47e P 01a76da15ef14c80b66d2706c0978eb8: No blocks or log segments found. Creating new log. I0703 01:40:03.644794 12251 log.cc:525] T aa3ae539f4c6418496bbb47ff00cb47e P 01a76da15ef14c80b66d2706c0978eb8: Log is configured to *not* fsync() on all Append() calls I0703 01:40:03.645725 12252 tablet_bootstrap.cc:578] T aa3ae539f4c6418496bbb47ff00cb47e P 655d2894f7d14417bffae7af9cca3733: No blocks or log segments found. Creating new log. I0703 01:40:03.646312 12253 ts_tablet_manager.cc:1028] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5: Bootstrapping tablet I0703 01:40:03.646672 12252 log.cc:525] T aa3ae539f4c6418496bbb47ff00cb47e P 655d2894f7d14417bffae7af9cca3733: Log is configured to *not* fsync() on all Append() calls I0703 01:40:03.646776 12253 tablet_bootstrap.cc:427] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5: Bootstrap starting. I0703 01:40:03.648845 12253 tablet_bootstrap.cc:578] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5: No blocks or log segments found. Creating new log. I0703 01:40:03.649477 12253 log.cc:525] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5: Log is configured to *not* fsync() on all Append() calls I0703 01:40:03.728893 11987 heartbeater.cc:467] Master 127.5.240.254:45595 was elected leader, sending a full tablet report... I0703 01:40:03.795681 12251 tablet_bootstrap.cc:427] T aa3ae539f4c6418496bbb47ff00cb47e P 01a76da15ef14c80b66d2706c0978eb8: No bootstrap required, opened a new log I0703 01:40:03.795948 12251 ts_tablet_manager.cc:1045] T aa3ae539f4c6418496bbb47ff00cb47e P 01a76da15ef14c80b66d2706c0978eb8: Time spent bootstrapping tablet: real 0.156s user 0.002s sys 0.004s I0703 01:40:03.796170 12253 tablet_bootstrap.cc:427] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5: No bootstrap required, opened a new log I0703 01:40:03.796321 12252 tablet_bootstrap.cc:427] T aa3ae539f4c6418496bbb47ff00cb47e P 655d2894f7d14417bffae7af9cca3733: No bootstrap required, opened a new log I0703 01:40:03.796349 12253 ts_tablet_manager.cc:1045] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5: Time spent bootstrapping tablet: real 0.150s user 0.002s sys 0.002s I0703 01:40:03.796571 12252 ts_tablet_manager.cc:1045] T aa3ae539f4c6418496bbb47ff00cb47e P 655d2894f7d14417bffae7af9cca3733: Time spent bootstrapping tablet: real 0.154s user 0.002s sys 0.004s I0703 01:40:03.802608 12253 raft_consensus.cc:325] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655d2894f7d14417bffae7af9cca3733" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 50417 } } peers { permanent_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 37351 } } peers { permanent_uuid: "01a76da15ef14c80b66d2706c0978eb8" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43693 } } I0703 01:40:03.802915 12253 raft_consensus.cc:351] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:40:03.802995 12253 raft_consensus.cc:628] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea5f6a64e88b44c58375146ddaaac0b5, State: Initialized, Role: FOLLOWER I0703 01:40:03.803411 12253 consensus_queue.cc:227] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655d2894f7d14417bffae7af9cca3733" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 50417 } } peers { permanent_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 37351 } } peers { permanent_uuid: "01a76da15ef14c80b66d2706c0978eb8" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43693 } } I0703 01:40:03.804673 12253 ts_tablet_manager.cc:1073] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5: Time spent starting tablet: real 0.008s user 0.007s sys 0.001s I0703 01:40:03.804643 12251 raft_consensus.cc:325] T aa3ae539f4c6418496bbb47ff00cb47e P 01a76da15ef14c80b66d2706c0978eb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655d2894f7d14417bffae7af9cca3733" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 50417 } } peers { permanent_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 37351 } } peers { permanent_uuid: "01a76da15ef14c80b66d2706c0978eb8" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43693 } } I0703 01:40:03.805028 12251 raft_consensus.cc:351] T aa3ae539f4c6418496bbb47ff00cb47e P 01a76da15ef14c80b66d2706c0978eb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:40:03.805099 12251 raft_consensus.cc:628] T aa3ae539f4c6418496bbb47ff00cb47e P 01a76da15ef14c80b66d2706c0978eb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01a76da15ef14c80b66d2706c0978eb8, State: Initialized, Role: FOLLOWER I0703 01:40:03.805490 12251 consensus_queue.cc:227] T aa3ae539f4c6418496bbb47ff00cb47e P 01a76da15ef14c80b66d2706c0978eb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655d2894f7d14417bffae7af9cca3733" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 50417 } } peers { permanent_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 37351 } } peers { permanent_uuid: "01a76da15ef14c80b66d2706c0978eb8" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43693 } } I0703 01:40:03.805814 12252 raft_consensus.cc:325] T aa3ae539f4c6418496bbb47ff00cb47e P 655d2894f7d14417bffae7af9cca3733 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655d2894f7d14417bffae7af9cca3733" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 50417 } } peers { permanent_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 37351 } } peers { permanent_uuid: "01a76da15ef14c80b66d2706c0978eb8" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43693 } } I0703 01:40:03.806325 12252 raft_consensus.cc:351] T aa3ae539f4c6418496bbb47ff00cb47e P 655d2894f7d14417bffae7af9cca3733 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:40:03.806406 12252 raft_consensus.cc:628] T aa3ae539f4c6418496bbb47ff00cb47e P 655d2894f7d14417bffae7af9cca3733 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 655d2894f7d14417bffae7af9cca3733, State: Initialized, Role: FOLLOWER I0703 01:40:03.806954 12231 heartbeater.cc:467] Master 127.5.240.254:45595 was elected leader, sending a full tablet report... I0703 01:40:03.807102 12252 consensus_queue.cc:227] T aa3ae539f4c6418496bbb47ff00cb47e P 655d2894f7d14417bffae7af9cca3733 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655d2894f7d14417bffae7af9cca3733" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 50417 } } peers { permanent_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 37351 } } peers { permanent_uuid: "01a76da15ef14c80b66d2706c0978eb8" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43693 } } I0703 01:40:03.808121 12251 ts_tablet_manager.cc:1073] T aa3ae539f4c6418496bbb47ff00cb47e P 01a76da15ef14c80b66d2706c0978eb8: Time spent starting tablet: real 0.012s user 0.009s sys 0.002s I0703 01:40:03.808349 12109 heartbeater.cc:467] Master 127.5.240.254:45595 was elected leader, sending a full tablet report... I0703 01:40:03.808759 12252 ts_tablet_manager.cc:1073] T aa3ae539f4c6418496bbb47ff00cb47e P 655d2894f7d14417bffae7af9cca3733: Time spent starting tablet: real 0.012s user 0.006s sys 0.004s W0703 01:40:03.822428 12232 tablet.cc:1678] T aa3ae539f4c6418496bbb47ff00cb47e P 01a76da15ef14c80b66d2706c0978eb8: Can't schedule compaction. Clean time has not been advanced past its initial value. W0703 01:40:03.825179 12110 tablet.cc:1678] T aa3ae539f4c6418496bbb47ff00cb47e P 655d2894f7d14417bffae7af9cca3733: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:40:03.954414 12257 raft_consensus.cc:457] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0703 01:40:03.954530 12257 raft_consensus.cc:2730] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5: Snoozing failure detection for 1.816s (starting election) I0703 01:40:03.954664 12257 raft_consensus.cc:479] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655d2894f7d14417bffae7af9cca3733" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 50417 } } peers { permanent_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 37351 } } peers { permanent_uuid: "01a76da15ef14c80b66d2706c0978eb8" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43693 } } I0703 01:40:03.955816 12257 leader_election.cc:243] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 655d2894f7d14417bffae7af9cca3733 (127.5.240.196:50417) I0703 01:40:03.956164 12257 leader_election.cc:243] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 01a76da15ef14c80b66d2706c0978eb8 (127.5.240.197:43693) I0703 01:40:03.964865 12066 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "aa3ae539f4c6418496bbb47ff00cb47e" candidate_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "655d2894f7d14417bffae7af9cca3733" is_pre_election: true I0703 01:40:03.964615 12188 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "aa3ae539f4c6418496bbb47ff00cb47e" candidate_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01a76da15ef14c80b66d2706c0978eb8" is_pre_election: true I0703 01:40:03.965106 12066 raft_consensus.cc:2730] T aa3ae539f4c6418496bbb47ff00cb47e P 655d2894f7d14417bffae7af9cca3733: Snoozing failure detection for 2.042s (vote granted) I0703 01:40:03.965204 12066 raft_consensus.cc:2243] T aa3ae539f4c6418496bbb47ff00cb47e P 655d2894f7d14417bffae7af9cca3733 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea5f6a64e88b44c58375146ddaaac0b5 in term 0. I0703 01:40:03.965237 12188 raft_consensus.cc:2730] T aa3ae539f4c6418496bbb47ff00cb47e P 01a76da15ef14c80b66d2706c0978eb8: Snoozing failure detection for 1.806s (vote granted) I0703 01:40:03.965329 12188 raft_consensus.cc:2243] T aa3ae539f4c6418496bbb47ff00cb47e P 01a76da15ef14c80b66d2706c0978eb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea5f6a64e88b44c58375146ddaaac0b5 in term 0. I0703 01:40:03.965842 11755 leader_election.cc:379] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 655d2894f7d14417bffae7af9cca3733 (127.5.240.196:50417) I0703 01:40:03.965947 11755 leader_election.cc:270] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0703 01:40:03.966087 11756 leader_election.cc:379] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 01a76da15ef14c80b66d2706c0978eb8 (127.5.240.197:43693) I0703 01:40:03.966269 12257 raft_consensus.cc:2730] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5: Snoozing failure detection for 1.773s (election complete) I0703 01:40:03.966373 12257 raft_consensus.cc:2537] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0703 01:40:03.966434 12257 raft_consensus.cc:457] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0703 01:40:03.966491 12257 raft_consensus.cc:2730] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5: Snoozing failure detection for 2.162s (starting election) I0703 01:40:03.966548 12257 raft_consensus.cc:2782] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:40:03.968083 12257 raft_consensus.cc:479] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655d2894f7d14417bffae7af9cca3733" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 50417 } } peers { permanent_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 37351 } } peers { permanent_uuid: "01a76da15ef14c80b66d2706c0978eb8" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43693 } } I0703 01:40:03.968412 12257 leader_election.cc:243] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 [CANDIDATE]: Term 1 election: Requesting vote from peer 655d2894f7d14417bffae7af9cca3733 (127.5.240.196:50417) I0703 01:40:03.968554 12257 leader_election.cc:243] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 [CANDIDATE]: Term 1 election: Requesting vote from peer 01a76da15ef14c80b66d2706c0978eb8 (127.5.240.197:43693) I0703 01:40:03.969130 12066 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "aa3ae539f4c6418496bbb47ff00cb47e" candidate_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "655d2894f7d14417bffae7af9cca3733" I0703 01:40:03.969240 12066 raft_consensus.cc:2782] T aa3ae539f4c6418496bbb47ff00cb47e P 655d2894f7d14417bffae7af9cca3733 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:40:03.969249 12188 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "aa3ae539f4c6418496bbb47ff00cb47e" candidate_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01a76da15ef14c80b66d2706c0978eb8" I0703 01:40:03.969316 12066 raft_consensus.cc:2730] T aa3ae539f4c6418496bbb47ff00cb47e P 655d2894f7d14417bffae7af9cca3733: Snoozing failure detection for 1.750s (vote granted) I0703 01:40:03.969369 12188 raft_consensus.cc:2782] T aa3ae539f4c6418496bbb47ff00cb47e P 01a76da15ef14c80b66d2706c0978eb8 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:40:03.969457 12188 raft_consensus.cc:2730] T aa3ae539f4c6418496bbb47ff00cb47e P 01a76da15ef14c80b66d2706c0978eb8: Snoozing failure detection for 1.800s (vote granted) I0703 01:40:03.970652 12066 raft_consensus.cc:2243] T aa3ae539f4c6418496bbb47ff00cb47e P 655d2894f7d14417bffae7af9cca3733 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea5f6a64e88b44c58375146ddaaac0b5 in term 1. I0703 01:40:03.970757 12188 raft_consensus.cc:2243] T aa3ae539f4c6418496bbb47ff00cb47e P 01a76da15ef14c80b66d2706c0978eb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea5f6a64e88b44c58375146ddaaac0b5 in term 1. I0703 01:40:03.971089 11755 leader_election.cc:379] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 [CANDIDATE]: Term 1 election: Vote granted by peer 655d2894f7d14417bffae7af9cca3733 (127.5.240.196:50417) I0703 01:40:03.971186 11755 leader_election.cc:270] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:40:03.971297 11756 leader_election.cc:379] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 [CANDIDATE]: Term 1 election: Vote granted by peer 01a76da15ef14c80b66d2706c0978eb8 (127.5.240.197:43693) I0703 01:40:03.971398 12257 raft_consensus.cc:2730] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5: Snoozing failure detection for 1.844s (election complete) I0703 01:40:03.971493 12257 raft_consensus.cc:2537] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:40:03.972342 12257 raft_consensus.cc:593] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 [term 1 LEADER]: Becoming Leader. State: Replica: ea5f6a64e88b44c58375146ddaaac0b5, State: Running, Role: LEADER I0703 01:40:03.972745 12257 consensus_queue.cc:204] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655d2894f7d14417bffae7af9cca3733" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 50417 } } peers { permanent_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 37351 } } peers { permanent_uuid: "01a76da15ef14c80b66d2706c0978eb8" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43693 } } I0703 01:40:03.976830 11574 catalog_manager.cc:3798] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 reported cstate change: term changed from 0 to 1, leader changed from to ea5f6a64e88b44c58375146ddaaac0b5 (127.5.240.194). New cstate: current_term: 1 leader_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655d2894f7d14417bffae7af9cca3733" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 50417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 37351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01a76da15ef14c80b66d2706c0978eb8" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43693 } health_report { overall_health: UNKNOWN } } } W0703 01:40:04.024621 11866 tablet.cc:1678] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:40:04.063885 6083 external_mini_cluster.cc:490] 5 TS(s) registered with all masters I0703 01:40:04.064992 6083 ts_itest-base.cc:233] Waiting for 0 tablets on tserver ee1b80f8727c4321a63259acf818aff8 to finish bootstrapping I0703 01:40:04.071712 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver ea5f6a64e88b44c58375146ddaaac0b5 to finish bootstrapping I0703 01:40:04.076242 6083 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 7e8e64942ea3471ab77a706b397f0c4f to finish bootstrapping I0703 01:40:04.082296 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 655d2894f7d14417bffae7af9cca3733 to finish bootstrapping I0703 01:40:04.087340 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 01a76da15ef14c80b66d2706c0978eb8 to finish bootstrapping I0703 01:40:04.092084 6083 kudu-admin-test.cc:842] Waiting for Master to see the current replicas... I0703 01:40:04.093287 6083 kudu-admin-test.cc:849] Tablet locations: tablet_id: "aa3ae539f4c6418496bbb47ff00cb47e" replicas { ts_info { permanent_uuid: "655d2894f7d14417bffae7af9cca3733" rpc_addresses { host: "127.5.240.196" port: 50417 } } role: FOLLOWER } replicas { ts_info { permanent_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" rpc_addresses { host: "127.5.240.194" port: 37351 } } role: LEADER } replicas { ts_info { permanent_uuid: "01a76da15ef14c80b66d2706c0978eb8" rpc_addresses { host: "127.5.240.197" port: 43693 } } role: FOLLOWER } DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } I0703 01:40:04.426681 12066 raft_consensus.cc:1081] T aa3ae539f4c6418496bbb47ff00cb47e P 655d2894f7d14417bffae7af9cca3733 [term 1 FOLLOWER]: Refusing update from remote peer ea5f6a64e88b44c58375146ddaaac0b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0703 01:40:04.427472 12257 consensus_queue.cc:962] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655d2894f7d14417bffae7af9cca3733" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 50417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0703 01:40:04.434700 12188 raft_consensus.cc:1081] T aa3ae539f4c6418496bbb47ff00cb47e P 01a76da15ef14c80b66d2706c0978eb8 [term 1 FOLLOWER]: Refusing update from remote peer ea5f6a64e88b44c58375146ddaaac0b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0703 01:40:04.436440 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 11990 I0703 01:40:04.436792 12257 consensus_queue.cc:962] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01a76da15ef14c80b66d2706c0978eb8" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0703 01:40:04.437134 12106 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:40:04.482031 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 11990 W0703 01:40:04.489403 11556 connection.cc:512] server connection from 127.5.240.196:50595 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.196:50595): Connection reset by peer (error 104) I0703 01:40:04.489595 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 12112 W0703 01:40:04.489773 11755 connection.cc:512] client connection to 127.5.240.196:50417 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.196:50417): Connection reset by peer (error 104) I0703 01:40:04.490411 12228 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:40:04.533599 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 12112 W0703 01:40:04.541095 11756 connection.cc:512] client connection to 127.5.240.197:43693 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.197:43693): Connection reset by peer (error 104) W0703 01:40:04.541213 11557 connection.cc:512] server connection from 127.5.240.197:36749 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.197:36749): Connection reset by peer (error 104) I0703 01:40:04.541689 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master pid 11549 I0703 01:40:04.542398 11605 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:40:04.611344 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master with pid 11549 W0703 01:40:04.624240 11878 connection.cc:512] client connection to 127.5.240.254:45595 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.254:45595): Connection reset by peer (error 104) W0703 01:40:04.624243 11633 connection.cc:512] client connection to 127.5.240.254:45595 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.254:45595): Connection reset by peer (error 104) W0703 01:40:04.624828 11756 connection.cc:512] client connection to 127.5.240.254:45595 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.254:45595): Connection reset by peer (error 104) I0703 01:40:04.624843 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:45595 --webserver_interface=127.5.240.254 --webserver_port=41699 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --rpc_reuseport=true with env {} W0703 01:40:04.731763 11987 heartbeater.cc:587] Failed to heartbeat to 127.5.240.254:45595: Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.240.254:45595: connect: Connection refused (error 111) WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:40:04.804572 12274 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:40:04.804800 12274 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:40:04.806988 12274 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0703 01:40:04.807029 12274 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:40:04.807061 12274 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0703 01:40:04.807093 12274 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0703 01:40:04.815130 12274 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:45595 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.5.240.254 --webserver_port=41699 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:40:04.820802 12274 master_main.cc:80] Initializing master server... I0703 01:40:04.821868 12274 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 155708us I0703 01:40:04.822921 12274 fs_manager.cc:261] Metadata directory not provided I0703 01:40:04.822978 12274 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:40:04.830612 12274 fs_manager.cc:397] Time spent opening directory manager: real 0.005s user 0.002s sys 0.002s I0703 01:40:04.831451 12274 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:40:04.831893 12274 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:40:04.835618 12274 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s I0703 01:40:04.835692 12274 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal uuid: "a5ace5d56f5f4273b353e57fa3bfaa3e" format_stamp: "Formatted at 2018-07-03 08:40:01 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:04.836555 12274 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:40:04.870230 12274 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:40:04.871664 12274 master_main.cc:83] Starting Master server... I0703 01:40:04.908630 12274 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.254:45595 I0703 01:40:04.909088 12274 webserver.cc:175] Starting webserver on 127.5.240.254:41699 I0703 01:40:04.909133 12274 webserver.cc:186] Document root disabled I0703 01:40:04.909826 12274 webserver.cc:313] Webserver started. Bound to: http://127.5.240.254:41699/ I0703 01:40:04.910490 12274 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb I0703 01:40:04.915551 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master as pid 12274 I0703 01:40:04.917091 11822 consensus_peers.cc:455] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 -> Peer 01a76da15ef14c80b66d2706c0978eb8 (127.5.240.197:43693): Closing peer: 01a76da15ef14c80b66d2706c0978eb8 I0703 01:40:04.917174 11822 consensus_peers.cc:455] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 -> Peer 655d2894f7d14417bffae7af9cca3733 (127.5.240.196:50417): Closing peer: 655d2894f7d14417bffae7af9cca3733 I0703 01:40:04.917449 11822 consensus_queue.cc:204] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "655d2894f7d14417bffae7af9cca3733" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 50417 } } peers { permanent_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 37351 } } W0703 01:40:04.919242 11755 consensus_peers.cc:433] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 -> Peer 655d2894f7d14417bffae7af9cca3733 (127.5.240.196:50417): Couldn't send request to peer 655d2894f7d14417bffae7af9cca3733 for tablet aa3ae539f4c6418496bbb47ff00cb47e. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.196:50417: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. I0703 01:40:04.920568 12334 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P a5ace5d56f5f4273b353e57fa3bfaa3e: Bootstrap starting. I0703 01:40:04.932605 12334 log.cc:525] T 00000000000000000000000000000000 P a5ace5d56f5f4273b353e57fa3bfaa3e: Log is configured to *not* fsync() on all Append() calls I0703 01:40:04.941689 12334 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P a5ace5d56f5f4273b353e57fa3bfaa3e: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=1} inserts{seen=4 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:40:04.942100 12334 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P a5ace5d56f5f4273b353e57fa3bfaa3e: Bootstrap complete. I0703 01:40:04.951514 12334 raft_consensus.cc:325] T 00000000000000000000000000000000 P a5ace5d56f5f4273b353e57fa3bfaa3e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5ace5d56f5f4273b353e57fa3bfaa3e" member_type: VOTER } I0703 01:40:04.952242 12334 raft_consensus.cc:628] T 00000000000000000000000000000000 P a5ace5d56f5f4273b353e57fa3bfaa3e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5ace5d56f5f4273b353e57fa3bfaa3e, State: Initialized, Role: FOLLOWER I0703 01:40:04.952754 12334 consensus_queue.cc:227] T 00000000000000000000000000000000 P a5ace5d56f5f4273b353e57fa3bfaa3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5ace5d56f5f4273b353e57fa3bfaa3e" member_type: VOTER } I0703 01:40:04.952888 12334 raft_consensus.cc:365] T 00000000000000000000000000000000 P a5ace5d56f5f4273b353e57fa3bfaa3e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:40:04.952962 12334 raft_consensus.cc:457] T 00000000000000000000000000000000 P a5ace5d56f5f4273b353e57fa3bfaa3e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:40:04.953038 12334 raft_consensus.cc:2730] T 00000000000000000000000000000000 P a5ace5d56f5f4273b353e57fa3bfaa3e: Snoozing failure detection for 1.967s (starting election) I0703 01:40:04.953119 12334 raft_consensus.cc:2782] T 00000000000000000000000000000000 P a5ace5d56f5f4273b353e57fa3bfaa3e [term 1 FOLLOWER]: Advancing to term 2 I0703 01:40:04.956490 12334 raft_consensus.cc:479] T 00000000000000000000000000000000 P a5ace5d56f5f4273b353e57fa3bfaa3e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5ace5d56f5f4273b353e57fa3bfaa3e" member_type: VOTER } I0703 01:40:04.956852 12334 leader_election.cc:270] T 00000000000000000000000000000000 P a5ace5d56f5f4273b353e57fa3bfaa3e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0703 01:40:04.957657 12337 raft_consensus.cc:2730] T 00000000000000000000000000000000 P a5ace5d56f5f4273b353e57fa3bfaa3e: Snoozing failure detection for 2.102s (election complete) I0703 01:40:04.957846 12337 raft_consensus.cc:2537] T 00000000000000000000000000000000 P a5ace5d56f5f4273b353e57fa3bfaa3e [term 2 FOLLOWER]: Leader election won for term 2 I0703 01:40:04.960717 12337 raft_consensus.cc:593] T 00000000000000000000000000000000 P a5ace5d56f5f4273b353e57fa3bfaa3e [term 2 LEADER]: Becoming Leader. State: Replica: a5ace5d56f5f4273b353e57fa3bfaa3e, State: Running, Role: LEADER I0703 01:40:04.961561 12337 consensus_queue.cc:204] T 00000000000000000000000000000000 P a5ace5d56f5f4273b353e57fa3bfaa3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5ace5d56f5f4273b353e57fa3bfaa3e" member_type: VOTER } I0703 01:40:04.964689 12334 sys_catalog.cc:337] T 00000000000000000000000000000000 P a5ace5d56f5f4273b353e57fa3bfaa3e [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 2 leader_uuid: "a5ace5d56f5f4273b353e57fa3bfaa3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5ace5d56f5f4273b353e57fa3bfaa3e" member_type: VOTER } } I0703 01:40:04.965042 12334 sys_catalog.cc:340] T 00000000000000000000000000000000 P a5ace5d56f5f4273b353e57fa3bfaa3e [sys.catalog]: This master's current role is: LEADER I0703 01:40:04.964582 12338 sys_catalog.cc:337] T 00000000000000000000000000000000 P a5ace5d56f5f4273b353e57fa3bfaa3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a5ace5d56f5f4273b353e57fa3bfaa3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5ace5d56f5f4273b353e57fa3bfaa3e" member_type: VOTER } } I0703 01:40:04.965771 12338 sys_catalog.cc:340] T 00000000000000000000000000000000 P a5ace5d56f5f4273b353e57fa3bfaa3e [sys.catalog]: This master's current role is: LEADER I0703 01:40:04.965981 12340 sys_catalog.cc:337] T 00000000000000000000000000000000 P a5ace5d56f5f4273b353e57fa3bfaa3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5ace5d56f5f4273b353e57fa3bfaa3e. Latest consensus state: current_term: 2 leader_uuid: "a5ace5d56f5f4273b353e57fa3bfaa3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5ace5d56f5f4273b353e57fa3bfaa3e" member_type: VOTER } } I0703 01:40:04.968739 12334 sys_catalog.cc:424] T 00000000000000000000000000000000 P a5ace5d56f5f4273b353e57fa3bfaa3e [sys.catalog]: configured and running, proceeding with master startup. I0703 01:40:04.968943 12340 sys_catalog.cc:340] T 00000000000000000000000000000000 P a5ace5d56f5f4273b353e57fa3bfaa3e [sys.catalog]: This master's current role is: LEADER I0703 01:40:04.969152 12343 catalog_manager.cc:1006] Loading table and tablet metadata into memory... I0703 01:40:04.970985 12274 master_main.cc:86] Master server successfully started. I0703 01:40:04.976804 12343 catalog_manager.cc:323] Loaded metadata for table TestTable [id=eade82a1ab6a4ddbb599737edcdea82f] I0703 01:40:04.980062 12301 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "ee1b80f8727c4321a63259acf818aff8" instance_seqno: 1530607201908992) as {username='jenkins'} at 127.5.240.193:34169; Asking this server to re-register. I0703 01:40:04.980572 12343 catalog_manager.cc:377] Loaded metadata for tablet aa3ae539f4c6418496bbb47ff00cb47e (table TestTable [id=eade82a1ab6a4ddbb599737edcdea82f]) I0703 01:40:04.980828 12343 catalog_manager.cc:1017] Initializing Kudu internal certificate authority... I0703 01:40:04.981163 11742 heartbeater.cc:426] Registering TS with master... I0703 01:40:04.982373 12301 ts_manager.cc:98] Registered new tserver with Master: ee1b80f8727c4321a63259acf818aff8 (127.5.240.193:46472) I0703 01:40:04.986021 12343 catalog_manager.cc:1026] Loading token signing keys... I0703 01:40:04.987540 12343 catalog_manager.cc:4113] T 00000000000000000000000000000000 P a5ace5d56f5f4273b353e57fa3bfaa3e: Loaded TSK: 0 W0703 01:40:05.328781 11755 consensus_peers.cc:433] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 -> Peer 655d2894f7d14417bffae7af9cca3733 (127.5.240.196:50417): Couldn't send request to peer 655d2894f7d14417bffae7af9cca3733 for tablet aa3ae539f4c6418496bbb47ff00cb47e. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.196:50417: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. I0703 01:40:05.441164 12301 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" instance_seqno: 1530607202251974) as {username='jenkins'} at 127.5.240.194:33054; Asking this server to re-register. I0703 01:40:05.441675 11865 heartbeater.cc:426] Registering TS with master... I0703 01:40:05.441790 11865 heartbeater.cc:475] Master 127.5.240.254:45595 requested a full tablet report, sending... I0703 01:40:05.442749 12301 ts_manager.cc:98] Registered new tserver with Master: ea5f6a64e88b44c58375146ddaaac0b5 (127.5.240.194:37351) I0703 01:40:05.736788 11987 heartbeater.cc:346] Connected to a master server at 127.5.240.254:45595 I0703 01:40:05.737486 12301 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "7e8e64942ea3471ab77a706b397f0c4f" instance_seqno: 1530607202689327) as {username='jenkins'} at 127.5.240.195:53825; Asking this server to re-register. I0703 01:40:05.738152 11987 heartbeater.cc:426] Registering TS with master... I0703 01:40:05.738270 11987 heartbeater.cc:475] Master 127.5.240.254:45595 requested a full tablet report, sending... I0703 01:40:05.738984 12301 ts_manager.cc:98] Registered new tserver with Master: 7e8e64942ea3471ab77a706b397f0c4f (127.5.240.195:41189) W0703 01:40:05.937243 11755 consensus_peers.cc:433] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 -> Peer 655d2894f7d14417bffae7af9cca3733 (127.5.240.196:50417): Couldn't send request to peer 655d2894f7d14417bffae7af9cca3733 for tablet aa3ae539f4c6418496bbb47ff00cb47e. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.196:50417: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times. W0703 01:40:06.378104 11755 consensus_peers.cc:433] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 -> Peer 655d2894f7d14417bffae7af9cca3733 (127.5.240.196:50417): Couldn't send request to peer 655d2894f7d14417bffae7af9cca3733 for tablet aa3ae539f4c6418496bbb47ff00cb47e. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.196:50417: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 4 times. W0703 01:40:06.933826 11755 consensus_peers.cc:433] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 -> Peer 655d2894f7d14417bffae7af9cca3733 (127.5.240.196:50417): Couldn't send request to peer 655d2894f7d14417bffae7af9cca3733 for tablet aa3ae539f4c6418496bbb47ff00cb47e. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.196:50417: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 5 times. I0703 01:40:06.984833 11742 heartbeater.cc:467] Master 127.5.240.254:45595 was elected leader, sending a full tablet report... I0703 01:40:07.186273 11822 tablet_service.cc:1140] Received LeaderStepDown RPC: tablet_id: "aa3ae539f4c6418496bbb47ff00cb47e" dest_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" from {username='jenkins'} at 127.5.240.194:53267 I0703 01:40:07.186455 11822 raft_consensus.cc:552] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 [term 1 LEADER]: Received request to step down I0703 01:40:07.186511 11822 raft_consensus.cc:628] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ea5f6a64e88b44c58375146ddaaac0b5, State: Running, Role: LEADER I0703 01:40:07.186764 11822 consensus_queue.cc:227] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "655d2894f7d14417bffae7af9cca3733" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 50417 } } peers { permanent_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 37351 } } I0703 01:40:07.186882 11822 consensus_peers.cc:455] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 -> Peer 655d2894f7d14417bffae7af9cca3733 (127.5.240.196:50417): Closing peer: 655d2894f7d14417bffae7af9cca3733 I0703 01:40:07.186975 11822 raft_consensus.cc:2730] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5: Snoozing failure detection for 3.000s (explicit stepdown request) I0703 01:40:07.231736 6083 kudu-admin-test.cc:887] Change Config Op timed out, Sending a Replace config command when change config op is pending on the leader. W0703 01:40:07.443881 12367 flags.cc:406] Enabled unsafe flag: --never_fsync=true I0703 01:40:07.492658 11822 tablet_service.cc:1077] Received UnsafeChangeConfig RPC: dest_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" tablet_id: "aa3ae539f4c6418496bbb47ff00cb47e" caller_id: "kudu-tools" new_config { peers { permanent_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" } } from {username='jenkins'} at 127.5.240.194:53269 W0703 01:40:07.492888 11822 raft_consensus.cc:1918] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 [term 1 FOLLOWER]: Replica has a pending config, but the new config will be unsafely changed anyway. Currently pending config on the node: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "655d2894f7d14417bffae7af9cca3733" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 50417 } } peers { permanent_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 37351 } } W0703 01:40:07.493212 11822 raft_consensus.cc:2017] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 [term 1 FOLLOWER]: PROCEEDING WITH UNSAFE CONFIG CHANGE ON THIS SERVER, COMMITTED CONFIG: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655d2894f7d14417bffae7af9cca3733" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 50417 } } peers { permanent_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 37351 } } peers { permanent_uuid: "01a76da15ef14c80b66d2706c0978eb8" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43693 } }NEW CONFIG: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 37351 } } unsafe_config_change: true I0703 01:40:07.493568 11822 raft_consensus.cc:2782] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 [term 1 FOLLOWER]: Advancing to term 2 I0703 01:40:07.495733 11822 raft_consensus.cc:2849] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 [term 2 FOLLOWER]: Allowing unsafe config change even though there is a pending config! Existing pending config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "655d2894f7d14417bffae7af9cca3733" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 50417 } } peers { permanent_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 37351 } }; New pending config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 37351 } } unsafe_config_change: true I0703 01:40:08.996093 12384 raft_consensus.cc:457] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader kudu-tools) I0703 01:40:08.996179 12384 raft_consensus.cc:2730] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5: Snoozing failure detection for 2.096s (starting election) I0703 01:40:08.996281 12384 raft_consensus.cc:479] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 37351 } } unsafe_config_change: true I0703 01:40:08.996486 12384 leader_election.cc:270] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. I0703 01:40:08.996863 12384 raft_consensus.cc:2730] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5: Snoozing failure detection for 2.221s (election complete) I0703 01:40:08.996922 12384 raft_consensus.cc:2537] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 [term 2 FOLLOWER]: Leader pre-election won for term 3 I0703 01:40:08.996976 12384 raft_consensus.cc:457] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader kudu-tools) I0703 01:40:08.997025 12384 raft_consensus.cc:2730] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5: Snoozing failure detection for 1.928s (starting election) I0703 01:40:08.997105 12384 raft_consensus.cc:2782] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 [term 2 FOLLOWER]: Advancing to term 3 I0703 01:40:08.998450 12384 raft_consensus.cc:479] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 37351 } } unsafe_config_change: true I0703 01:40:08.998616 12384 leader_election.cc:270] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. I0703 01:40:08.998803 12385 raft_consensus.cc:2730] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5: Snoozing failure detection for 1.700s (election complete) I0703 01:40:08.998867 12385 raft_consensus.cc:2537] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 [term 3 FOLLOWER]: Leader election won for term 3 I0703 01:40:08.998944 12385 raft_consensus.cc:593] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 [term 3 LEADER]: Becoming Leader. State: Replica: ea5f6a64e88b44c58375146ddaaac0b5, State: Running, Role: LEADER I0703 01:40:08.999205 12385 consensus_queue.cc:204] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.3, Last appended by leader: 0, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 37351 } } unsafe_config_change: true I0703 01:40:09.000326 12385 raft_consensus.cc:2684] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 [term 3 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 01a76da15ef14c80b66d2706c0978eb8 (127.5.240.197) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "655d2894f7d14417bffae7af9cca3733" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 50417 } } peers { permanent_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 37351 } } } I0703 01:40:09.000710 12301 catalog_manager.cc:3798] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 reported cstate change: term changed from 1 to 3, now has a pending config: VOTER ea5f6a64e88b44c58375146ddaaac0b5 (127.5.240.194). New cstate: current_term: 3 leader_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "655d2894f7d14417bffae7af9cca3733" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 50417 } } peers { permanent_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 37351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01a76da15ef14c80b66d2706c0978eb8" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43693 } } } pending_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 37351 } } unsafe_config_change: true } W0703 01:40:09.002589 12385 rpcz_store.cc:251] Call kudu.consensus.ConsensusService.ChangeConfig from 127.5.240.194:53265 (request call id 35) took 4086ms (client timeout 2000). W0703 01:40:09.003171 12385 rpcz_store.cc:255] Trace: 0703 01:40:04.916338 (+ 0us) service_pool.cc:162] Inserting onto call queue 0703 01:40:04.916394 (+ 56us) service_pool.cc:221] Handling call 0703 01:40:04.918006 (+ 1612us) log.cc:587] Serialized 379 byte log entry 0703 01:40:09.002509 (+4084503us) inbound_call.cc:162] Queueing success response Metrics: {"dns_us":61,"raft.queue_time_us":58,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":146} I0703 01:40:09.003438 12385 raft_consensus.cc:2684] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 [term 3 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER 01a76da15ef14c80b66d2706c0978eb8 (127.5.240.197) evicted, VOTER 655d2894f7d14417bffae7af9cca3733 (127.5.240.196) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 37351 } } unsafe_config_change: true } I0703 01:40:09.009483 12301 catalog_manager.cc:3798] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 reported cstate change: config changed from index -1 to 3, VOTER 01a76da15ef14c80b66d2706c0978eb8 (127.5.240.197) evicted, VOTER 655d2894f7d14417bffae7af9cca3733 (127.5.240.196) evicted, no longer has a pending config: VOTER ea5f6a64e88b44c58375146ddaaac0b5 (127.5.240.194). New cstate: current_term: 3 leader_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 37351 } health_report { overall_health: HEALTHY } } unsafe_config_change: true } I0703 01:40:09.011432 12301 catalog_manager.cc:2903] Scheduling retry of aa3ae539f4c6418496bbb47ff00cb47e Delete Tablet RPC for TS=655d2894f7d14417bffae7af9cca3733 with a delay of 26 ms (attempt = 1) W0703 01:40:09.011535 12301 catalog_manager.cc:3957] Failed to send aa3ae539f4c6418496bbb47ff00cb47e Delete Tablet RPC for TS=655d2894f7d14417bffae7af9cca3733: Not found: failed to reset TS proxy: Could not find TS for UUID 655d2894f7d14417bffae7af9cca3733 I0703 01:40:09.011662 12301 catalog_manager.cc:2903] Scheduling retry of aa3ae539f4c6418496bbb47ff00cb47e Delete Tablet RPC for TS=01a76da15ef14c80b66d2706c0978eb8 with a delay of 21 ms (attempt = 1) W0703 01:40:09.011718 12301 catalog_manager.cc:3957] Failed to send aa3ae539f4c6418496bbb47ff00cb47e Delete Tablet RPC for TS=01a76da15ef14c80b66d2706c0978eb8: Not found: failed to reset TS proxy: Could not find TS for UUID 01a76da15ef14c80b66d2706c0978eb8 I0703 01:40:09.012073 12301 catalog_manager.cc:3437] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet aa3ae539f4c6418496bbb47ff00cb47e (attempt 1) I0703 01:40:09.017705 11822 consensus_queue.cc:204] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 0, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 37351 } } peers { permanent_uuid: "7e8e64942ea3471ab77a706b397f0c4f" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 41189 } attrs { promote: true } } unsafe_config_change: true I0703 01:40:09.018522 12384 raft_consensus.cc:2684] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 3 to 5, NON_VOTER 7e8e64942ea3471ab77a706b397f0c4f (127.5.240.195) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 37351 } } peers { permanent_uuid: "7e8e64942ea3471ab77a706b397f0c4f" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 41189 } attrs { promote: true } } unsafe_config_change: true } I0703 01:40:09.021234 12283 catalog_manager.cc:3352] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet aa3ae539f4c6418496bbb47ff00cb47e with cas_config_opid_index 3: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0703 01:40:09.021977 12301 catalog_manager.cc:3798] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 reported cstate change: config changed from index 3 to 5, NON_VOTER 7e8e64942ea3471ab77a706b397f0c4f (127.5.240.195) added. New cstate: current_term: 3 leader_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 37351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e8e64942ea3471ab77a706b397f0c4f" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 41189 } attrs { promote: true } health_report { overall_health: UNKNOWN } } unsafe_config_change: true } I0703 01:40:09.024096 12301 catalog_manager.cc:3437] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet aa3ae539f4c6418496bbb47ff00cb47e (attempt 1) I0703 01:40:09.024888 11822 consensus_peers.cc:455] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 -> Peer 7e8e64942ea3471ab77a706b397f0c4f (127.5.240.195:41189): Closing peer: 7e8e64942ea3471ab77a706b397f0c4f I0703 01:40:09.025009 11822 consensus_queue.cc:204] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 0, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 37351 } } peers { permanent_uuid: "7e8e64942ea3471ab77a706b397f0c4f" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 41189 } attrs { promote: true } } peers { permanent_uuid: "ee1b80f8727c4321a63259acf818aff8" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 46472 } attrs { promote: true } } unsafe_config_change: true I0703 01:40:09.026204 12384 raft_consensus.cc:2684] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 [term 3 LEADER]: Committing config change with OpId 3.6: config changed from index 5 to 6, NON_VOTER ee1b80f8727c4321a63259acf818aff8 (127.5.240.193) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 37351 } } peers { permanent_uuid: "7e8e64942ea3471ab77a706b397f0c4f" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 41189 } attrs { promote: true } } peers { permanent_uuid: "ee1b80f8727c4321a63259acf818aff8" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 46472 } attrs { promote: true } } unsafe_config_change: true } W0703 01:40:09.027638 11754 consensus_peers.cc:433] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 -> Peer 7e8e64942ea3471ab77a706b397f0c4f (127.5.240.195:41189): Couldn't send request to peer 7e8e64942ea3471ab77a706b397f0c4f for tablet aa3ae539f4c6418496bbb47ff00cb47e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aa3ae539f4c6418496bbb47ff00cb47e. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:40:09.028645 12283 catalog_manager.cc:3352] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet aa3ae539f4c6418496bbb47ff00cb47e with cas_config_opid_index 5: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0703 01:40:09.030750 12301 catalog_manager.cc:3798] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 reported cstate change: config changed from index 5 to 6, NON_VOTER ee1b80f8727c4321a63259acf818aff8 (127.5.240.193) added. New cstate: current_term: 3 leader_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 37351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e8e64942ea3471ab77a706b397f0c4f" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 41189 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee1b80f8727c4321a63259acf818aff8" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 46472 } attrs { promote: true } health_report { overall_health: UNKNOWN } } unsafe_config_change: true } W0703 01:40:09.032871 11756 consensus_peers.cc:433] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 -> Peer ee1b80f8727c4321a63259acf818aff8 (127.5.240.193:46472): Couldn't send request to peer ee1b80f8727c4321a63259acf818aff8 for tablet aa3ae539f4c6418496bbb47ff00cb47e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aa3ae539f4c6418496bbb47ff00cb47e. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:40:09.033140 12284 catalog_manager.cc:2903] Scheduling retry of aa3ae539f4c6418496bbb47ff00cb47e Delete Tablet RPC for TS=01a76da15ef14c80b66d2706c0978eb8 with a delay of 50 ms (attempt = 2) W0703 01:40:09.033210 12284 catalog_manager.cc:2922] Async tablet task aa3ae539f4c6418496bbb47ff00cb47e Delete Tablet RPC for TS=01a76da15ef14c80b66d2706c0978eb8 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 01a76da15ef14c80b66d2706c0978eb8 I0703 01:40:09.037905 12282 catalog_manager.cc:2903] Scheduling retry of aa3ae539f4c6418496bbb47ff00cb47e Delete Tablet RPC for TS=655d2894f7d14417bffae7af9cca3733 with a delay of 43 ms (attempt = 2) W0703 01:40:09.037956 12282 catalog_manager.cc:2922] Async tablet task aa3ae539f4c6418496bbb47ff00cb47e Delete Tablet RPC for TS=655d2894f7d14417bffae7af9cca3733 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 655d2894f7d14417bffae7af9cca3733 I0703 01:40:09.081552 12282 catalog_manager.cc:2903] Scheduling retry of aa3ae539f4c6418496bbb47ff00cb47e Delete Tablet RPC for TS=655d2894f7d14417bffae7af9cca3733 with a delay of 106 ms (attempt = 3) W0703 01:40:09.081599 12282 catalog_manager.cc:2922] Async tablet task aa3ae539f4c6418496bbb47ff00cb47e Delete Tablet RPC for TS=655d2894f7d14417bffae7af9cca3733 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 655d2894f7d14417bffae7af9cca3733 I0703 01:40:09.083729 12284 catalog_manager.cc:2903] Scheduling retry of aa3ae539f4c6418496bbb47ff00cb47e Delete Tablet RPC for TS=01a76da15ef14c80b66d2706c0978eb8 with a delay of 96 ms (attempt = 3) W0703 01:40:09.083783 12284 catalog_manager.cc:2922] Async tablet task aa3ae539f4c6418496bbb47ff00cb47e Delete Tablet RPC for TS=01a76da15ef14c80b66d2706c0978eb8 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 01a76da15ef14c80b66d2706c0978eb8 I0703 01:40:09.180379 12284 catalog_manager.cc:2903] Scheduling retry of aa3ae539f4c6418496bbb47ff00cb47e Delete Tablet RPC for TS=01a76da15ef14c80b66d2706c0978eb8 with a delay of 174 ms (attempt = 4) W0703 01:40:09.180435 12284 catalog_manager.cc:2922] Async tablet task aa3ae539f4c6418496bbb47ff00cb47e Delete Tablet RPC for TS=01a76da15ef14c80b66d2706c0978eb8 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 01a76da15ef14c80b66d2706c0978eb8 I0703 01:40:09.188210 12282 catalog_manager.cc:2903] Scheduling retry of aa3ae539f4c6418496bbb47ff00cb47e Delete Tablet RPC for TS=655d2894f7d14417bffae7af9cca3733 with a delay of 157 ms (attempt = 4) W0703 01:40:09.188257 12282 catalog_manager.cc:2922] Async tablet task aa3ae539f4c6418496bbb47ff00cb47e Delete Tablet RPC for TS=655d2894f7d14417bffae7af9cca3733 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 655d2894f7d14417bffae7af9cca3733 I0703 01:40:09.346237 12282 catalog_manager.cc:2903] Scheduling retry of aa3ae539f4c6418496bbb47ff00cb47e Delete Tablet RPC for TS=655d2894f7d14417bffae7af9cca3733 with a delay of 277 ms (attempt = 5) W0703 01:40:09.346309 12282 catalog_manager.cc:2922] Async tablet task aa3ae539f4c6418496bbb47ff00cb47e Delete Tablet RPC for TS=655d2894f7d14417bffae7af9cca3733 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 655d2894f7d14417bffae7af9cca3733 I0703 01:40:09.355401 12284 catalog_manager.cc:2903] Scheduling retry of aa3ae539f4c6418496bbb47ff00cb47e Delete Tablet RPC for TS=01a76da15ef14c80b66d2706c0978eb8 with a delay of 276 ms (attempt = 5) W0703 01:40:09.355448 12284 catalog_manager.cc:2922] Async tablet task aa3ae539f4c6418496bbb47ff00cb47e Delete Tablet RPC for TS=01a76da15ef14c80b66d2706c0978eb8 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 01a76da15ef14c80b66d2706c0978eb8 I0703 01:40:09.494684 12397 ts_tablet_manager.cc:669] T aa3ae539f4c6418496bbb47ff00cb47e P 7e8e64942ea3471ab77a706b397f0c4f: Initiating tablet copy from peer ea5f6a64e88b44c58375146ddaaac0b5 (127.5.240.194:37351) I0703 01:40:09.494951 12397 tablet_copy_client.cc:239] T aa3ae539f4c6418496bbb47ff00cb47e P 7e8e64942ea3471ab77a706b397f0c4f: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.194:37351 I0703 01:40:09.500932 11842 tablet_copy_service.cc:135] P ea5f6a64e88b44c58375146ddaaac0b5: Received BeginTabletCopySession request for tablet aa3ae539f4c6418496bbb47ff00cb47e from peer 7e8e64942ea3471ab77a706b397f0c4f ({username='jenkins'} at 127.5.240.195:59298) I0703 01:40:09.501078 11842 tablet_copy_service.cc:156] P ea5f6a64e88b44c58375146ddaaac0b5: Beginning new tablet copy session on tablet aa3ae539f4c6418496bbb47ff00cb47e from peer 7e8e64942ea3471ab77a706b397f0c4f at {username='jenkins'} at 127.5.240.195:59298: session id = 7e8e64942ea3471ab77a706b397f0c4f-aa3ae539f4c6418496bbb47ff00cb47e I0703 01:40:09.503057 11842 tablet_copy_source_session.cc:202] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:40:09.505761 12397 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet aa3ae539f4c6418496bbb47ff00cb47e. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:40:09.516404 12397 ts_tablet_manager.cc:1173] T aa3ae539f4c6418496bbb47ff00cb47e P 7e8e64942ea3471ab77a706b397f0c4f: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:40:09.516688 12397 tablet_copy_client.cc:566] T aa3ae539f4c6418496bbb47ff00cb47e P 7e8e64942ea3471ab77a706b397f0c4f: tablet copy: Starting download of 0 data blocks... I0703 01:40:09.516875 12397 tablet_copy_client.cc:529] T aa3ae539f4c6418496bbb47ff00cb47e P 7e8e64942ea3471ab77a706b397f0c4f: tablet copy: Starting download of 1 WAL segments... I0703 01:40:09.518937 12397 tablet_copy_client.cc:405] T aa3ae539f4c6418496bbb47ff00cb47e P 7e8e64942ea3471ab77a706b397f0c4f: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:40:09.520606 12397 ts_tablet_manager.cc:1028] T aa3ae539f4c6418496bbb47ff00cb47e P 7e8e64942ea3471ab77a706b397f0c4f: Bootstrapping tablet I0703 01:40:09.520826 12397 tablet_bootstrap.cc:427] T aa3ae539f4c6418496bbb47ff00cb47e P 7e8e64942ea3471ab77a706b397f0c4f: Bootstrap starting. I0703 01:40:09.526629 12397 log.cc:525] T aa3ae539f4c6418496bbb47ff00cb47e P 7e8e64942ea3471ab77a706b397f0c4f: Log is configured to *not* fsync() on all Append() calls I0703 01:40:09.530829 12397 tablet_bootstrap.cc:427] T aa3ae539f4c6418496bbb47ff00cb47e P 7e8e64942ea3471ab77a706b397f0c4f: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:40:09.531141 12397 tablet_bootstrap.cc:427] T aa3ae539f4c6418496bbb47ff00cb47e P 7e8e64942ea3471ab77a706b397f0c4f: Bootstrap complete. I0703 01:40:09.531350 12397 ts_tablet_manager.cc:1045] T aa3ae539f4c6418496bbb47ff00cb47e P 7e8e64942ea3471ab77a706b397f0c4f: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s I0703 01:40:09.536298 12397 raft_consensus.cc:325] T aa3ae539f4c6418496bbb47ff00cb47e P 7e8e64942ea3471ab77a706b397f0c4f [term 3 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 37351 } } peers { permanent_uuid: "7e8e64942ea3471ab77a706b397f0c4f" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 41189 } attrs { promote: true } } peers { permanent_uuid: "ee1b80f8727c4321a63259acf818aff8" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 46472 } attrs { promote: true } } unsafe_config_change: true I0703 01:40:09.536639 12397 raft_consensus.cc:628] T aa3ae539f4c6418496bbb47ff00cb47e P 7e8e64942ea3471ab77a706b397f0c4f [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 7e8e64942ea3471ab77a706b397f0c4f, State: Initialized, Role: LEARNER I0703 01:40:09.536881 12397 consensus_queue.cc:227] T aa3ae539f4c6418496bbb47ff00cb47e P 7e8e64942ea3471ab77a706b397f0c4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 37351 } } peers { permanent_uuid: "7e8e64942ea3471ab77a706b397f0c4f" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 41189 } attrs { promote: true } } peers { permanent_uuid: "ee1b80f8727c4321a63259acf818aff8" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 46472 } attrs { promote: true } } unsafe_config_change: true I0703 01:40:09.538275 12397 ts_tablet_manager.cc:1073] T aa3ae539f4c6418496bbb47ff00cb47e P 7e8e64942ea3471ab77a706b397f0c4f: Time spent starting tablet: real 0.007s user 0.004s sys 0.003s I0703 01:40:09.538918 11842 tablet_copy_service.cc:325] P ea5f6a64e88b44c58375146ddaaac0b5: Request end of tablet copy session 7e8e64942ea3471ab77a706b397f0c4f-aa3ae539f4c6418496bbb47ff00cb47e received from {username='jenkins'} at 127.5.240.195:59298 I0703 01:40:09.539001 11842 tablet_copy_service.cc:413] P ea5f6a64e88b44c58375146ddaaac0b5: ending tablet copy session 7e8e64942ea3471ab77a706b397f0c4f-aa3ae539f4c6418496bbb47ff00cb47e on tablet aa3ae539f4c6418496bbb47ff00cb47e with peer 7e8e64942ea3471ab77a706b397f0c4f I0703 01:40:09.624275 12282 catalog_manager.cc:2903] Scheduling retry of aa3ae539f4c6418496bbb47ff00cb47e Delete Tablet RPC for TS=655d2894f7d14417bffae7af9cca3733 with a delay of 546 ms (attempt = 6) W0703 01:40:09.624336 12282 catalog_manager.cc:2922] Async tablet task aa3ae539f4c6418496bbb47ff00cb47e Delete Tablet RPC for TS=655d2894f7d14417bffae7af9cca3733 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 655d2894f7d14417bffae7af9cca3733 I0703 01:40:09.632522 12284 catalog_manager.cc:2903] Scheduling retry of aa3ae539f4c6418496bbb47ff00cb47e Delete Tablet RPC for TS=01a76da15ef14c80b66d2706c0978eb8 with a delay of 534 ms (attempt = 6) W0703 01:40:09.632570 12284 catalog_manager.cc:2922] Async tablet task aa3ae539f4c6418496bbb47ff00cb47e Delete Tablet RPC for TS=01a76da15ef14c80b66d2706c0978eb8 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 01a76da15ef14c80b66d2706c0978eb8 I0703 01:40:09.651365 12401 ts_tablet_manager.cc:669] T aa3ae539f4c6418496bbb47ff00cb47e P ee1b80f8727c4321a63259acf818aff8: Initiating tablet copy from peer ea5f6a64e88b44c58375146ddaaac0b5 (127.5.240.194:37351) I0703 01:40:09.651636 12401 tablet_copy_client.cc:239] T aa3ae539f4c6418496bbb47ff00cb47e P ee1b80f8727c4321a63259acf818aff8: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.194:37351 I0703 01:40:09.656419 11842 tablet_copy_service.cc:135] P ea5f6a64e88b44c58375146ddaaac0b5: Received BeginTabletCopySession request for tablet aa3ae539f4c6418496bbb47ff00cb47e from peer ee1b80f8727c4321a63259acf818aff8 ({username='jenkins'} at 127.5.240.193:54630) I0703 01:40:09.656533 11842 tablet_copy_service.cc:156] P ea5f6a64e88b44c58375146ddaaac0b5: Beginning new tablet copy session on tablet aa3ae539f4c6418496bbb47ff00cb47e from peer ee1b80f8727c4321a63259acf818aff8 at {username='jenkins'} at 127.5.240.193:54630: session id = ee1b80f8727c4321a63259acf818aff8-aa3ae539f4c6418496bbb47ff00cb47e I0703 01:40:09.658083 11842 tablet_copy_source_session.cc:202] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:40:09.660564 12401 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet aa3ae539f4c6418496bbb47ff00cb47e. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:40:09.670922 12401 ts_tablet_manager.cc:1173] T aa3ae539f4c6418496bbb47ff00cb47e P ee1b80f8727c4321a63259acf818aff8: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:40:09.671169 12401 tablet_copy_client.cc:566] T aa3ae539f4c6418496bbb47ff00cb47e P ee1b80f8727c4321a63259acf818aff8: tablet copy: Starting download of 0 data blocks... I0703 01:40:09.671329 12401 tablet_copy_client.cc:529] T aa3ae539f4c6418496bbb47ff00cb47e P ee1b80f8727c4321a63259acf818aff8: tablet copy: Starting download of 1 WAL segments... I0703 01:40:09.672772 12401 tablet_copy_client.cc:405] T aa3ae539f4c6418496bbb47ff00cb47e P ee1b80f8727c4321a63259acf818aff8: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:40:09.674463 12401 ts_tablet_manager.cc:1028] T aa3ae539f4c6418496bbb47ff00cb47e P ee1b80f8727c4321a63259acf818aff8: Bootstrapping tablet I0703 01:40:09.674701 12401 tablet_bootstrap.cc:427] T aa3ae539f4c6418496bbb47ff00cb47e P ee1b80f8727c4321a63259acf818aff8: Bootstrap starting. I0703 01:40:09.679258 12401 log.cc:525] T aa3ae539f4c6418496bbb47ff00cb47e P ee1b80f8727c4321a63259acf818aff8: Log is configured to *not* fsync() on all Append() calls I0703 01:40:09.684012 12401 tablet_bootstrap.cc:427] T aa3ae539f4c6418496bbb47ff00cb47e P ee1b80f8727c4321a63259acf818aff8: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:40:09.684325 12401 tablet_bootstrap.cc:427] T aa3ae539f4c6418496bbb47ff00cb47e P ee1b80f8727c4321a63259acf818aff8: Bootstrap complete. I0703 01:40:09.684553 12401 ts_tablet_manager.cc:1045] T aa3ae539f4c6418496bbb47ff00cb47e P ee1b80f8727c4321a63259acf818aff8: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.003s I0703 01:40:09.689380 12401 raft_consensus.cc:325] T aa3ae539f4c6418496bbb47ff00cb47e P ee1b80f8727c4321a63259acf818aff8 [term 3 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 37351 } } peers { permanent_uuid: "7e8e64942ea3471ab77a706b397f0c4f" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 41189 } attrs { promote: true } } peers { permanent_uuid: "ee1b80f8727c4321a63259acf818aff8" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 46472 } attrs { promote: true } } unsafe_config_change: true I0703 01:40:09.689683 12401 raft_consensus.cc:628] T aa3ae539f4c6418496bbb47ff00cb47e P ee1b80f8727c4321a63259acf818aff8 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: ee1b80f8727c4321a63259acf818aff8, State: Initialized, Role: LEARNER I0703 01:40:09.689929 12401 consensus_queue.cc:227] T aa3ae539f4c6418496bbb47ff00cb47e P ee1b80f8727c4321a63259acf818aff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 37351 } } peers { permanent_uuid: "7e8e64942ea3471ab77a706b397f0c4f" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 41189 } attrs { promote: true } } peers { permanent_uuid: "ee1b80f8727c4321a63259acf818aff8" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 46472 } attrs { promote: true } } unsafe_config_change: true I0703 01:40:09.691643 12401 ts_tablet_manager.cc:1073] T aa3ae539f4c6418496bbb47ff00cb47e P ee1b80f8727c4321a63259acf818aff8: Time spent starting tablet: real 0.007s user 0.004s sys 0.003s I0703 01:40:09.692322 11842 tablet_copy_service.cc:325] P ea5f6a64e88b44c58375146ddaaac0b5: Request end of tablet copy session ee1b80f8727c4321a63259acf818aff8-aa3ae539f4c6418496bbb47ff00cb47e received from {username='jenkins'} at 127.5.240.193:54630 I0703 01:40:09.692389 11842 tablet_copy_service.cc:413] P ea5f6a64e88b44c58375146ddaaac0b5: ending tablet copy session ee1b80f8727c4321a63259acf818aff8-aa3ae539f4c6418496bbb47ff00cb47e on tablet aa3ae539f4c6418496bbb47ff00cb47e with peer ee1b80f8727c4321a63259acf818aff8 W0703 01:40:09.714495 11988 tablet.cc:1678] T aa3ae539f4c6418496bbb47ff00cb47e P 7e8e64942ea3471ab77a706b397f0c4f: Can't schedule compaction. Clean time has not been advanced past its initial value. W0703 01:40:09.937085 11743 tablet.cc:1678] T aa3ae539f4c6418496bbb47ff00cb47e P ee1b80f8727c4321a63259acf818aff8: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:40:10.008782 11944 raft_consensus.cc:1026] T aa3ae539f4c6418496bbb47ff00cb47e P 7e8e64942ea3471ab77a706b397f0c4f [term 3 LEARNER]: Deduplicated request from leader. Original: 3.5->[3.6-3.6] Dedup: 3.6->[] I0703 01:40:10.167779 12284 catalog_manager.cc:2903] Scheduling retry of aa3ae539f4c6418496bbb47ff00cb47e Delete Tablet RPC for TS=01a76da15ef14c80b66d2706c0978eb8 with a delay of 1051 ms (attempt = 7) W0703 01:40:10.167850 12284 catalog_manager.cc:2922] Async tablet task aa3ae539f4c6418496bbb47ff00cb47e Delete Tablet RPC for TS=01a76da15ef14c80b66d2706c0978eb8 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 01a76da15ef14c80b66d2706c0978eb8 I0703 01:40:10.171444 12282 catalog_manager.cc:2903] Scheduling retry of aa3ae539f4c6418496bbb47ff00cb47e Delete Tablet RPC for TS=655d2894f7d14417bffae7af9cca3733 with a delay of 1058 ms (attempt = 7) W0703 01:40:10.171489 12282 catalog_manager.cc:2922] Async tablet task aa3ae539f4c6418496bbb47ff00cb47e Delete Tablet RPC for TS=655d2894f7d14417bffae7af9cca3733 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 655d2894f7d14417bffae7af9cca3733 I0703 01:40:10.256876 11699 raft_consensus.cc:1026] T aa3ae539f4c6418496bbb47ff00cb47e P ee1b80f8727c4321a63259acf818aff8 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.5->[3.6-3.6] Dedup: 3.6->[] I0703 01:40:10.557382 12406 raft_consensus.cc:890] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5: attempting to promote NON_VOTER 7e8e64942ea3471ab77a706b397f0c4f to VOTER I0703 01:40:10.557694 12406 consensus_peers.cc:455] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 -> Peer ee1b80f8727c4321a63259acf818aff8 (127.5.240.193:46472): Closing peer: ee1b80f8727c4321a63259acf818aff8 I0703 01:40:10.557754 12406 consensus_peers.cc:455] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 -> Peer 7e8e64942ea3471ab77a706b397f0c4f (127.5.240.195:41189): Closing peer: 7e8e64942ea3471ab77a706b397f0c4f I0703 01:40:10.557953 12406 consensus_queue.cc:204] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 0, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 37351 } } peers { permanent_uuid: "7e8e64942ea3471ab77a706b397f0c4f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41189 } attrs { promote: false } } peers { permanent_uuid: "ee1b80f8727c4321a63259acf818aff8" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 46472 } attrs { promote: true } } unsafe_config_change: true I0703 01:40:10.560001 11944 raft_consensus.cc:1081] T aa3ae539f4c6418496bbb47ff00cb47e P 7e8e64942ea3471ab77a706b397f0c4f [term 3 LEARNER]: Refusing update from remote peer ea5f6a64e88b44c58375146ddaaac0b5: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch) I0703 01:40:10.560020 11699 raft_consensus.cc:1081] T aa3ae539f4c6418496bbb47ff00cb47e P ee1b80f8727c4321a63259acf818aff8 [term 3 LEARNER]: Refusing update from remote peer ea5f6a64e88b44c58375146ddaaac0b5: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch) I0703 01:40:10.560590 12405 consensus_queue.cc:962] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e8e64942ea3471ab77a706b397f0c4f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I0703 01:40:10.560827 12396 consensus_queue.cc:962] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee1b80f8727c4321a63259acf818aff8" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 46472 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I0703 01:40:10.563797 12406 raft_consensus.cc:2684] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 [term 3 LEADER]: Committing config change with OpId 3.7: config changed from index 6 to 7, 7e8e64942ea3471ab77a706b397f0c4f (127.5.240.195) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 37351 } } peers { permanent_uuid: "7e8e64942ea3471ab77a706b397f0c4f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41189 } attrs { promote: false } } peers { permanent_uuid: "ee1b80f8727c4321a63259acf818aff8" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 46472 } attrs { promote: true } } unsafe_config_change: true } I0703 01:40:10.564474 11699 raft_consensus.cc:2684] T aa3ae539f4c6418496bbb47ff00cb47e P ee1b80f8727c4321a63259acf818aff8 [term 3 LEARNER]: Committing config change with OpId 3.7: config changed from index 6 to 7, 7e8e64942ea3471ab77a706b397f0c4f (127.5.240.195) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 37351 } } peers { permanent_uuid: "7e8e64942ea3471ab77a706b397f0c4f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41189 } attrs { promote: false } } peers { permanent_uuid: "ee1b80f8727c4321a63259acf818aff8" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 46472 } attrs { promote: true } } unsafe_config_change: true } I0703 01:40:10.564507 11944 raft_consensus.cc:2684] T aa3ae539f4c6418496bbb47ff00cb47e P 7e8e64942ea3471ab77a706b397f0c4f [term 3 FOLLOWER]: Committing config change with OpId 3.7: config changed from index 6 to 7, 7e8e64942ea3471ab77a706b397f0c4f (127.5.240.195) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 37351 } } peers { permanent_uuid: "7e8e64942ea3471ab77a706b397f0c4f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41189 } attrs { promote: false } } peers { permanent_uuid: "ee1b80f8727c4321a63259acf818aff8" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 46472 } attrs { promote: true } } unsafe_config_change: true } I0703 01:40:10.567469 12406 raft_consensus.cc:890] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5: attempting to promote NON_VOTER ee1b80f8727c4321a63259acf818aff8 to VOTER I0703 01:40:10.568210 12301 catalog_manager.cc:3798] T aa3ae539f4c6418496bbb47ff00cb47e P ee1b80f8727c4321a63259acf818aff8 reported cstate change: config changed from index 6 to 7, 7e8e64942ea3471ab77a706b397f0c4f (127.5.240.195) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 37351 } } peers { permanent_uuid: "7e8e64942ea3471ab77a706b397f0c4f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41189 } attrs { promote: false } } peers { permanent_uuid: "ee1b80f8727c4321a63259acf818aff8" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 46472 } attrs { promote: true } } unsafe_config_change: true } I0703 01:40:10.572296 12406 consensus_peers.cc:455] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 -> Peer ee1b80f8727c4321a63259acf818aff8 (127.5.240.193:46472): Closing peer: ee1b80f8727c4321a63259acf818aff8 I0703 01:40:10.572382 12406 consensus_peers.cc:455] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 -> Peer 7e8e64942ea3471ab77a706b397f0c4f (127.5.240.195:41189): Closing peer: 7e8e64942ea3471ab77a706b397f0c4f I0703 01:40:10.572686 12406 consensus_queue.cc:204] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 3.7, Last appended by leader: 0, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 37351 } } peers { permanent_uuid: "7e8e64942ea3471ab77a706b397f0c4f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41189 } attrs { promote: false } } peers { permanent_uuid: "ee1b80f8727c4321a63259acf818aff8" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 46472 } attrs { promote: false } } unsafe_config_change: true I0703 01:40:10.573843 12405 raft_consensus.cc:853] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 [term 3 LEADER]: attempt to promote peer ee1b80f8727c4321a63259acf818aff8: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing. I0703 01:40:10.574942 11944 raft_consensus.cc:1081] T aa3ae539f4c6418496bbb47ff00cb47e P 7e8e64942ea3471ab77a706b397f0c4f [term 3 FOLLOWER]: Refusing update from remote peer ea5f6a64e88b44c58375146ddaaac0b5: Log matching property violated. Preceding OpId in replica: term: 3 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch) I0703 01:40:10.575362 11699 raft_consensus.cc:1081] T aa3ae539f4c6418496bbb47ff00cb47e P ee1b80f8727c4321a63259acf818aff8 [term 3 LEARNER]: Refusing update from remote peer ea5f6a64e88b44c58375146ddaaac0b5: Log matching property violated. Preceding OpId in replica: term: 3 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch) I0703 01:40:10.575917 12412 consensus_queue.cc:962] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e8e64942ea3471ab77a706b397f0c4f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41189 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I0703 01:40:10.576294 12406 consensus_queue.cc:962] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee1b80f8727c4321a63259acf818aff8" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 46472 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I0703 01:40:10.578313 12406 raft_consensus.cc:2684] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 [term 3 LEADER]: Committing config change with OpId 3.8: config changed from index 7 to 8, ee1b80f8727c4321a63259acf818aff8 (127.5.240.193) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 37351 } } peers { permanent_uuid: "7e8e64942ea3471ab77a706b397f0c4f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41189 } attrs { promote: false } } peers { permanent_uuid: "ee1b80f8727c4321a63259acf818aff8" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 46472 } attrs { promote: false } } unsafe_config_change: true } I0703 01:40:10.579787 11699 raft_consensus.cc:2684] T aa3ae539f4c6418496bbb47ff00cb47e P ee1b80f8727c4321a63259acf818aff8 [term 3 FOLLOWER]: Committing config change with OpId 3.8: config changed from index 7 to 8, ee1b80f8727c4321a63259acf818aff8 (127.5.240.193) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 37351 } } peers { permanent_uuid: "7e8e64942ea3471ab77a706b397f0c4f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41189 } attrs { promote: false } } peers { permanent_uuid: "ee1b80f8727c4321a63259acf818aff8" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 46472 } attrs { promote: false } } unsafe_config_change: true } I0703 01:40:10.579087 11944 raft_consensus.cc:2684] T aa3ae539f4c6418496bbb47ff00cb47e P 7e8e64942ea3471ab77a706b397f0c4f [term 3 FOLLOWER]: Committing config change with OpId 3.8: config changed from index 7 to 8, ee1b80f8727c4321a63259acf818aff8 (127.5.240.193) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 37351 } } peers { permanent_uuid: "7e8e64942ea3471ab77a706b397f0c4f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41189 } attrs { promote: false } } peers { permanent_uuid: "ee1b80f8727c4321a63259acf818aff8" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 46472 } attrs { promote: false } } unsafe_config_change: true } I0703 01:40:10.581511 12301 catalog_manager.cc:3798] T aa3ae539f4c6418496bbb47ff00cb47e P ea5f6a64e88b44c58375146ddaaac0b5 reported cstate change: config changed from index 7 to 8, ee1b80f8727c4321a63259acf818aff8 (127.5.240.193) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 37351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e8e64942ea3471ab77a706b397f0c4f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41189 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee1b80f8727c4321a63259acf818aff8" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 46472 } attrs { promote: false } health_report { overall_health: HEALTHY } } unsafe_config_change: true } I0703 01:40:10.644755 6083 kudu-admin-test.cc:912] Waiting for Master to see new config... I0703 01:40:10.649055 6083 kudu-admin-test.cc:917] Tablet locations: tablet_id: "aa3ae539f4c6418496bbb47ff00cb47e" replicas { ts_info { permanent_uuid: "ea5f6a64e88b44c58375146ddaaac0b5" rpc_addresses { host: "127.5.240.194" port: 37351 } } role: LEADER } replicas { ts_info { permanent_uuid: "7e8e64942ea3471ab77a706b397f0c4f" rpc_addresses { host: "127.5.240.195" port: 41189 } } role: FOLLOWER } replicas { ts_info { permanent_uuid: "ee1b80f8727c4321a63259acf818aff8" rpc_addresses { host: "127.5.240.193" port: 46472 } } role: FOLLOWER } DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } I0703 01:40:10.649920 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 11610 I0703 01:40:10.651615 11739 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:40:10.694890 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 11610 W0703 01:40:10.703197 11753 connection.cc:512] server connection from 127.5.240.193:54630 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:54630): Connection reset by peer (error 104) W0703 01:40:10.703205 12284 connection.cc:512] server connection from 127.5.240.193:34169 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:34169): Connection reset by peer (error 104) W0703 01:40:10.703331 11756 connection.cc:512] client connection to 127.5.240.193:46472 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:46472): Connection reset by peer (error 104) I0703 01:40:10.703701 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 11746 I0703 01:40:10.704367 11862 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:40:10.751778 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 11746 W0703 01:40:10.760394 12283 connection.cc:512] server connection from 127.5.240.194:33054 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:33054): Connection reset by peer (error 104) W0703 01:40:10.760632 11876 connection.cc:512] client connection to 127.5.240.194:37351 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:37351): Connection reset by peer (error 104) W0703 01:40:10.760740 11878 connection.cc:512] server connection from 127.5.240.194:32929 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:32929): Connection reset by peer (error 104) I0703 01:40:10.760990 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 11868 I0703 01:40:10.761759 11984 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:40:10.805634 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 11868 W0703 01:40:10.814339 12285 connection.cc:512] server connection from 127.5.240.195:53825 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.195:53825): Connection reset by peer (error 104) I0703 01:40:10.814465 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master pid 12274 I0703 01:40:10.815240 12331 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:40:10.879186 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master with pid 12274 [ OK ] AdminCliTest.TestUnsafeChangeConfigFollowerWithPendingConfig (9664 ms) [ RUN ] AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL I0703 01:40:10.902070 6083 test_util.cc:208] Using random seed: 1125691939 I0703 01:40:10.903785 6083 ts_itest-base.cc:113] Starting cluster with: I0703 01:40:10.903826 6083 ts_itest-base.cc:114] -------------- I0703 01:40:10.903864 6083 ts_itest-base.cc:115] 5 tablet servers I0703 01:40:10.903894 6083 ts_itest-base.cc:116] 3 replicas per TS I0703 01:40:10.903921 6083 ts_itest-base.cc:117] -------------- I0703 01:40:10.905058 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:33510 --webserver_interface=127.5.240.254 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:40:11.086601 12428 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:40:11.086848 12428 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:40:11.089093 12428 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0703 01:40:11.089140 12428 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:40:11.089210 12428 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0703 01:40:11.089258 12428 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0703 01:40:11.097306 12428 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:33510 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.5.240.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:40:11.103018 12428 master_main.cc:80] Initializing master server... I0703 01:40:11.104127 12428 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 159208us I0703 01:40:11.105199 12428 fs_manager.cc:261] Metadata directory not provided I0703 01:40:11.105263 12428 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:40:11.105504 12428 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:40:11.105576 12428 server_base.cc:433] Attempting to create new FS layout instead I0703 01:40:11.109732 12428 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/instance: uuid: "333227bb32fd427eb883629eaf76fe88" format_stamp: "Formatted at 2018-07-03 08:40:11 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:11.110262 12428 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "333227bb32fd427eb883629eaf76fe88" format_stamp: "Formatted at 2018-07-03 08:40:11 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:11.118482 12428 fs_manager.cc:501] Time spent creating directory manager: real 0.008s user 0.002s sys 0.005s I0703 01:40:11.119709 12428 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:40:11.120102 12428 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:40:11.123733 12428 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0703 01:40:11.123792 12428 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal uuid: "333227bb32fd427eb883629eaf76fe88" format_stamp: "Formatted at 2018-07-03 08:40:11 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:11.123955 12428 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:40:11.176848 12428 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:40:11.178272 12428 master_main.cc:83] Starting Master server... I0703 01:40:11.214752 12428 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.254:33510 I0703 01:40:11.215203 12428 webserver.cc:175] Starting webserver on 127.5.240.254:0 I0703 01:40:11.215247 12428 webserver.cc:186] Document root disabled I0703 01:40:11.215986 12428 webserver.cc:313] Webserver started. Bound to: http://127.5.240.254:48462/ I0703 01:40:11.216565 12428 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb I0703 01:40:11.219694 12487 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:40:11.225816 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master as pid 12428 I0703 01:40:11.226428 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.5.240.193:0 --local_ip_for_outbound_sockets=127.5.240.193 --webserver_interface=127.5.240.193 --webserver_port=0 --tserver_master_addrs=127.5.240.254:33510 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0703 01:40:11.231003 12487 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 333227bb32fd427eb883629eaf76fe88: Bootstrap starting. I0703 01:40:11.233207 12487 tablet_bootstrap.cc:578] T 00000000000000000000000000000000 P 333227bb32fd427eb883629eaf76fe88: No blocks or log segments found. Creating new log. I0703 01:40:11.233958 12487 log.cc:525] T 00000000000000000000000000000000 P 333227bb32fd427eb883629eaf76fe88: Log is configured to *not* fsync() on all Append() calls I0703 01:40:11.236230 12487 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 333227bb32fd427eb883629eaf76fe88: No bootstrap required, opened a new log I0703 01:40:11.242419 12487 raft_consensus.cc:325] T 00000000000000000000000000000000 P 333227bb32fd427eb883629eaf76fe88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333227bb32fd427eb883629eaf76fe88" member_type: VOTER } I0703 01:40:11.242630 12487 raft_consensus.cc:351] T 00000000000000000000000000000000 P 333227bb32fd427eb883629eaf76fe88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:40:11.242692 12487 raft_consensus.cc:628] T 00000000000000000000000000000000 P 333227bb32fd427eb883629eaf76fe88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 333227bb32fd427eb883629eaf76fe88, State: Initialized, Role: FOLLOWER I0703 01:40:11.243140 12487 consensus_queue.cc:227] T 00000000000000000000000000000000 P 333227bb32fd427eb883629eaf76fe88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333227bb32fd427eb883629eaf76fe88" member_type: VOTER } I0703 01:40:11.243290 12487 raft_consensus.cc:365] T 00000000000000000000000000000000 P 333227bb32fd427eb883629eaf76fe88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:40:11.243371 12487 raft_consensus.cc:457] T 00000000000000000000000000000000 P 333227bb32fd427eb883629eaf76fe88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:40:11.243455 12487 raft_consensus.cc:2730] T 00000000000000000000000000000000 P 333227bb32fd427eb883629eaf76fe88: Snoozing failure detection for 1.531s (starting election) I0703 01:40:11.243542 12487 raft_consensus.cc:2782] T 00000000000000000000000000000000 P 333227bb32fd427eb883629eaf76fe88 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:40:11.244884 12487 raft_consensus.cc:479] T 00000000000000000000000000000000 P 333227bb32fd427eb883629eaf76fe88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333227bb32fd427eb883629eaf76fe88" member_type: VOTER } I0703 01:40:11.245250 12487 leader_election.cc:270] T 00000000000000000000000000000000 P 333227bb32fd427eb883629eaf76fe88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:40:11.246702 12490 raft_consensus.cc:2730] T 00000000000000000000000000000000 P 333227bb32fd427eb883629eaf76fe88: Snoozing failure detection for 2.089s (election complete) I0703 01:40:11.247061 12490 raft_consensus.cc:2537] T 00000000000000000000000000000000 P 333227bb32fd427eb883629eaf76fe88 [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:40:11.252743 12490 raft_consensus.cc:593] T 00000000000000000000000000000000 P 333227bb32fd427eb883629eaf76fe88 [term 1 LEADER]: Becoming Leader. State: Replica: 333227bb32fd427eb883629eaf76fe88, State: Running, Role: LEADER I0703 01:40:11.253028 12490 consensus_queue.cc:204] T 00000000000000000000000000000000 P 333227bb32fd427eb883629eaf76fe88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333227bb32fd427eb883629eaf76fe88" member_type: VOTER } I0703 01:40:11.254881 12487 sys_catalog.cc:337] T 00000000000000000000000000000000 P 333227bb32fd427eb883629eaf76fe88 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "333227bb32fd427eb883629eaf76fe88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333227bb32fd427eb883629eaf76fe88" member_type: VOTER } } I0703 01:40:11.255095 12487 sys_catalog.cc:340] T 00000000000000000000000000000000 P 333227bb32fd427eb883629eaf76fe88 [sys.catalog]: This master's current role is: LEADER I0703 01:40:11.255904 12491 sys_catalog.cc:337] T 00000000000000000000000000000000 P 333227bb32fd427eb883629eaf76fe88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "333227bb32fd427eb883629eaf76fe88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333227bb32fd427eb883629eaf76fe88" member_type: VOTER } } I0703 01:40:11.256917 12491 sys_catalog.cc:340] T 00000000000000000000000000000000 P 333227bb32fd427eb883629eaf76fe88 [sys.catalog]: This master's current role is: LEADER I0703 01:40:11.256052 12492 sys_catalog.cc:337] T 00000000000000000000000000000000 P 333227bb32fd427eb883629eaf76fe88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 333227bb32fd427eb883629eaf76fe88. Latest consensus state: current_term: 1 leader_uuid: "333227bb32fd427eb883629eaf76fe88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333227bb32fd427eb883629eaf76fe88" member_type: VOTER } } I0703 01:40:11.258273 12492 sys_catalog.cc:340] T 00000000000000000000000000000000 P 333227bb32fd427eb883629eaf76fe88 [sys.catalog]: This master's current role is: LEADER I0703 01:40:11.259564 12487 sys_catalog.cc:424] T 00000000000000000000000000000000 P 333227bb32fd427eb883629eaf76fe88 [sys.catalog]: configured and running, proceeding with master startup. I0703 01:40:11.259913 12494 catalog_manager.cc:1006] Loading table and tablet metadata into memory... I0703 01:40:11.261229 12428 master_main.cc:86] Master server successfully started. I0703 01:40:11.266384 12494 catalog_manager.cc:1017] Initializing Kudu internal certificate authority... I0703 01:40:11.290915 12494 catalog_manager.cc:904] Generated new certificate authority record I0703 01:40:11.292845 12494 catalog_manager.cc:1026] Loading token signing keys... I0703 01:40:11.305737 12494 catalog_manager.cc:4102] T 00000000000000000000000000000000 P 333227bb32fd427eb883629eaf76fe88: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:40:11.393244 12488 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:40:11.393455 12488 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:40:11.395465 12488 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:40:11.395596 12488 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.193 I0703 01:40:11.402213 12488 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.5.240.193 --webserver_port=0 --tserver_master_addrs=127.5.240.254:33510 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.193 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:40:11.408834 12488 tablet_server_main.cc:85] Initializing tablet server... I0703 01:40:11.410030 12488 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 159208us I0703 01:40:11.411003 12488 fs_manager.cc:261] Metadata directory not provided I0703 01:40:11.411068 12488 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:40:11.411697 12488 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:40:11.411763 12488 server_base.cc:433] Attempting to create new FS layout instead I0703 01:40:11.415959 12488 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "654e91af3cb64033a64c989ab937d69e" format_stamp: "Formatted at 2018-07-03 08:40:11 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:11.416451 12488 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "654e91af3cb64033a64c989ab937d69e" format_stamp: "Formatted at 2018-07-03 08:40:11 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:11.424919 12488 fs_manager.cc:501] Time spent creating directory manager: real 0.008s user 0.004s sys 0.003s I0703 01:40:11.426136 12488 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:40:11.426618 12488 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:40:11.430009 12488 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0703 01:40:11.430070 12488 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal uuid: "654e91af3cb64033a64c989ab937d69e" format_stamp: "Formatted at 2018-07-03 08:40:11 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:11.430197 12488 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:40:11.454754 12488 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:40:11.457329 12488 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:40:11.457787 12488 tablet_server_main.cc:90] Starting tablet server... I0703 01:40:11.562707 12488 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.193:52067 I0703 01:40:11.563170 12488 webserver.cc:175] Starting webserver on 127.5.240.193:0 I0703 01:40:11.563215 12488 webserver.cc:186] Document root disabled I0703 01:40:11.563995 12488 webserver.cc:313] Webserver started. Bound to: http://127.5.240.193:55739/ I0703 01:40:11.564517 12488 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb I0703 01:40:11.567147 12488 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:40:11.569476 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 12488 I0703 01:40:11.570374 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.5.240.194:0 --local_ip_for_outbound_sockets=127.5.240.194 --webserver_interface=127.5.240.194 --webserver_port=0 --tserver_master_addrs=127.5.240.254:33510 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0703 01:40:11.593094 12617 heartbeater.cc:346] Connected to a master server at 127.5.240.254:33510 I0703 01:40:11.593279 12617 heartbeater.cc:426] Registering TS with master... I0703 01:40:11.594238 12617 heartbeater.cc:475] Master 127.5.240.254:33510 requested a full tablet report, sending... I0703 01:40:11.596433 12454 ts_manager.cc:98] Registered new tserver with Master: 654e91af3cb64033a64c989ab937d69e (127.5.240.193:52067) I0703 01:40:11.599603 12454 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.193:58547 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:40:11.737752 12621 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:40:11.737998 12621 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:40:11.740059 12621 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:40:11.740180 12621 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.194 I0703 01:40:11.746856 12621 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.194:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.5.240.194 --webserver_port=0 --tserver_master_addrs=127.5.240.254:33510 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.194 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:40:11.753484 12621 tablet_server_main.cc:85] Initializing tablet server... I0703 01:40:11.754647 12621 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 159208us I0703 01:40:11.755595 12621 fs_manager.cc:261] Metadata directory not provided I0703 01:40:11.755659 12621 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:40:11.756268 12621 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:40:11.756330 12621 server_base.cc:433] Attempting to create new FS layout instead I0703 01:40:11.760550 12621 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "29af252f1f394c1a8da721d528c7e0d5" format_stamp: "Formatted at 2018-07-03 08:40:11 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:11.761095 12621 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "29af252f1f394c1a8da721d528c7e0d5" format_stamp: "Formatted at 2018-07-03 08:40:11 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:11.768913 12621 fs_manager.cc:501] Time spent creating directory manager: real 0.008s user 0.005s sys 0.002s I0703 01:40:11.770083 12621 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:40:11.770462 12621 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:40:11.773597 12621 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0703 01:40:11.773655 12621 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal uuid: "29af252f1f394c1a8da721d528c7e0d5" format_stamp: "Formatted at 2018-07-03 08:40:11 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:11.773775 12621 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:40:11.819541 12621 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:40:11.821856 12621 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:40:11.822206 12621 tablet_server_main.cc:90] Starting tablet server... I0703 01:40:11.906832 12621 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.194:56471 I0703 01:40:11.907258 12621 webserver.cc:175] Starting webserver on 127.5.240.194:0 I0703 01:40:11.907294 12621 webserver.cc:186] Document root disabled I0703 01:40:11.908013 12621 webserver.cc:313] Webserver started. Bound to: http://127.5.240.194:59742/ I0703 01:40:11.908562 12621 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb I0703 01:40:11.910917 12621 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:40:11.914386 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 12621 I0703 01:40:11.915280 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.5.240.195:0 --local_ip_for_outbound_sockets=127.5.240.195 --webserver_interface=127.5.240.195 --webserver_port=0 --tserver_master_addrs=127.5.240.254:33510 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0703 01:40:11.923774 12740 heartbeater.cc:346] Connected to a master server at 127.5.240.254:33510 I0703 01:40:11.923913 12740 heartbeater.cc:426] Registering TS with master... I0703 01:40:11.924700 12740 heartbeater.cc:475] Master 127.5.240.254:33510 requested a full tablet report, sending... I0703 01:40:11.925752 12454 ts_manager.cc:98] Registered new tserver with Master: 29af252f1f394c1a8da721d528c7e0d5 (127.5.240.194:56471) I0703 01:40:11.927037 12454 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.194:53770 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:40:12.075652 12743 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:40:12.075878 12743 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:40:12.077872 12743 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:40:12.077961 12743 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.195 I0703 01:40:12.084946 12743 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.195:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.5.240.195 --webserver_port=0 --tserver_master_addrs=127.5.240.254:33510 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.195 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:40:12.091450 12743 tablet_server_main.cc:85] Initializing tablet server... I0703 01:40:12.092572 12743 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 159708us I0703 01:40:12.093505 12743 fs_manager.cc:261] Metadata directory not provided I0703 01:40:12.093557 12743 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:40:12.093791 12743 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:40:12.093847 12743 server_base.cc:433] Attempting to create new FS layout instead I0703 01:40:12.097915 12743 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "e024b7ef889942f3b5c09f699d28f932" format_stamp: "Formatted at 2018-07-03 08:40:12 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:12.098423 12743 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "e024b7ef889942f3b5c09f699d28f932" format_stamp: "Formatted at 2018-07-03 08:40:12 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:12.105937 12743 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.002s sys 0.004s I0703 01:40:12.107144 12743 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:40:12.107528 12743 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:40:12.111151 12743 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s I0703 01:40:12.111204 12743 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal uuid: "e024b7ef889942f3b5c09f699d28f932" format_stamp: "Formatted at 2018-07-03 08:40:12 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:12.111320 12743 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:40:12.154893 12743 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:40:12.157196 12743 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:40:12.157565 12743 tablet_server_main.cc:90] Starting tablet server... I0703 01:40:12.248178 12743 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.195:49918 I0703 01:40:12.248636 12743 webserver.cc:175] Starting webserver on 127.5.240.195:0 I0703 01:40:12.248673 12743 webserver.cc:186] Document root disabled I0703 01:40:12.249366 12743 webserver.cc:313] Webserver started. Bound to: http://127.5.240.195:56025/ I0703 01:40:12.249923 12743 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb I0703 01:40:12.251683 12743 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:40:12.252003 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 12743 I0703 01:40:12.252640 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.5.240.196:0 --local_ip_for_outbound_sockets=127.5.240.196 --webserver_interface=127.5.240.196 --webserver_port=0 --tserver_master_addrs=127.5.240.254:33510 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0703 01:40:12.264590 12862 heartbeater.cc:346] Connected to a master server at 127.5.240.254:33510 I0703 01:40:12.264734 12862 heartbeater.cc:426] Registering TS with master... I0703 01:40:12.265501 12862 heartbeater.cc:475] Master 127.5.240.254:33510 requested a full tablet report, sending... I0703 01:40:12.266508 12454 ts_manager.cc:98] Registered new tserver with Master: e024b7ef889942f3b5c09f699d28f932 (127.5.240.195:49918) I0703 01:40:12.267343 12454 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.195:54280 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:40:12.410583 12865 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:40:12.410815 12865 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:40:12.412850 12865 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:40:12.412964 12865 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.196 I0703 01:40:12.419108 12865 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.196:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb --webserver_interface=127.5.240.196 --webserver_port=0 --tserver_master_addrs=127.5.240.254:33510 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.196 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:40:12.425729 12865 tablet_server_main.cc:85] Initializing tablet server... I0703 01:40:12.426870 12865 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 159708us I0703 01:40:12.427817 12865 fs_manager.cc:261] Metadata directory not provided I0703 01:40:12.427881 12865 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:40:12.428100 12865 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:40:12.428160 12865 server_base.cc:433] Attempting to create new FS layout instead I0703 01:40:12.432268 12865 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/instance: uuid: "4610b121814a4c109a1458de7333ab01" format_stamp: "Formatted at 2018-07-03 08:40:12 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:12.432847 12865 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal/instance: uuid: "4610b121814a4c109a1458de7333ab01" format_stamp: "Formatted at 2018-07-03 08:40:12 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:12.440606 12865 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.001s sys 0.005s I0703 01:40:12.441779 12865 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:40:12.442162 12865 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:40:12.445257 12865 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0703 01:40:12.445317 12865 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal uuid: "4610b121814a4c109a1458de7333ab01" format_stamp: "Formatted at 2018-07-03 08:40:12 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:12.445433 12865 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:40:12.499410 12865 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:40:12.501210 12865 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:40:12.501546 12865 tablet_server_main.cc:90] Starting tablet server... I0703 01:40:12.584722 12865 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.196:40959 I0703 01:40:12.585145 12865 webserver.cc:175] Starting webserver on 127.5.240.196:0 I0703 01:40:12.585186 12865 webserver.cc:186] Document root disabled I0703 01:40:12.586350 12865 webserver.cc:313] Webserver started. Bound to: http://127.5.240.196:41429/ I0703 01:40:12.586894 12865 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb I0703 01:40:12.588809 12865 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:40:12.595647 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 12865 I0703 01:40:12.596513 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data --block_manager=log --rpc_bind_addresses=127.5.240.197:0 --local_ip_for_outbound_sockets=127.5.240.197 --webserver_interface=127.5.240.197 --webserver_port=0 --tserver_master_addrs=127.5.240.254:33510 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0703 01:40:12.599714 12984 heartbeater.cc:346] Connected to a master server at 127.5.240.254:33510 I0703 01:40:12.599861 12984 heartbeater.cc:426] Registering TS with master... I0703 01:40:12.600658 12984 heartbeater.cc:475] Master 127.5.240.254:33510 requested a full tablet report, sending... I0703 01:40:12.601619 12454 ts_manager.cc:98] Registered new tserver with Master: 4610b121814a4c109a1458de7333ab01 (127.5.240.196:40959) I0703 01:40:12.602514 12454 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.196:35529 I0703 01:40:12.602568 12617 heartbeater.cc:467] Master 127.5.240.254:33510 was elected leader, sending a full tablet report... WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:40:12.752521 12987 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:40:12.752764 12987 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:40:12.754820 12987 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:40:12.754909 12987 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.197 I0703 01:40:12.760973 12987 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.197:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb --webserver_interface=127.5.240.197 --webserver_port=0 --tserver_master_addrs=127.5.240.254:33510 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.197 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:40:12.767563 12987 tablet_server_main.cc:85] Initializing tablet server... I0703 01:40:12.768669 12987 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 159708us I0703 01:40:12.769632 12987 fs_manager.cc:261] Metadata directory not provided I0703 01:40:12.769707 12987 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:40:12.770303 12987 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:40:12.770360 12987 server_base.cc:433] Attempting to create new FS layout instead I0703 01:40:12.774413 12987 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/instance: uuid: "bd5b113b657d4bdab1691cfc27ee69b5" format_stamp: "Formatted at 2018-07-03 08:40:12 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:12.774940 12987 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal/instance: uuid: "bd5b113b657d4bdab1691cfc27ee69b5" format_stamp: "Formatted at 2018-07-03 08:40:12 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:12.783269 12987 fs_manager.cc:501] Time spent creating directory manager: real 0.008s user 0.003s sys 0.004s I0703 01:40:12.784466 12987 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:40:12.784883 12987 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:40:12.788072 12987 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0703 01:40:12.788123 12987 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal uuid: "bd5b113b657d4bdab1691cfc27ee69b5" format_stamp: "Formatted at 2018-07-03 08:40:12 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:12.788242 12987 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:40:12.830536 12987 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:40:12.832329 12987 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:40:12.832662 12987 tablet_server_main.cc:90] Starting tablet server... I0703 01:40:12.916309 12987 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.197:58456 I0703 01:40:12.916745 12987 webserver.cc:175] Starting webserver on 127.5.240.197:0 I0703 01:40:12.916777 12987 webserver.cc:186] Document root disabled I0703 01:40:12.917929 12987 webserver.cc:313] Webserver started. Bound to: http://127.5.240.197:47988/ I0703 01:40:12.918457 12987 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb I0703 01:40:12.918879 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 12987 I0703 01:40:12.920481 12987 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:40:12.928897 12740 heartbeater.cc:467] Master 127.5.240.254:33510 was elected leader, sending a full tablet report... I0703 01:40:12.933689 13106 heartbeater.cc:346] Connected to a master server at 127.5.240.254:33510 I0703 01:40:12.933849 13106 heartbeater.cc:426] Registering TS with master... I0703 01:40:12.934736 13106 heartbeater.cc:475] Master 127.5.240.254:33510 requested a full tablet report, sending... I0703 01:40:12.935770 12453 ts_manager.cc:98] Registered new tserver with Master: bd5b113b657d4bdab1691cfc27ee69b5 (127.5.240.197:58456) I0703 01:40:12.936614 12453 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.197:36561 I0703 01:40:12.937573 6083 external_mini_cluster.cc:490] 5 TS(s) registered with all masters I0703 01:40:12.948199 12454 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.5.240.254:50521: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0703 01:40:12.980975 12554 tablet_service.cc:759] Processing CreateTablet for tablet 5ce2098a78134ca29646d1cec9ed7c61 (table=TestTable [id=f04bc5a48b834b74b717ad68380b6f1c]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:40:12.981854 12554 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 5ce2098a78134ca29646d1cec9ed7c61. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:40:12.986217 13043 tablet_service.cc:759] Processing CreateTablet for tablet 5ce2098a78134ca29646d1cec9ed7c61 (table=TestTable [id=f04bc5a48b834b74b717ad68380b6f1c]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:40:12.986992 13043 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 5ce2098a78134ca29646d1cec9ed7c61. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:40:12.987798 12677 tablet_service.cc:759] Processing CreateTablet for tablet 5ce2098a78134ca29646d1cec9ed7c61 (table=TestTable [id=f04bc5a48b834b74b717ad68380b6f1c]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:40:12.988513 12677 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 5ce2098a78134ca29646d1cec9ed7c61. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:40:12.996537 13043 ts_tablet_manager.cc:1173] T 5ce2098a78134ca29646d1cec9ed7c61 P bd5b113b657d4bdab1691cfc27ee69b5: Registered tablet (data state: TABLET_DATA_READY) I0703 01:40:12.997761 12554 ts_tablet_manager.cc:1173] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e: Registered tablet (data state: TABLET_DATA_READY) I0703 01:40:12.998565 13127 ts_tablet_manager.cc:1028] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e: Bootstrapping tablet I0703 01:40:12.999569 13127 tablet_bootstrap.cc:427] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e: Bootstrap starting. I0703 01:40:13.000111 13126 ts_tablet_manager.cc:1028] T 5ce2098a78134ca29646d1cec9ed7c61 P bd5b113b657d4bdab1691cfc27ee69b5: Bootstrapping tablet I0703 01:40:13.000476 13126 tablet_bootstrap.cc:427] T 5ce2098a78134ca29646d1cec9ed7c61 P bd5b113b657d4bdab1691cfc27ee69b5: Bootstrap starting. I0703 01:40:13.001790 12677 ts_tablet_manager.cc:1173] T 5ce2098a78134ca29646d1cec9ed7c61 P 29af252f1f394c1a8da721d528c7e0d5: Registered tablet (data state: TABLET_DATA_READY) I0703 01:40:13.002142 13127 tablet_bootstrap.cc:578] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e: No blocks or log segments found. Creating new log. I0703 01:40:13.002794 13128 ts_tablet_manager.cc:1028] T 5ce2098a78134ca29646d1cec9ed7c61 P 29af252f1f394c1a8da721d528c7e0d5: Bootstrapping tablet I0703 01:40:13.003126 13128 tablet_bootstrap.cc:427] T 5ce2098a78134ca29646d1cec9ed7c61 P 29af252f1f394c1a8da721d528c7e0d5: Bootstrap starting. I0703 01:40:13.003295 13127 log.cc:525] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e: Log is configured to *not* fsync() on all Append() calls I0703 01:40:13.003598 13126 tablet_bootstrap.cc:578] T 5ce2098a78134ca29646d1cec9ed7c61 P bd5b113b657d4bdab1691cfc27ee69b5: No blocks or log segments found. Creating new log. I0703 01:40:13.004467 13126 log.cc:525] T 5ce2098a78134ca29646d1cec9ed7c61 P bd5b113b657d4bdab1691cfc27ee69b5: Log is configured to *not* fsync() on all Append() calls I0703 01:40:13.005525 13128 tablet_bootstrap.cc:578] T 5ce2098a78134ca29646d1cec9ed7c61 P 29af252f1f394c1a8da721d528c7e0d5: No blocks or log segments found. Creating new log. I0703 01:40:13.005707 13127 tablet_bootstrap.cc:427] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e: No bootstrap required, opened a new log I0703 01:40:13.005952 13127 ts_tablet_manager.cc:1045] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s I0703 01:40:13.006006 13126 tablet_bootstrap.cc:427] T 5ce2098a78134ca29646d1cec9ed7c61 P bd5b113b657d4bdab1691cfc27ee69b5: No bootstrap required, opened a new log I0703 01:40:13.006162 13126 ts_tablet_manager.cc:1045] T 5ce2098a78134ca29646d1cec9ed7c61 P bd5b113b657d4bdab1691cfc27ee69b5: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s I0703 01:40:13.006471 13128 log.cc:525] T 5ce2098a78134ca29646d1cec9ed7c61 P 29af252f1f394c1a8da721d528c7e0d5: Log is configured to *not* fsync() on all Append() calls I0703 01:40:13.008718 13128 tablet_bootstrap.cc:427] T 5ce2098a78134ca29646d1cec9ed7c61 P 29af252f1f394c1a8da721d528c7e0d5: No bootstrap required, opened a new log I0703 01:40:13.008942 13128 ts_tablet_manager.cc:1045] T 5ce2098a78134ca29646d1cec9ed7c61 P 29af252f1f394c1a8da721d528c7e0d5: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s I0703 01:40:13.012326 13126 raft_consensus.cc:325] T 5ce2098a78134ca29646d1cec9ed7c61 P bd5b113b657d4bdab1691cfc27ee69b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd5b113b657d4bdab1691cfc27ee69b5" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 58456 } } peers { permanent_uuid: "654e91af3cb64033a64c989ab937d69e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 52067 } } peers { permanent_uuid: "29af252f1f394c1a8da721d528c7e0d5" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 56471 } } I0703 01:40:13.012647 13126 raft_consensus.cc:351] T 5ce2098a78134ca29646d1cec9ed7c61 P bd5b113b657d4bdab1691cfc27ee69b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:40:13.012720 13126 raft_consensus.cc:628] T 5ce2098a78134ca29646d1cec9ed7c61 P bd5b113b657d4bdab1691cfc27ee69b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd5b113b657d4bdab1691cfc27ee69b5, State: Initialized, Role: FOLLOWER I0703 01:40:13.013195 13126 consensus_queue.cc:227] T 5ce2098a78134ca29646d1cec9ed7c61 P bd5b113b657d4bdab1691cfc27ee69b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd5b113b657d4bdab1691cfc27ee69b5" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 58456 } } peers { permanent_uuid: "654e91af3cb64033a64c989ab937d69e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 52067 } } peers { permanent_uuid: "29af252f1f394c1a8da721d528c7e0d5" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 56471 } } I0703 01:40:13.014591 13106 heartbeater.cc:467] Master 127.5.240.254:33510 was elected leader, sending a full tablet report... I0703 01:40:13.015413 13126 ts_tablet_manager.cc:1073] T 5ce2098a78134ca29646d1cec9ed7c61 P bd5b113b657d4bdab1691cfc27ee69b5: Time spent starting tablet: real 0.009s user 0.008s sys 0.002s I0703 01:40:13.015295 13127 raft_consensus.cc:325] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd5b113b657d4bdab1691cfc27ee69b5" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 58456 } } peers { permanent_uuid: "654e91af3cb64033a64c989ab937d69e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 52067 } } peers { permanent_uuid: "29af252f1f394c1a8da721d528c7e0d5" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 56471 } } I0703 01:40:13.015704 13127 raft_consensus.cc:351] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:40:13.015775 13127 raft_consensus.cc:628] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 654e91af3cb64033a64c989ab937d69e, State: Initialized, Role: FOLLOWER I0703 01:40:13.016222 13127 consensus_queue.cc:227] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd5b113b657d4bdab1691cfc27ee69b5" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 58456 } } peers { permanent_uuid: "654e91af3cb64033a64c989ab937d69e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 52067 } } peers { permanent_uuid: "29af252f1f394c1a8da721d528c7e0d5" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 56471 } } I0703 01:40:13.017693 13128 raft_consensus.cc:325] T 5ce2098a78134ca29646d1cec9ed7c61 P 29af252f1f394c1a8da721d528c7e0d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd5b113b657d4bdab1691cfc27ee69b5" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 58456 } } peers { permanent_uuid: "654e91af3cb64033a64c989ab937d69e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 52067 } } peers { permanent_uuid: "29af252f1f394c1a8da721d528c7e0d5" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 56471 } } I0703 01:40:13.018064 13128 raft_consensus.cc:351] T 5ce2098a78134ca29646d1cec9ed7c61 P 29af252f1f394c1a8da721d528c7e0d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:40:13.018213 13128 raft_consensus.cc:628] T 5ce2098a78134ca29646d1cec9ed7c61 P 29af252f1f394c1a8da721d528c7e0d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29af252f1f394c1a8da721d528c7e0d5, State: Initialized, Role: FOLLOWER I0703 01:40:13.018702 13128 consensus_queue.cc:227] T 5ce2098a78134ca29646d1cec9ed7c61 P 29af252f1f394c1a8da721d528c7e0d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd5b113b657d4bdab1691cfc27ee69b5" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 58456 } } peers { permanent_uuid: "654e91af3cb64033a64c989ab937d69e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 52067 } } peers { permanent_uuid: "29af252f1f394c1a8da721d528c7e0d5" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 56471 } } I0703 01:40:13.019889 13127 ts_tablet_manager.cc:1073] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e: Time spent starting tablet: real 0.014s user 0.007s sys 0.005s I0703 01:40:13.021153 13128 ts_tablet_manager.cc:1073] T 5ce2098a78134ca29646d1cec9ed7c61 P 29af252f1f394c1a8da721d528c7e0d5: Time spent starting tablet: real 0.012s user 0.009s sys 0.003s I0703 01:40:13.026798 13133 raft_consensus.cc:457] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0703 01:40:13.026891 13133 raft_consensus.cc:2730] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e: Snoozing failure detection for 1.534s (starting election) I0703 01:40:13.026978 13133 raft_consensus.cc:479] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd5b113b657d4bdab1691cfc27ee69b5" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 58456 } } peers { permanent_uuid: "654e91af3cb64033a64c989ab937d69e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 52067 } } peers { permanent_uuid: "29af252f1f394c1a8da721d528c7e0d5" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 56471 } } I0703 01:40:13.028069 13133 leader_election.cc:243] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer bd5b113b657d4bdab1691cfc27ee69b5 (127.5.240.197:58456) I0703 01:40:13.028385 13133 leader_election.cc:243] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 29af252f1f394c1a8da721d528c7e0d5 (127.5.240.194:56471) I0703 01:40:13.039429 13063 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "5ce2098a78134ca29646d1cec9ed7c61" candidate_uuid: "654e91af3cb64033a64c989ab937d69e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd5b113b657d4bdab1691cfc27ee69b5" is_pre_election: true I0703 01:40:13.039427 12697 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "5ce2098a78134ca29646d1cec9ed7c61" candidate_uuid: "654e91af3cb64033a64c989ab937d69e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29af252f1f394c1a8da721d528c7e0d5" is_pre_election: true I0703 01:40:13.040076 12697 raft_consensus.cc:2730] T 5ce2098a78134ca29646d1cec9ed7c61 P 29af252f1f394c1a8da721d528c7e0d5: Snoozing failure detection for 2.182s (vote granted) I0703 01:40:13.040099 13063 raft_consensus.cc:2730] T 5ce2098a78134ca29646d1cec9ed7c61 P bd5b113b657d4bdab1691cfc27ee69b5: Snoozing failure detection for 2.105s (vote granted) I0703 01:40:13.040163 12697 raft_consensus.cc:2243] T 5ce2098a78134ca29646d1cec9ed7c61 P 29af252f1f394c1a8da721d528c7e0d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 654e91af3cb64033a64c989ab937d69e in term 0. I0703 01:40:13.040186 13063 raft_consensus.cc:2243] T 5ce2098a78134ca29646d1cec9ed7c61 P bd5b113b657d4bdab1691cfc27ee69b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 654e91af3cb64033a64c989ab937d69e in term 0. I0703 01:40:13.040786 12505 leader_election.cc:379] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [CANDIDATE]: Term 1 pre-election: Vote granted by peer 29af252f1f394c1a8da721d528c7e0d5 (127.5.240.194:56471) I0703 01:40:13.040892 12505 leader_election.cc:270] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0703 01:40:13.041098 12506 leader_election.cc:379] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [CANDIDATE]: Term 1 pre-election: Vote granted by peer bd5b113b657d4bdab1691cfc27ee69b5 (127.5.240.197:58456) I0703 01:40:13.041254 13133 raft_consensus.cc:2730] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e: Snoozing failure detection for 1.638s (election complete) I0703 01:40:13.041353 13133 raft_consensus.cc:2537] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [term 0 FOLLOWER]: Leader pre-election won for term 1 I0703 01:40:13.041414 13133 raft_consensus.cc:457] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0703 01:40:13.041469 13133 raft_consensus.cc:2730] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e: Snoozing failure detection for 2.138s (starting election) I0703 01:40:13.041525 13133 raft_consensus.cc:2782] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [term 0 FOLLOWER]: Advancing to term 1 I0703 01:40:13.042999 13133 raft_consensus.cc:479] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd5b113b657d4bdab1691cfc27ee69b5" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 58456 } } peers { permanent_uuid: "654e91af3cb64033a64c989ab937d69e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 52067 } } peers { permanent_uuid: "29af252f1f394c1a8da721d528c7e0d5" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 56471 } } I0703 01:40:13.043373 13133 leader_election.cc:243] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [CANDIDATE]: Term 1 election: Requesting vote from peer bd5b113b657d4bdab1691cfc27ee69b5 (127.5.240.197:58456) I0703 01:40:13.043530 13133 leader_election.cc:243] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [CANDIDATE]: Term 1 election: Requesting vote from peer 29af252f1f394c1a8da721d528c7e0d5 (127.5.240.194:56471) I0703 01:40:13.044015 13063 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "5ce2098a78134ca29646d1cec9ed7c61" candidate_uuid: "654e91af3cb64033a64c989ab937d69e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd5b113b657d4bdab1691cfc27ee69b5" I0703 01:40:13.044128 13063 raft_consensus.cc:2782] T 5ce2098a78134ca29646d1cec9ed7c61 P bd5b113b657d4bdab1691cfc27ee69b5 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:40:13.044149 12697 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "5ce2098a78134ca29646d1cec9ed7c61" candidate_uuid: "654e91af3cb64033a64c989ab937d69e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29af252f1f394c1a8da721d528c7e0d5" I0703 01:40:13.044219 13063 raft_consensus.cc:2730] T 5ce2098a78134ca29646d1cec9ed7c61 P bd5b113b657d4bdab1691cfc27ee69b5: Snoozing failure detection for 2.204s (vote granted) I0703 01:40:13.044256 12697 raft_consensus.cc:2782] T 5ce2098a78134ca29646d1cec9ed7c61 P 29af252f1f394c1a8da721d528c7e0d5 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:40:13.044337 12697 raft_consensus.cc:2730] T 5ce2098a78134ca29646d1cec9ed7c61 P 29af252f1f394c1a8da721d528c7e0d5: Snoozing failure detection for 1.763s (vote granted) I0703 01:40:13.045555 13063 raft_consensus.cc:2243] T 5ce2098a78134ca29646d1cec9ed7c61 P bd5b113b657d4bdab1691cfc27ee69b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 654e91af3cb64033a64c989ab937d69e in term 1. I0703 01:40:13.045696 12697 raft_consensus.cc:2243] T 5ce2098a78134ca29646d1cec9ed7c61 P 29af252f1f394c1a8da721d528c7e0d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 654e91af3cb64033a64c989ab937d69e in term 1. I0703 01:40:13.046043 12506 leader_election.cc:379] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [CANDIDATE]: Term 1 election: Vote granted by peer bd5b113b657d4bdab1691cfc27ee69b5 (127.5.240.197:58456) I0703 01:40:13.046126 12506 leader_election.cc:270] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:40:13.046234 12505 leader_election.cc:379] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [CANDIDATE]: Term 1 election: Vote granted by peer 29af252f1f394c1a8da721d528c7e0d5 (127.5.240.194:56471) I0703 01:40:13.046325 13133 raft_consensus.cc:2730] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e: Snoozing failure detection for 2.197s (election complete) I0703 01:40:13.046411 13133 raft_consensus.cc:2537] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:40:13.047699 13133 raft_consensus.cc:593] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [term 1 LEADER]: Becoming Leader. State: Replica: 654e91af3cb64033a64c989ab937d69e, State: Running, Role: LEADER I0703 01:40:13.048058 13133 consensus_queue.cc:204] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd5b113b657d4bdab1691cfc27ee69b5" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 58456 } } peers { permanent_uuid: "654e91af3cb64033a64c989ab937d69e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 52067 } } peers { permanent_uuid: "29af252f1f394c1a8da721d528c7e0d5" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 56471 } } I0703 01:40:13.051986 12453 catalog_manager.cc:3798] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e reported cstate change: term changed from 0 to 1, leader changed from to 654e91af3cb64033a64c989ab937d69e (127.5.240.193). New cstate: current_term: 1 leader_uuid: "654e91af3cb64033a64c989ab937d69e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd5b113b657d4bdab1691cfc27ee69b5" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 58456 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "654e91af3cb64033a64c989ab937d69e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 52067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29af252f1f394c1a8da721d528c7e0d5" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 56471 } health_report { overall_health: UNKNOWN } } } W0703 01:40:13.068347 12618 tablet.cc:1678] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:40:13.074968 6083 external_mini_cluster.cc:490] 5 TS(s) registered with all masters I0703 01:40:13.076082 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 654e91af3cb64033a64c989ab937d69e to finish bootstrapping I0703 01:40:13.081158 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 29af252f1f394c1a8da721d528c7e0d5 to finish bootstrapping I0703 01:40:13.086302 6083 ts_itest-base.cc:233] Waiting for 0 tablets on tserver e024b7ef889942f3b5c09f699d28f932 to finish bootstrapping I0703 01:40:13.092321 6083 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 4610b121814a4c109a1458de7333ab01 to finish bootstrapping I0703 01:40:13.098127 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver bd5b113b657d4bdab1691cfc27ee69b5 to finish bootstrapping I0703 01:40:13.102640 6083 kudu-admin-test.cc:948] Waiting for Master to see the current replicas... I0703 01:40:13.103667 6083 kudu-admin-test.cc:955] Tablet locations: tablet_id: "5ce2098a78134ca29646d1cec9ed7c61" replicas { ts_info { permanent_uuid: "bd5b113b657d4bdab1691cfc27ee69b5" rpc_addresses { host: "127.5.240.197" port: 58456 } } role: FOLLOWER } replicas { ts_info { permanent_uuid: "654e91af3cb64033a64c989ab937d69e" rpc_addresses { host: "127.5.240.193" port: 52067 } } role: LEADER } replicas { ts_info { permanent_uuid: "29af252f1f394c1a8da721d528c7e0d5" rpc_addresses { host: "127.5.240.194" port: 56471 } } role: FOLLOWER } DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } W0703 01:40:13.161749 12741 tablet.cc:1678] T 5ce2098a78134ca29646d1cec9ed7c61 P 29af252f1f394c1a8da721d528c7e0d5: Can't schedule compaction. Clean time has not been advanced past its initial value. W0703 01:40:13.173478 13107 tablet.cc:1678] T 5ce2098a78134ca29646d1cec9ed7c61 P bd5b113b657d4bdab1691cfc27ee69b5: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:40:13.269114 12862 heartbeater.cc:467] Master 127.5.240.254:33510 was elected leader, sending a full tablet report... I0703 01:40:13.557934 13063 raft_consensus.cc:1081] T 5ce2098a78134ca29646d1cec9ed7c61 P bd5b113b657d4bdab1691cfc27ee69b5 [term 1 FOLLOWER]: Refusing update from remote peer 654e91af3cb64033a64c989ab937d69e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0703 01:40:13.558670 13142 consensus_queue.cc:962] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd5b113b657d4bdab1691cfc27ee69b5" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 58456 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0703 01:40:13.565235 12697 raft_consensus.cc:1081] T 5ce2098a78134ca29646d1cec9ed7c61 P 29af252f1f394c1a8da721d528c7e0d5 [term 1 FOLLOWER]: Refusing update from remote peer 654e91af3cb64033a64c989ab937d69e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0703 01:40:13.566411 13146 consensus_queue.cc:962] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [LEADER]: Connected to new peer: Peer: permanent_uuid: "29af252f1f394c1a8da721d528c7e0d5" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 56471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0703 01:40:13.575624 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 12987 I0703 01:40:13.576205 13103 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:40:13.605226 12984 heartbeater.cc:467] Master 127.5.240.254:33510 was elected leader, sending a full tablet report... I0703 01:40:13.619865 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 12987 W0703 01:40:13.627360 12437 connection.cc:512] server connection from 127.5.240.197:36561 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.197:36561): Connection reset by peer (error 104) W0703 01:40:13.627346 12506 connection.cc:512] client connection to 127.5.240.197:58456 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.197:58456): Connection reset by peer (error 104) I0703 01:40:13.627732 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 12621 I0703 01:40:13.628515 12737 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:40:13.671005 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 12621 W0703 01:40:13.679055 12436 connection.cc:512] server connection from 127.5.240.194:53770 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:53770): Connection reset by peer (error 104) W0703 01:40:13.679055 12505 connection.cc:512] client connection to 127.5.240.194:56471 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:56471): Connection reset by peer (error 104) I0703 01:40:13.680433 12574 consensus_peers.cc:455] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e -> Peer 29af252f1f394c1a8da721d528c7e0d5 (127.5.240.194:56471): Closing peer: 29af252f1f394c1a8da721d528c7e0d5 I0703 01:40:13.680503 12574 consensus_peers.cc:455] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e -> Peer bd5b113b657d4bdab1691cfc27ee69b5 (127.5.240.197:58456): Closing peer: bd5b113b657d4bdab1691cfc27ee69b5 I0703 01:40:13.680794 12574 consensus_queue.cc:204] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bd5b113b657d4bdab1691cfc27ee69b5" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 58456 } } peers { permanent_uuid: "654e91af3cb64033a64c989ab937d69e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 52067 } } W0703 01:40:13.682416 12506 consensus_peers.cc:433] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e -> Peer bd5b113b657d4bdab1691cfc27ee69b5 (127.5.240.197:58456): Couldn't send request to peer bd5b113b657d4bdab1691cfc27ee69b5 for tablet 5ce2098a78134ca29646d1cec9ed7c61. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.197:58456: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. W0703 01:40:14.283607 12506 consensus_peers.cc:433] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e -> Peer bd5b113b657d4bdab1691cfc27ee69b5 (127.5.240.197:58456): Couldn't send request to peer bd5b113b657d4bdab1691cfc27ee69b5 for tablet 5ce2098a78134ca29646d1cec9ed7c61. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.197:58456: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. W0703 01:40:14.730442 12506 consensus_peers.cc:433] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e -> Peer bd5b113b657d4bdab1691cfc27ee69b5 (127.5.240.197:58456): Couldn't send request to peer bd5b113b657d4bdab1691cfc27ee69b5 for tablet 5ce2098a78134ca29646d1cec9ed7c61. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.197:58456: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times. W0703 01:40:15.169559 12506 consensus_peers.cc:433] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e -> Peer bd5b113b657d4bdab1691cfc27ee69b5 (127.5.240.197:58456): Couldn't send request to peer bd5b113b657d4bdab1691cfc27ee69b5 for tablet 5ce2098a78134ca29646d1cec9ed7c61. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.197:58456: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 4 times. I0703 01:40:15.681362 6083 kudu-admin-test.cc:976] Change Config Op timed out, Sending a Replace config command when change config op is pending on the leader. W0703 01:40:15.787742 12506 consensus_peers.cc:433] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e -> Peer bd5b113b657d4bdab1691cfc27ee69b5 (127.5.240.197:58456): Couldn't send request to peer bd5b113b657d4bdab1691cfc27ee69b5 for tablet 5ce2098a78134ca29646d1cec9ed7c61. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.197:58456: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 5 times. W0703 01:40:15.892292 13154 flags.cc:406] Enabled unsafe flag: --never_fsync=true I0703 01:40:15.941295 12574 tablet_service.cc:1077] Received UnsafeChangeConfig RPC: dest_uuid: "654e91af3cb64033a64c989ab937d69e" tablet_id: "5ce2098a78134ca29646d1cec9ed7c61" caller_id: "kudu-tools" new_config { peers { permanent_uuid: "654e91af3cb64033a64c989ab937d69e" } } from {username='jenkins'} at 127.5.240.193:52443 W0703 01:40:15.941515 12574 raft_consensus.cc:1918] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [term 1 LEADER]: Replica has a pending config, but the new config will be unsafely changed anyway. Currently pending config on the node: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bd5b113b657d4bdab1691cfc27ee69b5" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 58456 } } peers { permanent_uuid: "654e91af3cb64033a64c989ab937d69e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 52067 } } W0703 01:40:15.941807 12574 raft_consensus.cc:2017] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [term 1 LEADER]: PROCEEDING WITH UNSAFE CONFIG CHANGE ON THIS SERVER, COMMITTED CONFIG: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd5b113b657d4bdab1691cfc27ee69b5" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 58456 } } peers { permanent_uuid: "654e91af3cb64033a64c989ab937d69e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 52067 } } peers { permanent_uuid: "29af252f1f394c1a8da721d528c7e0d5" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 56471 } }NEW CONFIG: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "654e91af3cb64033a64c989ab937d69e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 52067 } } unsafe_config_change: true I0703 01:40:15.942129 12574 raft_consensus.cc:2777] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [term 1 LEADER]: Stepping down as leader of term 1 I0703 01:40:15.942169 12574 raft_consensus.cc:628] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 654e91af3cb64033a64c989ab937d69e, State: Running, Role: LEADER I0703 01:40:15.942369 12574 consensus_queue.cc:227] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bd5b113b657d4bdab1691cfc27ee69b5" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 58456 } } peers { permanent_uuid: "654e91af3cb64033a64c989ab937d69e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 52067 } } I0703 01:40:15.942483 12574 consensus_peers.cc:455] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e -> Peer bd5b113b657d4bdab1691cfc27ee69b5 (127.5.240.197:58456): Closing peer: bd5b113b657d4bdab1691cfc27ee69b5 I0703 01:40:15.942596 12574 raft_consensus.cc:2782] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [term 1 FOLLOWER]: Advancing to term 2 I0703 01:40:15.944695 12574 raft_consensus.cc:2849] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [term 2 FOLLOWER]: Allowing unsafe config change even though there is a pending config! Existing pending config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bd5b113b657d4bdab1691cfc27ee69b5" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 58456 } } peers { permanent_uuid: "654e91af3cb64033a64c989ab937d69e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 52067 } }; New pending config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "654e91af3cb64033a64c989ab937d69e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 52067 } } unsafe_config_change: true W0703 01:40:15.989401 12614 generic_service.cc:126] {username='jenkins'} at 127.5.240.193:52435 forcing change of non-runtime-safe flag fault_crash_before_cmeta_flush I0703 01:40:15.989506 12614 generic_service.cc:146] {username='jenkins'} at 127.5.240.193:52435 changed flags via RPC: fault_crash_before_cmeta_flush from '0' to '1.0' I0703 01:40:15.990041 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master pid 12428 I0703 01:40:15.990703 12484 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:40:16.054360 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master with pid 12428 W0703 01:40:16.067116 12507 connection.cc:512] client connection to 127.5.240.254:33510 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.254:33510): Connection reset by peer (error 104) W0703 01:40:16.067137 12752 connection.cc:512] client connection to 127.5.240.254:33510 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.254:33510): Connection reset by peer (error 104) W0703 01:40:16.067777 12874 connection.cc:512] client connection to 127.5.240.254:33510 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.254:33510): Connection reset by peer (error 104) I0703 01:40:16.068560 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:33510 --webserver_interface=127.5.240.254 --webserver_port=48462 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:40:16.249766 13172 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:40:16.249971 13172 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:40:16.252180 13172 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0703 01:40:16.252218 13172 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:40:16.252250 13172 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0703 01:40:16.252282 13172 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0703 01:40:16.260025 13172 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:33510 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.5.240.254 --webserver_port=48462 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:40:16.265547 13172 master_main.cc:80] Initializing master server... I0703 01:40:16.266590 13172 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 161708us I0703 01:40:16.267637 13172 fs_manager.cc:261] Metadata directory not provided I0703 01:40:16.267693 13172 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:40:16.275244 13172 fs_manager.cc:397] Time spent opening directory manager: real 0.005s user 0.002s sys 0.002s W0703 01:40:16.275307 12862 heartbeater.cc:587] Failed to heartbeat to 127.5.240.254:33510: Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.240.254:33510: connect: Connection refused (error 111) I0703 01:40:16.276116 13172 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:40:16.276504 13172 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:40:16.279628 13172 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0703 01:40:16.279690 13172 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal uuid: "333227bb32fd427eb883629eaf76fe88" format_stamp: "Formatted at 2018-07-03 08:40:11 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:16.280519 13172 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:40:16.326174 13172 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:40:16.327636 13172 master_main.cc:83] Starting Master server... I0703 01:40:16.365032 13172 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.254:33510 I0703 01:40:16.365507 13172 webserver.cc:175] Starting webserver on 127.5.240.254:48462 I0703 01:40:16.365555 13172 webserver.cc:186] Document root disabled I0703 01:40:16.366770 13172 webserver.cc:313] Webserver started. Bound to: http://127.5.240.254:48462/ I0703 01:40:16.367458 13172 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb I0703 01:40:16.369256 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master as pid 13172 I0703 01:40:16.375440 13232 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 333227bb32fd427eb883629eaf76fe88: Bootstrap starting. I0703 01:40:16.387231 13232 log.cc:525] T 00000000000000000000000000000000 P 333227bb32fd427eb883629eaf76fe88: Log is configured to *not* fsync() on all Append() calls I0703 01:40:16.396201 13232 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 333227bb32fd427eb883629eaf76fe88: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=1} inserts{seen=4 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:40:16.396589 13232 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 333227bb32fd427eb883629eaf76fe88: Bootstrap complete. I0703 01:40:16.405732 13232 raft_consensus.cc:325] T 00000000000000000000000000000000 P 333227bb32fd427eb883629eaf76fe88 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333227bb32fd427eb883629eaf76fe88" member_type: VOTER } I0703 01:40:16.406416 13232 raft_consensus.cc:628] T 00000000000000000000000000000000 P 333227bb32fd427eb883629eaf76fe88 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 333227bb32fd427eb883629eaf76fe88, State: Initialized, Role: FOLLOWER I0703 01:40:16.406857 13232 consensus_queue.cc:227] T 00000000000000000000000000000000 P 333227bb32fd427eb883629eaf76fe88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333227bb32fd427eb883629eaf76fe88" member_type: VOTER } I0703 01:40:16.406981 13232 raft_consensus.cc:365] T 00000000000000000000000000000000 P 333227bb32fd427eb883629eaf76fe88 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:40:16.407043 13232 raft_consensus.cc:457] T 00000000000000000000000000000000 P 333227bb32fd427eb883629eaf76fe88 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:40:16.407109 13232 raft_consensus.cc:2730] T 00000000000000000000000000000000 P 333227bb32fd427eb883629eaf76fe88: Snoozing failure detection for 2.195s (starting election) I0703 01:40:16.407178 13232 raft_consensus.cc:2782] T 00000000000000000000000000000000 P 333227bb32fd427eb883629eaf76fe88 [term 1 FOLLOWER]: Advancing to term 2 I0703 01:40:16.410521 13232 raft_consensus.cc:479] T 00000000000000000000000000000000 P 333227bb32fd427eb883629eaf76fe88 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333227bb32fd427eb883629eaf76fe88" member_type: VOTER } I0703 01:40:16.410866 13232 leader_election.cc:270] T 00000000000000000000000000000000 P 333227bb32fd427eb883629eaf76fe88 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0703 01:40:16.412151 13234 raft_consensus.cc:2730] T 00000000000000000000000000000000 P 333227bb32fd427eb883629eaf76fe88: Snoozing failure detection for 1.614s (election complete) I0703 01:40:16.412395 13234 raft_consensus.cc:2537] T 00000000000000000000000000000000 P 333227bb32fd427eb883629eaf76fe88 [term 2 FOLLOWER]: Leader election won for term 2 I0703 01:40:16.413995 13234 raft_consensus.cc:593] T 00000000000000000000000000000000 P 333227bb32fd427eb883629eaf76fe88 [term 2 LEADER]: Becoming Leader. State: Replica: 333227bb32fd427eb883629eaf76fe88, State: Running, Role: LEADER I0703 01:40:16.414373 13234 consensus_queue.cc:204] T 00000000000000000000000000000000 P 333227bb32fd427eb883629eaf76fe88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333227bb32fd427eb883629eaf76fe88" member_type: VOTER } I0703 01:40:16.416513 13232 sys_catalog.cc:337] T 00000000000000000000000000000000 P 333227bb32fd427eb883629eaf76fe88 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 2 leader_uuid: "333227bb32fd427eb883629eaf76fe88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333227bb32fd427eb883629eaf76fe88" member_type: VOTER } } I0703 01:40:16.416987 13232 sys_catalog.cc:340] T 00000000000000000000000000000000 P 333227bb32fd427eb883629eaf76fe88 [sys.catalog]: This master's current role is: LEADER I0703 01:40:16.416769 13236 sys_catalog.cc:337] T 00000000000000000000000000000000 P 333227bb32fd427eb883629eaf76fe88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 333227bb32fd427eb883629eaf76fe88. Latest consensus state: current_term: 2 leader_uuid: "333227bb32fd427eb883629eaf76fe88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333227bb32fd427eb883629eaf76fe88" member_type: VOTER } } I0703 01:40:16.416771 13235 sys_catalog.cc:337] T 00000000000000000000000000000000 P 333227bb32fd427eb883629eaf76fe88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "333227bb32fd427eb883629eaf76fe88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333227bb32fd427eb883629eaf76fe88" member_type: VOTER } } I0703 01:40:16.418406 13236 sys_catalog.cc:340] T 00000000000000000000000000000000 P 333227bb32fd427eb883629eaf76fe88 [sys.catalog]: This master's current role is: LEADER I0703 01:40:16.418506 13235 sys_catalog.cc:340] T 00000000000000000000000000000000 P 333227bb32fd427eb883629eaf76fe88 [sys.catalog]: This master's current role is: LEADER I0703 01:40:16.419242 13232 sys_catalog.cc:424] T 00000000000000000000000000000000 P 333227bb32fd427eb883629eaf76fe88 [sys.catalog]: configured and running, proceeding with master startup. I0703 01:40:16.419473 13238 catalog_manager.cc:1006] Loading table and tablet metadata into memory... I0703 01:40:16.420753 13172 master_main.cc:86] Master server successfully started. I0703 01:40:16.425004 13238 catalog_manager.cc:323] Loaded metadata for table TestTable [id=f04bc5a48b834b74b717ad68380b6f1c] I0703 01:40:16.426970 13238 catalog_manager.cc:377] Loaded metadata for tablet 5ce2098a78134ca29646d1cec9ed7c61 (table TestTable [id=f04bc5a48b834b74b717ad68380b6f1c]) I0703 01:40:16.427127 13238 catalog_manager.cc:1017] Initializing Kudu internal certificate authority... I0703 01:40:16.430047 13238 catalog_manager.cc:1026] Loading token signing keys... I0703 01:40:16.431149 13238 catalog_manager.cc:4113] T 00000000000000000000000000000000 P 333227bb32fd427eb883629eaf76fe88: Loaded TSK: 0 I0703 01:40:16.622869 13199 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "4610b121814a4c109a1458de7333ab01" instance_seqno: 1530607212568120) as {username='jenkins'} at 127.5.240.196:58480; Asking this server to re-register. I0703 01:40:16.624313 12984 heartbeater.cc:426] Registering TS with master... I0703 01:40:16.624439 12984 heartbeater.cc:475] Master 127.5.240.254:33510 requested a full tablet report, sending... I0703 01:40:16.625641 13199 ts_manager.cc:98] Registered new tserver with Master: 4610b121814a4c109a1458de7333ab01 (127.5.240.196:40959) I0703 01:40:16.951769 13199 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "654e91af3cb64033a64c989ab937d69e" instance_seqno: 1530607211543957) as {username='jenkins'} at 127.5.240.193:48419; Asking this server to re-register. I0703 01:40:16.952316 12617 heartbeater.cc:426] Registering TS with master... I0703 01:40:16.952435 12617 heartbeater.cc:475] Master 127.5.240.254:33510 requested a full tablet report, sending... I0703 01:40:16.953320 13199 ts_manager.cc:98] Registered new tserver with Master: 654e91af3cb64033a64c989ab937d69e (127.5.240.193:52067) I0703 01:40:17.280792 12862 heartbeater.cc:346] Connected to a master server at 127.5.240.254:33510 I0703 01:40:17.281519 13199 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "e024b7ef889942f3b5c09f699d28f932" instance_seqno: 1530607212230886) as {username='jenkins'} at 127.5.240.195:38995; Asking this server to re-register. I0703 01:40:17.282160 12862 heartbeater.cc:426] Registering TS with master... I0703 01:40:17.282269 12862 heartbeater.cc:475] Master 127.5.240.254:33510 requested a full tablet report, sending... I0703 01:40:17.282974 13199 ts_manager.cc:98] Registered new tserver with Master: e024b7ef889942f3b5c09f699d28f932 (127.5.240.195:49918) I0703 01:40:17.445605 13247 raft_consensus.cc:457] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader kudu-tools) I0703 01:40:17.445675 13247 raft_consensus.cc:2730] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e: Snoozing failure detection for 1.594s (starting election) I0703 01:40:17.445749 13247 raft_consensus.cc:479] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "654e91af3cb64033a64c989ab937d69e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 52067 } } unsafe_config_change: true I0703 01:40:17.445925 13247 leader_election.cc:270] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. I0703 01:40:17.446267 13247 raft_consensus.cc:2730] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e: Snoozing failure detection for 2.047s (election complete) I0703 01:40:17.446318 13247 raft_consensus.cc:2537] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [term 2 FOLLOWER]: Leader pre-election won for term 3 I0703 01:40:17.446359 13247 raft_consensus.cc:457] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [term 2 FOLLOWER]: Starting leader election (detected failure of leader kudu-tools) I0703 01:40:17.446396 13247 raft_consensus.cc:2730] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e: Snoozing failure detection for 1.996s (starting election) I0703 01:40:17.446439 13247 raft_consensus.cc:2782] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [term 2 FOLLOWER]: Advancing to term 3 W0703 01:40:17.446496 13247 fault_injection.cc:41] FAULT INJECTION ENABLED! W0703 01:40:17.446529 13247 fault_injection.cc:42] THIS SERVER MAY CRASH! E0703 01:40:17.446550 13247 fault_injection.cc:57] Injecting fault: FLAGS_fault_crash_before_cmeta_flush (process will exit) W0703 01:40:17.455510 13182 connection.cc:512] server connection from 127.5.240.193:48419 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:48419): Connection reset by peer (error 104) I0703 01:40:17.572798 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.5.240.193:52067 --local_ip_for_outbound_sockets=127.5.240.193 --webserver_port=55739 --webserver_interface=127.5.240.193 --tserver_master_addrs=127.5.240.254:33510 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:40:17.729885 13249 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:40:17.730115 13249 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:40:17.732164 13249 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:40:17.732272 13249 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.193 I0703 01:40:17.738505 13249 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.193:52067 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.5.240.193 --webserver_port=55739 --tserver_master_addrs=127.5.240.254:33510 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.193 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:40:17.745111 13249 tablet_server_main.cc:85] Initializing tablet server... I0703 01:40:17.746251 13249 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 162208us I0703 01:40:17.747215 13249 fs_manager.cc:261] Metadata directory not provided I0703 01:40:17.747282 13249 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:40:17.753067 13249 fs_manager.cc:397] Time spent opening directory manager: real 0.004s user 0.001s sys 0.002s I0703 01:40:17.753908 13249 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:40:17.754281 13249 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:40:17.757768 13249 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s I0703 01:40:17.757838 13249 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal uuid: "654e91af3cb64033a64c989ab937d69e" format_stamp: "Formatted at 2018-07-03 08:40:11 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:17.759449 13249 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:40:17.794433 13249 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:40:17.796351 13249 ts_tablet_manager.cc:344] Loading tablet metadata (0/1 complete) I0703 01:40:17.796406 13249 ts_tablet_manager.cc:1005] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e: Loading tablet metadata I0703 01:40:17.800825 13249 ts_tablet_manager.cc:356] Loaded tablet metadata (1 live tablets) I0703 01:40:17.802789 13249 ts_tablet_manager.cc:1173] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e: Registered tablet (data state: TABLET_DATA_READY) I0703 01:40:17.804138 13264 ts_tablet_manager.cc:1028] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e: Bootstrapping tablet I0703 01:40:17.805094 13249 tablet_server_main.cc:90] Starting tablet server... I0703 01:40:17.805290 13264 tablet_bootstrap.cc:427] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e: Bootstrap starting. I0703 01:40:17.833222 13264 log.cc:525] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e: Log is configured to *not* fsync() on all Append() calls I0703 01:40:17.838783 13264 tablet_bootstrap.cc:427] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 2 replicates I0703 01:40:17.840240 13264 tablet_bootstrap.cc:427] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e: Bootstrap complete. I0703 01:40:17.841519 13264 ts_tablet_manager.cc:1045] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e: Time spent bootstrapping tablet: real 0.036s user 0.006s sys 0.021s I0703 01:40:17.853492 13264 raft_consensus.cc:325] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [term 2 FOLLOWER]: Replica starting. Triggering 2 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd5b113b657d4bdab1691cfc27ee69b5" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 58456 } } peers { permanent_uuid: "654e91af3cb64033a64c989ab937d69e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 52067 } } peers { permanent_uuid: "29af252f1f394c1a8da721d528c7e0d5" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 56471 } } I0703 01:40:17.859907 13264 raft_consensus.cc:2849] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [term 2 FOLLOWER]: Allowing unsafe config change even though there is a pending config! Existing pending config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bd5b113b657d4bdab1691cfc27ee69b5" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 58456 } } peers { permanent_uuid: "654e91af3cb64033a64c989ab937d69e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 52067 } }; New pending config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "654e91af3cb64033a64c989ab937d69e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 52067 } } unsafe_config_change: true I0703 01:40:17.861125 13264 raft_consensus.cc:628] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 654e91af3cb64033a64c989ab937d69e, State: Initialized, Role: FOLLOWER I0703 01:40:17.861886 13264 consensus_queue.cc:227] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "654e91af3cb64033a64c989ab937d69e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 52067 } } unsafe_config_change: true I0703 01:40:17.866722 13264 ts_tablet_manager.cc:1073] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e: Time spent starting tablet: real 0.025s user 0.011s sys 0.010s I0703 01:40:17.920040 13249 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.193:52067 I0703 01:40:17.920440 13249 webserver.cc:175] Starting webserver on 127.5.240.193:55739 I0703 01:40:17.920485 13249 webserver.cc:186] Document root disabled I0703 01:40:17.921653 13249 webserver.cc:313] Webserver started. Bound to: http://127.5.240.193:55739/ I0703 01:40:17.922214 13249 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb I0703 01:40:17.924015 13249 tablet_server_main.cc:93] Tablet server successfully started. W0703 01:40:17.924008 13372 tablet.cc:1678] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:40:17.924804 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 13249 I0703 01:40:17.935906 13371 heartbeater.cc:346] Connected to a master server at 127.5.240.254:33510 I0703 01:40:17.936097 13371 heartbeater.cc:426] Registering TS with master... I0703 01:40:17.936471 13371 heartbeater.cc:475] Master 127.5.240.254:33510 requested a full tablet report, sending... I0703 01:40:17.938002 13199 ts_manager.cc:104] Re-registered known tserver with Master: 654e91af3cb64033a64c989ab937d69e (127.5.240.193:52067) I0703 01:40:17.939370 13199 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.193:47098 I0703 01:40:18.942240 13371 heartbeater.cc:467] Master 127.5.240.254:33510 was elected leader, sending a full tablet report... I0703 01:40:19.014286 13378 raft_consensus.cc:457] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0703 01:40:19.014418 13378 raft_consensus.cc:2730] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e: Snoozing failure detection for 2.045s (starting election) I0703 01:40:19.014533 13378 raft_consensus.cc:479] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "654e91af3cb64033a64c989ab937d69e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 52067 } } unsafe_config_change: true I0703 01:40:19.014950 13378 leader_election.cc:270] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. I0703 01:40:19.015408 13379 raft_consensus.cc:2730] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e: Snoozing failure detection for 1.770s (election complete) I0703 01:40:19.015470 13379 raft_consensus.cc:2537] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [term 2 FOLLOWER]: Leader pre-election won for term 3 I0703 01:40:19.015548 13379 raft_consensus.cc:457] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0703 01:40:19.015604 13379 raft_consensus.cc:2730] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e: Snoozing failure detection for 2.041s (starting election) I0703 01:40:19.015686 13379 raft_consensus.cc:2782] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [term 2 FOLLOWER]: Advancing to term 3 I0703 01:40:19.018055 13379 raft_consensus.cc:479] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "654e91af3cb64033a64c989ab937d69e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 52067 } } unsafe_config_change: true I0703 01:40:19.018218 13379 leader_election.cc:270] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. I0703 01:40:19.018401 13378 raft_consensus.cc:2730] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e: Snoozing failure detection for 1.640s (election complete) I0703 01:40:19.018461 13378 raft_consensus.cc:2537] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [term 3 FOLLOWER]: Leader election won for term 3 I0703 01:40:19.018604 13378 raft_consensus.cc:593] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [term 3 LEADER]: Becoming Leader. State: Replica: 654e91af3cb64033a64c989ab937d69e, State: Running, Role: LEADER I0703 01:40:19.018942 13378 consensus_queue.cc:204] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "654e91af3cb64033a64c989ab937d69e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 52067 } } unsafe_config_change: true I0703 01:40:19.020767 13378 raft_consensus.cc:2684] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [term 3 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 29af252f1f394c1a8da721d528c7e0d5 (127.5.240.194) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "bd5b113b657d4bdab1691cfc27ee69b5" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 58456 } } peers { permanent_uuid: "654e91af3cb64033a64c989ab937d69e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 52067 } } } I0703 01:40:19.021737 13199 catalog_manager.cc:3798] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e reported cstate change: term changed from 1 to 3, now has a pending config: VOTER 654e91af3cb64033a64c989ab937d69e (127.5.240.193). New cstate: current_term: 3 leader_uuid: "654e91af3cb64033a64c989ab937d69e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd5b113b657d4bdab1691cfc27ee69b5" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 58456 } } peers { permanent_uuid: "654e91af3cb64033a64c989ab937d69e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 52067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29af252f1f394c1a8da721d528c7e0d5" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 56471 } } } pending_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "654e91af3cb64033a64c989ab937d69e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 52067 } } unsafe_config_change: true } I0703 01:40:19.022830 13378 raft_consensus.cc:2684] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [term 3 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, VOTER 29af252f1f394c1a8da721d528c7e0d5 (127.5.240.194) evicted, VOTER bd5b113b657d4bdab1691cfc27ee69b5 (127.5.240.197) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "654e91af3cb64033a64c989ab937d69e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 52067 } } unsafe_config_change: true } I0703 01:40:19.030540 13199 catalog_manager.cc:3798] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e reported cstate change: config changed from index -1 to 3, VOTER 29af252f1f394c1a8da721d528c7e0d5 (127.5.240.194) evicted, VOTER bd5b113b657d4bdab1691cfc27ee69b5 (127.5.240.197) evicted, no longer has a pending config: VOTER 654e91af3cb64033a64c989ab937d69e (127.5.240.193). New cstate: current_term: 3 leader_uuid: "654e91af3cb64033a64c989ab937d69e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "654e91af3cb64033a64c989ab937d69e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 52067 } health_report { overall_health: HEALTHY } } unsafe_config_change: true } I0703 01:40:19.032589 13199 catalog_manager.cc:2903] Scheduling retry of 5ce2098a78134ca29646d1cec9ed7c61 Delete Tablet RPC for TS=bd5b113b657d4bdab1691cfc27ee69b5 with a delay of 26 ms (attempt = 1) W0703 01:40:19.032681 13199 catalog_manager.cc:3957] Failed to send 5ce2098a78134ca29646d1cec9ed7c61 Delete Tablet RPC for TS=bd5b113b657d4bdab1691cfc27ee69b5: Not found: failed to reset TS proxy: Could not find TS for UUID bd5b113b657d4bdab1691cfc27ee69b5 I0703 01:40:19.032771 13199 catalog_manager.cc:2903] Scheduling retry of 5ce2098a78134ca29646d1cec9ed7c61 Delete Tablet RPC for TS=29af252f1f394c1a8da721d528c7e0d5 with a delay of 21 ms (attempt = 1) W0703 01:40:19.032831 13199 catalog_manager.cc:3957] Failed to send 5ce2098a78134ca29646d1cec9ed7c61 Delete Tablet RPC for TS=29af252f1f394c1a8da721d528c7e0d5: Not found: failed to reset TS proxy: Could not find TS for UUID 29af252f1f394c1a8da721d528c7e0d5 I0703 01:40:19.033195 13199 catalog_manager.cc:3437] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 5ce2098a78134ca29646d1cec9ed7c61 (attempt 1) I0703 01:40:19.039615 13328 consensus_queue.cc:204] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 3, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "654e91af3cb64033a64c989ab937d69e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 52067 } } peers { permanent_uuid: "e024b7ef889942f3b5c09f699d28f932" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 49918 } attrs { promote: true } } unsafe_config_change: true I0703 01:40:19.040949 13378 raft_consensus.cc:2684] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 3 to 5, NON_VOTER e024b7ef889942f3b5c09f699d28f932 (127.5.240.195) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "654e91af3cb64033a64c989ab937d69e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 52067 } } peers { permanent_uuid: "e024b7ef889942f3b5c09f699d28f932" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 49918 } attrs { promote: true } } unsafe_config_change: true } I0703 01:40:19.043716 13180 catalog_manager.cc:3352] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5ce2098a78134ca29646d1cec9ed7c61 with cas_config_opid_index 3: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0703 01:40:19.044675 13199 catalog_manager.cc:3798] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e reported cstate change: config changed from index 3 to 5, NON_VOTER e024b7ef889942f3b5c09f699d28f932 (127.5.240.195) added. New cstate: current_term: 3 leader_uuid: "654e91af3cb64033a64c989ab937d69e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "654e91af3cb64033a64c989ab937d69e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 52067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e024b7ef889942f3b5c09f699d28f932" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 49918 } attrs { promote: true } health_report { overall_health: UNKNOWN } } unsafe_config_change: true } I0703 01:40:19.046792 13199 catalog_manager.cc:3437] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 5ce2098a78134ca29646d1cec9ed7c61 (attempt 1) I0703 01:40:19.047703 13328 consensus_peers.cc:455] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e -> Peer e024b7ef889942f3b5c09f699d28f932 (127.5.240.195:49918): Closing peer: e024b7ef889942f3b5c09f699d28f932 I0703 01:40:19.047888 13328 consensus_queue.cc:204] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 3, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "654e91af3cb64033a64c989ab937d69e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 52067 } } peers { permanent_uuid: "e024b7ef889942f3b5c09f699d28f932" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 49918 } attrs { promote: true } } peers { permanent_uuid: "4610b121814a4c109a1458de7333ab01" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 40959 } attrs { promote: true } } unsafe_config_change: true I0703 01:40:19.049538 13378 raft_consensus.cc:2684] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [term 3 LEADER]: Committing config change with OpId 3.6: config changed from index 5 to 6, NON_VOTER 4610b121814a4c109a1458de7333ab01 (127.5.240.196) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "654e91af3cb64033a64c989ab937d69e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 52067 } } peers { permanent_uuid: "e024b7ef889942f3b5c09f699d28f932" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 49918 } attrs { promote: true } } peers { permanent_uuid: "4610b121814a4c109a1458de7333ab01" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 40959 } attrs { promote: true } } unsafe_config_change: true } W0703 01:40:19.050835 13257 consensus_peers.cc:433] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e -> Peer e024b7ef889942f3b5c09f699d28f932 (127.5.240.195:49918): Couldn't send request to peer e024b7ef889942f3b5c09f699d28f932 for tablet 5ce2098a78134ca29646d1cec9ed7c61. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ce2098a78134ca29646d1cec9ed7c61. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:40:19.052186 13180 catalog_manager.cc:3352] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5ce2098a78134ca29646d1cec9ed7c61 with cas_config_opid_index 5: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0703 01:40:19.053113 13199 catalog_manager.cc:3798] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e reported cstate change: config changed from index 5 to 6, NON_VOTER 4610b121814a4c109a1458de7333ab01 (127.5.240.196) added. New cstate: current_term: 3 leader_uuid: "654e91af3cb64033a64c989ab937d69e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "654e91af3cb64033a64c989ab937d69e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 52067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e024b7ef889942f3b5c09f699d28f932" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 49918 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4610b121814a4c109a1458de7333ab01" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 40959 } attrs { promote: true } health_report { overall_health: UNKNOWN } } unsafe_config_change: true } I0703 01:40:19.054214 13182 catalog_manager.cc:2903] Scheduling retry of 5ce2098a78134ca29646d1cec9ed7c61 Delete Tablet RPC for TS=29af252f1f394c1a8da721d528c7e0d5 with a delay of 50 ms (attempt = 2) W0703 01:40:19.054342 13182 catalog_manager.cc:2922] Async tablet task 5ce2098a78134ca29646d1cec9ed7c61 Delete Tablet RPC for TS=29af252f1f394c1a8da721d528c7e0d5 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 29af252f1f394c1a8da721d528c7e0d5 W0703 01:40:19.057763 13256 consensus_peers.cc:433] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e -> Peer 4610b121814a4c109a1458de7333ab01 (127.5.240.196:40959): Couldn't send request to peer 4610b121814a4c109a1458de7333ab01 for tablet 5ce2098a78134ca29646d1cec9ed7c61. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ce2098a78134ca29646d1cec9ed7c61. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:40:19.059490 13180 catalog_manager.cc:2903] Scheduling retry of 5ce2098a78134ca29646d1cec9ed7c61 Delete Tablet RPC for TS=bd5b113b657d4bdab1691cfc27ee69b5 with a delay of 43 ms (attempt = 2) W0703 01:40:19.059558 13180 catalog_manager.cc:2922] Async tablet task 5ce2098a78134ca29646d1cec9ed7c61 Delete Tablet RPC for TS=bd5b113b657d4bdab1691cfc27ee69b5 failed: Not found: failed to reset TS proxy: Could not find TS for UUID bd5b113b657d4bdab1691cfc27ee69b5 I0703 01:40:19.103132 13180 catalog_manager.cc:2903] Scheduling retry of 5ce2098a78134ca29646d1cec9ed7c61 Delete Tablet RPC for TS=bd5b113b657d4bdab1691cfc27ee69b5 with a delay of 106 ms (attempt = 3) W0703 01:40:19.103186 13180 catalog_manager.cc:2922] Async tablet task 5ce2098a78134ca29646d1cec9ed7c61 Delete Tablet RPC for TS=bd5b113b657d4bdab1691cfc27ee69b5 failed: Not found: failed to reset TS proxy: Could not find TS for UUID bd5b113b657d4bdab1691cfc27ee69b5 I0703 01:40:19.105006 13182 catalog_manager.cc:2903] Scheduling retry of 5ce2098a78134ca29646d1cec9ed7c61 Delete Tablet RPC for TS=29af252f1f394c1a8da721d528c7e0d5 with a delay of 96 ms (attempt = 3) W0703 01:40:19.105072 13182 catalog_manager.cc:2922] Async tablet task 5ce2098a78134ca29646d1cec9ed7c61 Delete Tablet RPC for TS=29af252f1f394c1a8da721d528c7e0d5 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 29af252f1f394c1a8da721d528c7e0d5 I0703 01:40:19.201761 13182 catalog_manager.cc:2903] Scheduling retry of 5ce2098a78134ca29646d1cec9ed7c61 Delete Tablet RPC for TS=29af252f1f394c1a8da721d528c7e0d5 with a delay of 174 ms (attempt = 4) W0703 01:40:19.201827 13182 catalog_manager.cc:2922] Async tablet task 5ce2098a78134ca29646d1cec9ed7c61 Delete Tablet RPC for TS=29af252f1f394c1a8da721d528c7e0d5 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 29af252f1f394c1a8da721d528c7e0d5 I0703 01:40:19.209879 13180 catalog_manager.cc:2903] Scheduling retry of 5ce2098a78134ca29646d1cec9ed7c61 Delete Tablet RPC for TS=bd5b113b657d4bdab1691cfc27ee69b5 with a delay of 157 ms (attempt = 4) W0703 01:40:19.209935 13180 catalog_manager.cc:2922] Async tablet task 5ce2098a78134ca29646d1cec9ed7c61 Delete Tablet RPC for TS=bd5b113b657d4bdab1691cfc27ee69b5 failed: Not found: failed to reset TS proxy: Could not find TS for UUID bd5b113b657d4bdab1691cfc27ee69b5 I0703 01:40:19.367631 13180 catalog_manager.cc:2903] Scheduling retry of 5ce2098a78134ca29646d1cec9ed7c61 Delete Tablet RPC for TS=bd5b113b657d4bdab1691cfc27ee69b5 with a delay of 277 ms (attempt = 5) W0703 01:40:19.367688 13180 catalog_manager.cc:2922] Async tablet task 5ce2098a78134ca29646d1cec9ed7c61 Delete Tablet RPC for TS=bd5b113b657d4bdab1691cfc27ee69b5 failed: Not found: failed to reset TS proxy: Could not find TS for UUID bd5b113b657d4bdab1691cfc27ee69b5 I0703 01:40:19.376415 13182 catalog_manager.cc:2903] Scheduling retry of 5ce2098a78134ca29646d1cec9ed7c61 Delete Tablet RPC for TS=29af252f1f394c1a8da721d528c7e0d5 with a delay of 276 ms (attempt = 5) W0703 01:40:19.376471 13182 catalog_manager.cc:2922] Async tablet task 5ce2098a78134ca29646d1cec9ed7c61 Delete Tablet RPC for TS=29af252f1f394c1a8da721d528c7e0d5 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 29af252f1f394c1a8da721d528c7e0d5 I0703 01:40:19.594889 13391 ts_tablet_manager.cc:669] T 5ce2098a78134ca29646d1cec9ed7c61 P e024b7ef889942f3b5c09f699d28f932: Initiating tablet copy from peer 654e91af3cb64033a64c989ab937d69e (127.5.240.193:52067) I0703 01:40:19.595137 13391 tablet_copy_client.cc:239] T 5ce2098a78134ca29646d1cec9ed7c61 P e024b7ef889942f3b5c09f699d28f932: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.193:52067 I0703 01:40:19.600360 13348 tablet_copy_service.cc:135] P 654e91af3cb64033a64c989ab937d69e: Received BeginTabletCopySession request for tablet 5ce2098a78134ca29646d1cec9ed7c61 from peer e024b7ef889942f3b5c09f699d28f932 ({username='jenkins'} at 127.5.240.195:39399) I0703 01:40:19.600507 13348 tablet_copy_service.cc:156] P 654e91af3cb64033a64c989ab937d69e: Beginning new tablet copy session on tablet 5ce2098a78134ca29646d1cec9ed7c61 from peer e024b7ef889942f3b5c09f699d28f932 at {username='jenkins'} at 127.5.240.195:39399: session id = e024b7ef889942f3b5c09f699d28f932-5ce2098a78134ca29646d1cec9ed7c61 I0703 01:40:19.602418 13348 tablet_copy_source_session.cc:202] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:40:19.605232 13391 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 5ce2098a78134ca29646d1cec9ed7c61. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:40:19.615515 13391 ts_tablet_manager.cc:1173] T 5ce2098a78134ca29646d1cec9ed7c61 P e024b7ef889942f3b5c09f699d28f932: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:40:19.615759 13391 tablet_copy_client.cc:566] T 5ce2098a78134ca29646d1cec9ed7c61 P e024b7ef889942f3b5c09f699d28f932: tablet copy: Starting download of 0 data blocks... I0703 01:40:19.615970 13391 tablet_copy_client.cc:529] T 5ce2098a78134ca29646d1cec9ed7c61 P e024b7ef889942f3b5c09f699d28f932: tablet copy: Starting download of 1 WAL segments... I0703 01:40:19.617544 13391 tablet_copy_client.cc:405] T 5ce2098a78134ca29646d1cec9ed7c61 P e024b7ef889942f3b5c09f699d28f932: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:40:19.619220 13391 ts_tablet_manager.cc:1028] T 5ce2098a78134ca29646d1cec9ed7c61 P e024b7ef889942f3b5c09f699d28f932: Bootstrapping tablet I0703 01:40:19.619437 13391 tablet_bootstrap.cc:427] T 5ce2098a78134ca29646d1cec9ed7c61 P e024b7ef889942f3b5c09f699d28f932: Bootstrap starting. I0703 01:40:19.624140 13391 log.cc:525] T 5ce2098a78134ca29646d1cec9ed7c61 P e024b7ef889942f3b5c09f699d28f932: Log is configured to *not* fsync() on all Append() calls I0703 01:40:19.628898 13391 tablet_bootstrap.cc:427] T 5ce2098a78134ca29646d1cec9ed7c61 P e024b7ef889942f3b5c09f699d28f932: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:40:19.629237 13391 tablet_bootstrap.cc:427] T 5ce2098a78134ca29646d1cec9ed7c61 P e024b7ef889942f3b5c09f699d28f932: Bootstrap complete. I0703 01:40:19.629444 13391 ts_tablet_manager.cc:1045] T 5ce2098a78134ca29646d1cec9ed7c61 P e024b7ef889942f3b5c09f699d28f932: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s I0703 01:40:19.635184 13391 raft_consensus.cc:325] T 5ce2098a78134ca29646d1cec9ed7c61 P e024b7ef889942f3b5c09f699d28f932 [term 3 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "654e91af3cb64033a64c989ab937d69e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 52067 } } peers { permanent_uuid: "e024b7ef889942f3b5c09f699d28f932" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 49918 } attrs { promote: true } } peers { permanent_uuid: "4610b121814a4c109a1458de7333ab01" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 40959 } attrs { promote: true } } unsafe_config_change: true I0703 01:40:19.635470 13391 raft_consensus.cc:628] T 5ce2098a78134ca29646d1cec9ed7c61 P e024b7ef889942f3b5c09f699d28f932 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: e024b7ef889942f3b5c09f699d28f932, State: Initialized, Role: LEARNER I0703 01:40:19.635735 13391 consensus_queue.cc:227] T 5ce2098a78134ca29646d1cec9ed7c61 P e024b7ef889942f3b5c09f699d28f932 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "654e91af3cb64033a64c989ab937d69e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 52067 } } peers { permanent_uuid: "e024b7ef889942f3b5c09f699d28f932" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 49918 } attrs { promote: true } } peers { permanent_uuid: "4610b121814a4c109a1458de7333ab01" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 40959 } attrs { promote: true } } unsafe_config_change: true I0703 01:40:19.637470 13391 ts_tablet_manager.cc:1073] T 5ce2098a78134ca29646d1cec9ed7c61 P e024b7ef889942f3b5c09f699d28f932: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s I0703 01:40:19.638166 13348 tablet_copy_service.cc:325] P 654e91af3cb64033a64c989ab937d69e: Request end of tablet copy session e024b7ef889942f3b5c09f699d28f932-5ce2098a78134ca29646d1cec9ed7c61 received from {username='jenkins'} at 127.5.240.195:39399 I0703 01:40:19.638264 13348 tablet_copy_service.cc:413] P 654e91af3cb64033a64c989ab937d69e: ending tablet copy session e024b7ef889942f3b5c09f699d28f932-5ce2098a78134ca29646d1cec9ed7c61 on tablet 5ce2098a78134ca29646d1cec9ed7c61 with peer e024b7ef889942f3b5c09f699d28f932 I0703 01:40:19.645840 13180 catalog_manager.cc:2903] Scheduling retry of 5ce2098a78134ca29646d1cec9ed7c61 Delete Tablet RPC for TS=bd5b113b657d4bdab1691cfc27ee69b5 with a delay of 546 ms (attempt = 6) W0703 01:40:19.645915 13180 catalog_manager.cc:2922] Async tablet task 5ce2098a78134ca29646d1cec9ed7c61 Delete Tablet RPC for TS=bd5b113b657d4bdab1691cfc27ee69b5 failed: Not found: failed to reset TS proxy: Could not find TS for UUID bd5b113b657d4bdab1691cfc27ee69b5 I0703 01:40:19.646417 13396 ts_tablet_manager.cc:669] T 5ce2098a78134ca29646d1cec9ed7c61 P 4610b121814a4c109a1458de7333ab01: Initiating tablet copy from peer 654e91af3cb64033a64c989ab937d69e (127.5.240.193:52067) I0703 01:40:19.646670 13396 tablet_copy_client.cc:239] T 5ce2098a78134ca29646d1cec9ed7c61 P 4610b121814a4c109a1458de7333ab01: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.193:52067 I0703 01:40:19.650907 13348 tablet_copy_service.cc:135] P 654e91af3cb64033a64c989ab937d69e: Received BeginTabletCopySession request for tablet 5ce2098a78134ca29646d1cec9ed7c61 from peer 4610b121814a4c109a1458de7333ab01 ({username='jenkins'} at 127.5.240.196:51971) I0703 01:40:19.650991 13348 tablet_copy_service.cc:156] P 654e91af3cb64033a64c989ab937d69e: Beginning new tablet copy session on tablet 5ce2098a78134ca29646d1cec9ed7c61 from peer 4610b121814a4c109a1458de7333ab01 at {username='jenkins'} at 127.5.240.196:51971: session id = 4610b121814a4c109a1458de7333ab01-5ce2098a78134ca29646d1cec9ed7c61 I0703 01:40:19.652534 13348 tablet_copy_source_session.cc:202] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:40:19.653628 13182 catalog_manager.cc:2903] Scheduling retry of 5ce2098a78134ca29646d1cec9ed7c61 Delete Tablet RPC for TS=29af252f1f394c1a8da721d528c7e0d5 with a delay of 534 ms (attempt = 6) W0703 01:40:19.653697 13182 catalog_manager.cc:2922] Async tablet task 5ce2098a78134ca29646d1cec9ed7c61 Delete Tablet RPC for TS=29af252f1f394c1a8da721d528c7e0d5 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 29af252f1f394c1a8da721d528c7e0d5 I0703 01:40:19.655061 13396 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 5ce2098a78134ca29646d1cec9ed7c61. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:40:19.665228 13396 ts_tablet_manager.cc:1173] T 5ce2098a78134ca29646d1cec9ed7c61 P 4610b121814a4c109a1458de7333ab01: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:40:19.665488 13396 tablet_copy_client.cc:566] T 5ce2098a78134ca29646d1cec9ed7c61 P 4610b121814a4c109a1458de7333ab01: tablet copy: Starting download of 0 data blocks... I0703 01:40:19.665756 13396 tablet_copy_client.cc:529] T 5ce2098a78134ca29646d1cec9ed7c61 P 4610b121814a4c109a1458de7333ab01: tablet copy: Starting download of 1 WAL segments... I0703 01:40:19.667156 13396 tablet_copy_client.cc:405] T 5ce2098a78134ca29646d1cec9ed7c61 P 4610b121814a4c109a1458de7333ab01: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:40:19.668821 13396 ts_tablet_manager.cc:1028] T 5ce2098a78134ca29646d1cec9ed7c61 P 4610b121814a4c109a1458de7333ab01: Bootstrapping tablet I0703 01:40:19.669034 13396 tablet_bootstrap.cc:427] T 5ce2098a78134ca29646d1cec9ed7c61 P 4610b121814a4c109a1458de7333ab01: Bootstrap starting. I0703 01:40:19.673372 13396 log.cc:525] T 5ce2098a78134ca29646d1cec9ed7c61 P 4610b121814a4c109a1458de7333ab01: Log is configured to *not* fsync() on all Append() calls I0703 01:40:19.677817 13396 tablet_bootstrap.cc:427] T 5ce2098a78134ca29646d1cec9ed7c61 P 4610b121814a4c109a1458de7333ab01: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:40:19.678531 13396 tablet_bootstrap.cc:427] T 5ce2098a78134ca29646d1cec9ed7c61 P 4610b121814a4c109a1458de7333ab01: Bootstrap complete. I0703 01:40:19.678766 13396 ts_tablet_manager.cc:1045] T 5ce2098a78134ca29646d1cec9ed7c61 P 4610b121814a4c109a1458de7333ab01: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.004s I0703 01:40:19.683492 13396 raft_consensus.cc:325] T 5ce2098a78134ca29646d1cec9ed7c61 P 4610b121814a4c109a1458de7333ab01 [term 3 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "654e91af3cb64033a64c989ab937d69e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 52067 } } peers { permanent_uuid: "e024b7ef889942f3b5c09f699d28f932" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 49918 } attrs { promote: true } } peers { permanent_uuid: "4610b121814a4c109a1458de7333ab01" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 40959 } attrs { promote: true } } unsafe_config_change: true I0703 01:40:19.683805 13396 raft_consensus.cc:628] T 5ce2098a78134ca29646d1cec9ed7c61 P 4610b121814a4c109a1458de7333ab01 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 4610b121814a4c109a1458de7333ab01, State: Initialized, Role: LEARNER I0703 01:40:19.684051 13396 consensus_queue.cc:227] T 5ce2098a78134ca29646d1cec9ed7c61 P 4610b121814a4c109a1458de7333ab01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "654e91af3cb64033a64c989ab937d69e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 52067 } } peers { permanent_uuid: "e024b7ef889942f3b5c09f699d28f932" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 49918 } attrs { promote: true } } peers { permanent_uuid: "4610b121814a4c109a1458de7333ab01" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 40959 } attrs { promote: true } } unsafe_config_change: true I0703 01:40:19.685806 13396 ts_tablet_manager.cc:1073] T 5ce2098a78134ca29646d1cec9ed7c61 P 4610b121814a4c109a1458de7333ab01: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s I0703 01:40:19.686434 13348 tablet_copy_service.cc:325] P 654e91af3cb64033a64c989ab937d69e: Request end of tablet copy session 4610b121814a4c109a1458de7333ab01-5ce2098a78134ca29646d1cec9ed7c61 received from {username='jenkins'} at 127.5.240.196:51971 I0703 01:40:19.686504 13348 tablet_copy_service.cc:413] P 654e91af3cb64033a64c989ab937d69e: ending tablet copy session 4610b121814a4c109a1458de7333ab01-5ce2098a78134ca29646d1cec9ed7c61 on tablet 5ce2098a78134ca29646d1cec9ed7c61 with peer 4610b121814a4c109a1458de7333ab01 W0703 01:40:19.756191 12863 tablet.cc:1678] T 5ce2098a78134ca29646d1cec9ed7c61 P e024b7ef889942f3b5c09f699d28f932: Can't schedule compaction. Clean time has not been advanced past its initial value. W0703 01:40:19.843268 12985 tablet.cc:1678] T 5ce2098a78134ca29646d1cec9ed7c61 P 4610b121814a4c109a1458de7333ab01: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:40:20.071630 12819 raft_consensus.cc:1026] T 5ce2098a78134ca29646d1cec9ed7c61 P e024b7ef889942f3b5c09f699d28f932 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.5->[3.6-3.6] Dedup: 3.6->[] I0703 01:40:20.115682 12941 raft_consensus.cc:1026] T 5ce2098a78134ca29646d1cec9ed7c61 P 4610b121814a4c109a1458de7333ab01 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.5->[3.6-3.6] Dedup: 3.6->[] I0703 01:40:20.188671 13182 catalog_manager.cc:2903] Scheduling retry of 5ce2098a78134ca29646d1cec9ed7c61 Delete Tablet RPC for TS=29af252f1f394c1a8da721d528c7e0d5 with a delay of 1051 ms (attempt = 7) W0703 01:40:20.188767 13182 catalog_manager.cc:2922] Async tablet task 5ce2098a78134ca29646d1cec9ed7c61 Delete Tablet RPC for TS=29af252f1f394c1a8da721d528c7e0d5 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 29af252f1f394c1a8da721d528c7e0d5 I0703 01:40:20.193001 13180 catalog_manager.cc:2903] Scheduling retry of 5ce2098a78134ca29646d1cec9ed7c61 Delete Tablet RPC for TS=bd5b113b657d4bdab1691cfc27ee69b5 with a delay of 1058 ms (attempt = 7) W0703 01:40:20.193047 13180 catalog_manager.cc:2922] Async tablet task 5ce2098a78134ca29646d1cec9ed7c61 Delete Tablet RPC for TS=bd5b113b657d4bdab1691cfc27ee69b5 failed: Not found: failed to reset TS proxy: Could not find TS for UUID bd5b113b657d4bdab1691cfc27ee69b5 I0703 01:40:20.488905 13401 raft_consensus.cc:890] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e: attempting to promote NON_VOTER e024b7ef889942f3b5c09f699d28f932 to VOTER I0703 01:40:20.489207 13401 consensus_peers.cc:455] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e -> Peer 4610b121814a4c109a1458de7333ab01 (127.5.240.196:40959): Closing peer: 4610b121814a4c109a1458de7333ab01 I0703 01:40:20.489274 13401 consensus_peers.cc:455] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e -> Peer e024b7ef889942f3b5c09f699d28f932 (127.5.240.195:49918): Closing peer: e024b7ef889942f3b5c09f699d28f932 I0703 01:40:20.489573 13401 consensus_queue.cc:204] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "654e91af3cb64033a64c989ab937d69e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 52067 } } peers { permanent_uuid: "e024b7ef889942f3b5c09f699d28f932" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 49918 } attrs { promote: false } } peers { permanent_uuid: "4610b121814a4c109a1458de7333ab01" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 40959 } attrs { promote: true } } unsafe_config_change: true I0703 01:40:20.491575 12819 raft_consensus.cc:1081] T 5ce2098a78134ca29646d1cec9ed7c61 P e024b7ef889942f3b5c09f699d28f932 [term 3 LEARNER]: Refusing update from remote peer 654e91af3cb64033a64c989ab937d69e: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch) I0703 01:40:20.491876 12941 raft_consensus.cc:1081] T 5ce2098a78134ca29646d1cec9ed7c61 P 4610b121814a4c109a1458de7333ab01 [term 3 LEARNER]: Refusing update from remote peer 654e91af3cb64033a64c989ab937d69e: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch) I0703 01:40:20.492326 13400 consensus_queue.cc:962] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e024b7ef889942f3b5c09f699d28f932" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 49918 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I0703 01:40:20.493351 13400 consensus_queue.cc:962] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4610b121814a4c109a1458de7333ab01" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 40959 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s I0703 01:40:20.496168 13390 raft_consensus.cc:2684] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [term 3 LEADER]: Committing config change with OpId 3.7: config changed from index 6 to 7, e024b7ef889942f3b5c09f699d28f932 (127.5.240.195) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "654e91af3cb64033a64c989ab937d69e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 52067 } } peers { permanent_uuid: "e024b7ef889942f3b5c09f699d28f932" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 49918 } attrs { promote: false } } peers { permanent_uuid: "4610b121814a4c109a1458de7333ab01" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 40959 } attrs { promote: true } } unsafe_config_change: true } I0703 01:40:20.497021 12819 raft_consensus.cc:2684] T 5ce2098a78134ca29646d1cec9ed7c61 P e024b7ef889942f3b5c09f699d28f932 [term 3 FOLLOWER]: Committing config change with OpId 3.7: config changed from index 6 to 7, e024b7ef889942f3b5c09f699d28f932 (127.5.240.195) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "654e91af3cb64033a64c989ab937d69e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 52067 } } peers { permanent_uuid: "e024b7ef889942f3b5c09f699d28f932" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 49918 } attrs { promote: false } } peers { permanent_uuid: "4610b121814a4c109a1458de7333ab01" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 40959 } attrs { promote: true } } unsafe_config_change: true } I0703 01:40:20.497563 12941 raft_consensus.cc:2684] T 5ce2098a78134ca29646d1cec9ed7c61 P 4610b121814a4c109a1458de7333ab01 [term 3 LEARNER]: Committing config change with OpId 3.7: config changed from index 6 to 7, e024b7ef889942f3b5c09f699d28f932 (127.5.240.195) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "654e91af3cb64033a64c989ab937d69e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 52067 } } peers { permanent_uuid: "e024b7ef889942f3b5c09f699d28f932" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 49918 } attrs { promote: false } } peers { permanent_uuid: "4610b121814a4c109a1458de7333ab01" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 40959 } attrs { promote: true } } unsafe_config_change: true } I0703 01:40:20.498948 13390 raft_consensus.cc:890] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e: attempting to promote NON_VOTER 4610b121814a4c109a1458de7333ab01 to VOTER I0703 01:40:20.499222 13390 consensus_peers.cc:455] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e -> Peer 4610b121814a4c109a1458de7333ab01 (127.5.240.196:40959): Closing peer: 4610b121814a4c109a1458de7333ab01 I0703 01:40:20.499277 13390 consensus_peers.cc:455] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e -> Peer e024b7ef889942f3b5c09f699d28f932 (127.5.240.195:49918): Closing peer: e024b7ef889942f3b5c09f699d28f932 I0703 01:40:20.499421 13390 consensus_queue.cc:204] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 3.7, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "654e91af3cb64033a64c989ab937d69e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 52067 } } peers { permanent_uuid: "e024b7ef889942f3b5c09f699d28f932" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 49918 } attrs { promote: false } } peers { permanent_uuid: "4610b121814a4c109a1458de7333ab01" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 40959 } attrs { promote: false } } unsafe_config_change: true I0703 01:40:20.501788 12819 raft_consensus.cc:1081] T 5ce2098a78134ca29646d1cec9ed7c61 P e024b7ef889942f3b5c09f699d28f932 [term 3 FOLLOWER]: Refusing update from remote peer 654e91af3cb64033a64c989ab937d69e: Log matching property violated. Preceding OpId in replica: term: 3 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch) I0703 01:40:20.502557 13390 consensus_queue.cc:962] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e024b7ef889942f3b5c09f699d28f932" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 49918 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I0703 01:40:20.502261 13198 catalog_manager.cc:3798] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e reported cstate change: config changed from index 6 to 7, e024b7ef889942f3b5c09f699d28f932 (127.5.240.195) changed from NON_VOTER to VOTER, now has a pending config: VOTER 654e91af3cb64033a64c989ab937d69e (127.5.240.193), VOTER e024b7ef889942f3b5c09f699d28f932 (127.5.240.195), VOTER 4610b121814a4c109a1458de7333ab01 (127.5.240.196). New cstate: current_term: 3 leader_uuid: "654e91af3cb64033a64c989ab937d69e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "654e91af3cb64033a64c989ab937d69e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 52067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e024b7ef889942f3b5c09f699d28f932" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 49918 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4610b121814a4c109a1458de7333ab01" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 40959 } attrs { promote: true } health_report { overall_health: UNKNOWN } } unsafe_config_change: true } pending_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "654e91af3cb64033a64c989ab937d69e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 52067 } } peers { permanent_uuid: "e024b7ef889942f3b5c09f699d28f932" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 49918 } attrs { promote: false } } peers { permanent_uuid: "4610b121814a4c109a1458de7333ab01" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 40959 } attrs { promote: false } } unsafe_config_change: true } I0703 01:40:20.505115 12941 raft_consensus.cc:1081] T 5ce2098a78134ca29646d1cec9ed7c61 P 4610b121814a4c109a1458de7333ab01 [term 3 LEARNER]: Refusing update from remote peer 654e91af3cb64033a64c989ab937d69e: Log matching property violated. Preceding OpId in replica: term: 3 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch) I0703 01:40:20.505175 13400 raft_consensus.cc:2684] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [term 3 LEADER]: Committing config change with OpId 3.8: config changed from index 7 to 8, 4610b121814a4c109a1458de7333ab01 (127.5.240.196) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "654e91af3cb64033a64c989ab937d69e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 52067 } } peers { permanent_uuid: "e024b7ef889942f3b5c09f699d28f932" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 49918 } attrs { promote: false } } peers { permanent_uuid: "4610b121814a4c109a1458de7333ab01" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 40959 } attrs { promote: false } } unsafe_config_change: true } I0703 01:40:20.505921 12819 raft_consensus.cc:2684] T 5ce2098a78134ca29646d1cec9ed7c61 P e024b7ef889942f3b5c09f699d28f932 [term 3 FOLLOWER]: Committing config change with OpId 3.8: config changed from index 7 to 8, 4610b121814a4c109a1458de7333ab01 (127.5.240.196) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "654e91af3cb64033a64c989ab937d69e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 52067 } } peers { permanent_uuid: "e024b7ef889942f3b5c09f699d28f932" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 49918 } attrs { promote: false } } peers { permanent_uuid: "4610b121814a4c109a1458de7333ab01" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 40959 } attrs { promote: false } } unsafe_config_change: true } I0703 01:40:20.506621 13390 consensus_queue.cc:962] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4610b121814a4c109a1458de7333ab01" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 40959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s I0703 01:40:20.509171 12940 raft_consensus.cc:2684] T 5ce2098a78134ca29646d1cec9ed7c61 P 4610b121814a4c109a1458de7333ab01 [term 3 FOLLOWER]: Committing config change with OpId 3.8: config changed from index 7 to 8, 4610b121814a4c109a1458de7333ab01 (127.5.240.196) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "654e91af3cb64033a64c989ab937d69e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 52067 } } peers { permanent_uuid: "e024b7ef889942f3b5c09f699d28f932" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 49918 } attrs { promote: false } } peers { permanent_uuid: "4610b121814a4c109a1458de7333ab01" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 40959 } attrs { promote: false } } unsafe_config_change: true } I0703 01:40:20.513228 13197 catalog_manager.cc:3798] T 5ce2098a78134ca29646d1cec9ed7c61 P 654e91af3cb64033a64c989ab937d69e reported cstate change: config changed from index 7 to 8, 4610b121814a4c109a1458de7333ab01 (127.5.240.196) changed from NON_VOTER to VOTER, no longer has a pending config: VOTER 654e91af3cb64033a64c989ab937d69e (127.5.240.193), VOTER e024b7ef889942f3b5c09f699d28f932 (127.5.240.195), VOTER 4610b121814a4c109a1458de7333ab01 (127.5.240.196). New cstate: current_term: 3 leader_uuid: "654e91af3cb64033a64c989ab937d69e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "654e91af3cb64033a64c989ab937d69e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 52067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e024b7ef889942f3b5c09f699d28f932" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 49918 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4610b121814a4c109a1458de7333ab01" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 40959 } attrs { promote: false } health_report { overall_health: UNKNOWN } } unsafe_config_change: true } I0703 01:40:20.532565 6083 kudu-admin-test.cc:1021] Tablet locations: tablet_id: "5ce2098a78134ca29646d1cec9ed7c61" replicas { ts_info { permanent_uuid: "654e91af3cb64033a64c989ab937d69e" rpc_addresses { host: "127.5.240.193" port: 52067 } } role: LEADER } replicas { ts_info { permanent_uuid: "e024b7ef889942f3b5c09f699d28f932" rpc_addresses { host: "127.5.240.195" port: 49918 } } role: FOLLOWER } replicas { ts_info { permanent_uuid: "4610b121814a4c109a1458de7333ab01" rpc_addresses { host: "127.5.240.196" port: 40959 } } role: FOLLOWER } DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } I0703 01:40:20.533388 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 13249 I0703 01:40:20.538952 13368 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:40:20.584738 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 13249 W0703 01:40:20.593380 12872 connection.cc:512] server connection from 127.5.240.193:51607 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:51607): Connection reset by peer (error 104) W0703 01:40:20.593400 12750 connection.cc:512] client connection to 127.5.240.193:52067 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:52067): Connection reset by peer (error 104) W0703 01:40:20.593457 12752 connection.cc:512] server connection from 127.5.240.193:50210 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:50210): Connection reset by peer (error 104) W0703 01:40:20.593518 13180 connection.cc:512] server connection from 127.5.240.193:47098 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:47098): Connection reset by peer (error 104) W0703 01:40:20.593888 12872 connection.cc:512] client connection to 127.5.240.193:52067 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:52067): Connection reset by peer (error 104) I0703 01:40:20.594007 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 12743 I0703 01:40:20.594724 12859 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:40:20.637946 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 12743 I0703 01:40:20.646597 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 12865 W0703 01:40:20.646569 13183 connection.cc:512] server connection from 127.5.240.195:38995 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.195:38995): Connection reset by peer (error 104) I0703 01:40:20.647442 12981 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:40:20.690811 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 12865 W0703 01:40:20.698686 13181 connection.cc:512] server connection from 127.5.240.196:58480 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.196:58480): Connection reset by peer (error 104) I0703 01:40:20.698789 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master pid 13172 I0703 01:40:20.699512 13229 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:40:20.761098 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master with pid 13172 [ OK ] AdminCliTest.TestUnsafeChangeConfigWithPendingConfigsOnWAL (9882 ms) [ RUN ] AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs I0703 01:40:20.784497 6083 test_util.cc:208] Using random seed: 1135574374 I0703 01:40:20.786085 6083 ts_itest-base.cc:113] Starting cluster with: I0703 01:40:20.786108 6083 ts_itest-base.cc:114] -------------- I0703 01:40:20.786128 6083 ts_itest-base.cc:115] 9 tablet servers I0703 01:40:20.786147 6083 ts_itest-base.cc:116] 5 replicas per TS I0703 01:40:20.786167 6083 ts_itest-base.cc:117] -------------- I0703 01:40:20.787282 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:50751 --webserver_interface=127.5.240.254 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:40:20.966461 13421 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:40:20.966712 13421 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:40:20.968911 13421 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0703 01:40:20.968948 13421 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:40:20.968997 13421 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0703 01:40:20.969038 13421 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0703 01:40:20.976729 13421 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:50751 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.5.240.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:40:20.982307 13421 master_main.cc:80] Initializing master server... I0703 01:40:20.983443 13421 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 163708us I0703 01:40:20.984516 13421 fs_manager.cc:261] Metadata directory not provided I0703 01:40:20.984578 13421 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:40:20.985224 13421 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:40:20.985283 13421 server_base.cc:433] Attempting to create new FS layout instead I0703 01:40:20.989416 13421 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/instance: uuid: "541b5e7749d34adca822dc46c75b3e52" format_stamp: "Formatted at 2018-07-03 08:40:20 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:20.989954 13421 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "541b5e7749d34adca822dc46c75b3e52" format_stamp: "Formatted at 2018-07-03 08:40:20 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:20.998689 13421 fs_manager.cc:501] Time spent creating directory manager: real 0.008s user 0.003s sys 0.005s I0703 01:40:20.999896 13421 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:40:21.000298 13421 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:40:21.003489 13421 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0703 01:40:21.003540 13421 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal uuid: "541b5e7749d34adca822dc46c75b3e52" format_stamp: "Formatted at 2018-07-03 08:40:20 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:21.003674 13421 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:40:21.036787 13421 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:40:21.038182 13421 master_main.cc:83] Starting Master server... I0703 01:40:21.074725 13421 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.254:50751 I0703 01:40:21.075146 13421 webserver.cc:175] Starting webserver on 127.5.240.254:0 I0703 01:40:21.075186 13421 webserver.cc:186] Document root disabled I0703 01:40:21.076331 13421 webserver.cc:313] Webserver started. Bound to: http://127.5.240.254:36210/ I0703 01:40:21.076916 13421 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb I0703 01:40:21.077507 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master as pid 13421 I0703 01:40:21.078209 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.5.240.193:0 --local_ip_for_outbound_sockets=127.5.240.193 --webserver_interface=127.5.240.193 --webserver_port=0 --tserver_master_addrs=127.5.240.254:50751 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0703 01:40:21.080615 13480 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:40:21.092139 13480 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 541b5e7749d34adca822dc46c75b3e52: Bootstrap starting. I0703 01:40:21.094099 13480 tablet_bootstrap.cc:578] T 00000000000000000000000000000000 P 541b5e7749d34adca822dc46c75b3e52: No blocks or log segments found. Creating new log. I0703 01:40:21.094841 13480 log.cc:525] T 00000000000000000000000000000000 P 541b5e7749d34adca822dc46c75b3e52: Log is configured to *not* fsync() on all Append() calls I0703 01:40:21.097091 13480 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 541b5e7749d34adca822dc46c75b3e52: No bootstrap required, opened a new log I0703 01:40:21.103230 13480 raft_consensus.cc:325] T 00000000000000000000000000000000 P 541b5e7749d34adca822dc46c75b3e52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "541b5e7749d34adca822dc46c75b3e52" member_type: VOTER } I0703 01:40:21.103426 13480 raft_consensus.cc:351] T 00000000000000000000000000000000 P 541b5e7749d34adca822dc46c75b3e52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:40:21.103488 13480 raft_consensus.cc:628] T 00000000000000000000000000000000 P 541b5e7749d34adca822dc46c75b3e52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 541b5e7749d34adca822dc46c75b3e52, State: Initialized, Role: FOLLOWER I0703 01:40:21.103960 13480 consensus_queue.cc:227] T 00000000000000000000000000000000 P 541b5e7749d34adca822dc46c75b3e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "541b5e7749d34adca822dc46c75b3e52" member_type: VOTER } I0703 01:40:21.104107 13480 raft_consensus.cc:365] T 00000000000000000000000000000000 P 541b5e7749d34adca822dc46c75b3e52 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:40:21.104185 13480 raft_consensus.cc:457] T 00000000000000000000000000000000 P 541b5e7749d34adca822dc46c75b3e52 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:40:21.104262 13480 raft_consensus.cc:2730] T 00000000000000000000000000000000 P 541b5e7749d34adca822dc46c75b3e52: Snoozing failure detection for 1.655s (starting election) I0703 01:40:21.104349 13480 raft_consensus.cc:2782] T 00000000000000000000000000000000 P 541b5e7749d34adca822dc46c75b3e52 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:40:21.105743 13480 raft_consensus.cc:479] T 00000000000000000000000000000000 P 541b5e7749d34adca822dc46c75b3e52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "541b5e7749d34adca822dc46c75b3e52" member_type: VOTER } I0703 01:40:21.106109 13480 leader_election.cc:270] T 00000000000000000000000000000000 P 541b5e7749d34adca822dc46c75b3e52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:40:21.107626 13483 raft_consensus.cc:2730] T 00000000000000000000000000000000 P 541b5e7749d34adca822dc46c75b3e52: Snoozing failure detection for 2.136s (election complete) I0703 01:40:21.108028 13483 raft_consensus.cc:2537] T 00000000000000000000000000000000 P 541b5e7749d34adca822dc46c75b3e52 [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:40:21.109408 13483 raft_consensus.cc:593] T 00000000000000000000000000000000 P 541b5e7749d34adca822dc46c75b3e52 [term 1 LEADER]: Becoming Leader. State: Replica: 541b5e7749d34adca822dc46c75b3e52, State: Running, Role: LEADER I0703 01:40:21.109855 13483 consensus_queue.cc:204] T 00000000000000000000000000000000 P 541b5e7749d34adca822dc46c75b3e52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "541b5e7749d34adca822dc46c75b3e52" member_type: VOTER } I0703 01:40:21.113086 13480 sys_catalog.cc:337] T 00000000000000000000000000000000 P 541b5e7749d34adca822dc46c75b3e52 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "541b5e7749d34adca822dc46c75b3e52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "541b5e7749d34adca822dc46c75b3e52" member_type: VOTER } } I0703 01:40:21.113452 13480 sys_catalog.cc:340] T 00000000000000000000000000000000 P 541b5e7749d34adca822dc46c75b3e52 [sys.catalog]: This master's current role is: LEADER I0703 01:40:21.113157 13485 sys_catalog.cc:337] T 00000000000000000000000000000000 P 541b5e7749d34adca822dc46c75b3e52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 541b5e7749d34adca822dc46c75b3e52. Latest consensus state: current_term: 1 leader_uuid: "541b5e7749d34adca822dc46c75b3e52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "541b5e7749d34adca822dc46c75b3e52" member_type: VOTER } } I0703 01:40:21.113631 13485 sys_catalog.cc:340] T 00000000000000000000000000000000 P 541b5e7749d34adca822dc46c75b3e52 [sys.catalog]: This master's current role is: LEADER I0703 01:40:21.112556 13484 sys_catalog.cc:337] T 00000000000000000000000000000000 P 541b5e7749d34adca822dc46c75b3e52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "541b5e7749d34adca822dc46c75b3e52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "541b5e7749d34adca822dc46c75b3e52" member_type: VOTER } } I0703 01:40:21.113853 13484 sys_catalog.cc:340] T 00000000000000000000000000000000 P 541b5e7749d34adca822dc46c75b3e52 [sys.catalog]: This master's current role is: LEADER I0703 01:40:21.116564 13480 sys_catalog.cc:424] T 00000000000000000000000000000000 P 541b5e7749d34adca822dc46c75b3e52 [sys.catalog]: configured and running, proceeding with master startup. I0703 01:40:21.116837 13487 catalog_manager.cc:1006] Loading table and tablet metadata into memory... I0703 01:40:21.117851 13421 master_main.cc:86] Master server successfully started. I0703 01:40:21.122951 13487 catalog_manager.cc:1017] Initializing Kudu internal certificate authority... I0703 01:40:21.153069 13487 catalog_manager.cc:904] Generated new certificate authority record I0703 01:40:21.155026 13487 catalog_manager.cc:1026] Loading token signing keys... I0703 01:40:21.171142 13487 catalog_manager.cc:4102] T 00000000000000000000000000000000 P 541b5e7749d34adca822dc46c75b3e52: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:40:21.239421 13481 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:40:21.239676 13481 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:40:21.241690 13481 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:40:21.241796 13481 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.193 I0703 01:40:21.248270 13481 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.5.240.193 --webserver_port=0 --tserver_master_addrs=127.5.240.254:50751 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.193 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:40:21.254797 13481 tablet_server_main.cc:85] Initializing tablet server... I0703 01:40:21.255928 13481 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 164208us I0703 01:40:21.256875 13481 fs_manager.cc:261] Metadata directory not provided I0703 01:40:21.256940 13481 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:40:21.257545 13481 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:40:21.257634 13481 server_base.cc:433] Attempting to create new FS layout instead I0703 01:40:21.261768 13481 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "ff379419dfbc4624adc84127282b52d4" format_stamp: "Formatted at 2018-07-03 08:40:21 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:21.262221 13481 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "ff379419dfbc4624adc84127282b52d4" format_stamp: "Formatted at 2018-07-03 08:40:21 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:21.270826 13481 fs_manager.cc:501] Time spent creating directory manager: real 0.008s user 0.003s sys 0.004s I0703 01:40:21.272013 13481 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:40:21.272393 13481 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:40:21.275833 13481 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0703 01:40:21.275892 13481 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal uuid: "ff379419dfbc4624adc84127282b52d4" format_stamp: "Formatted at 2018-07-03 08:40:21 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:21.276008 13481 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:40:21.315953 13481 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:40:21.317950 13481 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:40:21.318361 13481 tablet_server_main.cc:90] Starting tablet server... I0703 01:40:21.434423 13481 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.193:49562 I0703 01:40:21.434917 13481 webserver.cc:175] Starting webserver on 127.5.240.193:0 I0703 01:40:21.434957 13481 webserver.cc:186] Document root disabled I0703 01:40:21.436287 13481 webserver.cc:313] Webserver started. Bound to: http://127.5.240.193:37748/ I0703 01:40:21.436841 13481 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb I0703 01:40:21.438930 13481 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:40:21.440217 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 13481 I0703 01:40:21.441099 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.5.240.194:0 --local_ip_for_outbound_sockets=127.5.240.194 --webserver_interface=127.5.240.194 --webserver_port=0 --tserver_master_addrs=127.5.240.254:50751 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0703 01:40:21.465185 13610 heartbeater.cc:346] Connected to a master server at 127.5.240.254:50751 I0703 01:40:21.465340 13610 heartbeater.cc:426] Registering TS with master... I0703 01:40:21.466174 13610 heartbeater.cc:475] Master 127.5.240.254:50751 requested a full tablet report, sending... I0703 01:40:21.467727 13447 ts_manager.cc:98] Registered new tserver with Master: ff379419dfbc4624adc84127282b52d4 (127.5.240.193:49562) I0703 01:40:21.470032 13447 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.193:40561 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:40:21.612643 13614 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:40:21.612890 13614 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:40:21.614917 13614 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:40:21.615023 13614 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.194 I0703 01:40:21.622031 13614 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.194:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.5.240.194 --webserver_port=0 --tserver_master_addrs=127.5.240.254:50751 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.194 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:40:21.628674 13614 tablet_server_main.cc:85] Initializing tablet server... I0703 01:40:21.629832 13614 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 164208us I0703 01:40:21.630796 13614 fs_manager.cc:261] Metadata directory not provided I0703 01:40:21.630858 13614 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:40:21.631095 13614 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:40:21.631160 13614 server_base.cc:433] Attempting to create new FS layout instead I0703 01:40:21.635181 13614 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "586291a7317b401c9f2f4bd48fe107e2" format_stamp: "Formatted at 2018-07-03 08:40:21 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:21.635721 13614 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "586291a7317b401c9f2f4bd48fe107e2" format_stamp: "Formatted at 2018-07-03 08:40:21 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:21.643048 13614 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.003s sys 0.003s I0703 01:40:21.644229 13614 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:40:21.644645 13614 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:40:21.648205 13614 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s I0703 01:40:21.648264 13614 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal uuid: "586291a7317b401c9f2f4bd48fe107e2" format_stamp: "Formatted at 2018-07-03 08:40:21 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:21.648382 13614 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:40:21.686957 13614 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:40:21.688733 13614 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:40:21.689038 13614 tablet_server_main.cc:90] Starting tablet server... I0703 01:40:21.772186 13614 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.194:53793 I0703 01:40:21.772634 13614 webserver.cc:175] Starting webserver on 127.5.240.194:0 I0703 01:40:21.772670 13614 webserver.cc:186] Document root disabled I0703 01:40:21.773834 13614 webserver.cc:313] Webserver started. Bound to: http://127.5.240.194:53744/ I0703 01:40:21.774353 13614 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb I0703 01:40:21.776346 13614 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:40:21.780925 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 13614 I0703 01:40:21.781530 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.5.240.195:0 --local_ip_for_outbound_sockets=127.5.240.195 --webserver_interface=127.5.240.195 --webserver_port=0 --tserver_master_addrs=127.5.240.254:50751 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0703 01:40:21.788946 13733 heartbeater.cc:346] Connected to a master server at 127.5.240.254:50751 I0703 01:40:21.789127 13733 heartbeater.cc:426] Registering TS with master... I0703 01:40:21.789796 13733 heartbeater.cc:475] Master 127.5.240.254:50751 requested a full tablet report, sending... I0703 01:40:21.791031 13447 ts_manager.cc:98] Registered new tserver with Master: 586291a7317b401c9f2f4bd48fe107e2 (127.5.240.194:53793) I0703 01:40:21.791895 13447 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.194:40987 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:40:21.943238 13736 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:40:21.943447 13736 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:40:21.945447 13736 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:40:21.945536 13736 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.195 I0703 01:40:21.952142 13736 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.195:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.5.240.195 --webserver_port=0 --tserver_master_addrs=127.5.240.254:50751 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.195 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:40:21.958735 13736 tablet_server_main.cc:85] Initializing tablet server... I0703 01:40:21.959867 13736 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 164208us I0703 01:40:21.960804 13736 fs_manager.cc:261] Metadata directory not provided I0703 01:40:21.960860 13736 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:40:21.961452 13736 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:40:21.961509 13736 server_base.cc:433] Attempting to create new FS layout instead I0703 01:40:21.965631 13736 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "01f745abbee546cc83f2570e2628db1a" format_stamp: "Formatted at 2018-07-03 08:40:21 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:21.966131 13736 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "01f745abbee546cc83f2570e2628db1a" format_stamp: "Formatted at 2018-07-03 08:40:21 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:21.973927 13736 fs_manager.cc:501] Time spent creating directory manager: real 0.008s user 0.005s sys 0.002s I0703 01:40:21.975121 13736 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:40:21.975507 13736 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:40:21.978619 13736 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0703 01:40:21.978670 13736 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal uuid: "01f745abbee546cc83f2570e2628db1a" format_stamp: "Formatted at 2018-07-03 08:40:21 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:21.978781 13736 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:40:22.052295 13736 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:40:22.054579 13736 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:40:22.054926 13736 tablet_server_main.cc:90] Starting tablet server... I0703 01:40:22.138695 13736 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.195:34366 I0703 01:40:22.139111 13736 webserver.cc:175] Starting webserver on 127.5.240.195:0 I0703 01:40:22.139143 13736 webserver.cc:186] Document root disabled I0703 01:40:22.139825 13736 webserver.cc:313] Webserver started. Bound to: http://127.5.240.195:44166/ I0703 01:40:22.140338 13736 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb I0703 01:40:22.142679 13736 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:40:22.146190 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 13736 I0703 01:40:22.146831 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.5.240.196:0 --local_ip_for_outbound_sockets=127.5.240.196 --webserver_interface=127.5.240.196 --webserver_port=0 --tserver_master_addrs=127.5.240.254:50751 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0703 01:40:22.153851 13855 heartbeater.cc:346] Connected to a master server at 127.5.240.254:50751 I0703 01:40:22.154003 13855 heartbeater.cc:426] Registering TS with master... I0703 01:40:22.154793 13855 heartbeater.cc:475] Master 127.5.240.254:50751 requested a full tablet report, sending... I0703 01:40:22.155874 13447 ts_manager.cc:98] Registered new tserver with Master: 01f745abbee546cc83f2570e2628db1a (127.5.240.195:34366) I0703 01:40:22.156738 13447 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.195:38956 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:40:22.304378 13858 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:40:22.304585 13858 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:40:22.306627 13858 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:40:22.306716 13858 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.196 I0703 01:40:22.313277 13858 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.196:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb --webserver_interface=127.5.240.196 --webserver_port=0 --tserver_master_addrs=127.5.240.254:50751 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.196 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:40:22.321033 13858 tablet_server_main.cc:85] Initializing tablet server... I0703 01:40:22.322144 13858 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 164708us I0703 01:40:22.323092 13858 fs_manager.cc:261] Metadata directory not provided I0703 01:40:22.323145 13858 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:40:22.323348 13858 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:40:22.323405 13858 server_base.cc:433] Attempting to create new FS layout instead I0703 01:40:22.327368 13858 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/instance: uuid: "e6994a142c374c4ba8577054aeaccbd7" format_stamp: "Formatted at 2018-07-03 08:40:22 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:22.327886 13858 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal/instance: uuid: "e6994a142c374c4ba8577054aeaccbd7" format_stamp: "Formatted at 2018-07-03 08:40:22 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:22.335094 13858 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.002s sys 0.004s I0703 01:40:22.336254 13858 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:40:22.336644 13858 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:40:22.340186 13858 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s I0703 01:40:22.340236 13858 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal uuid: "e6994a142c374c4ba8577054aeaccbd7" format_stamp: "Formatted at 2018-07-03 08:40:22 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:22.340345 13858 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:40:22.362470 13858 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:40:22.364702 13858 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:40:22.365051 13858 tablet_server_main.cc:90] Starting tablet server... I0703 01:40:22.448882 13858 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.196:59097 I0703 01:40:22.449297 13858 webserver.cc:175] Starting webserver on 127.5.240.196:0 I0703 01:40:22.449329 13858 webserver.cc:186] Document root disabled I0703 01:40:22.450011 13858 webserver.cc:313] Webserver started. Bound to: http://127.5.240.196:33133/ I0703 01:40:22.450515 13858 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb I0703 01:40:22.451104 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 13858 I0703 01:40:22.451987 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data --block_manager=log --rpc_bind_addresses=127.5.240.197:0 --local_ip_for_outbound_sockets=127.5.240.197 --webserver_interface=127.5.240.197 --webserver_port=0 --tserver_master_addrs=127.5.240.254:50751 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0703 01:40:22.452895 13858 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:40:22.464885 13977 heartbeater.cc:346] Connected to a master server at 127.5.240.254:50751 I0703 01:40:22.465046 13977 heartbeater.cc:426] Registering TS with master... I0703 01:40:22.465847 13977 heartbeater.cc:475] Master 127.5.240.254:50751 requested a full tablet report, sending... I0703 01:40:22.466895 13447 ts_manager.cc:98] Registered new tserver with Master: e6994a142c374c4ba8577054aeaccbd7 (127.5.240.196:59097) I0703 01:40:22.467754 13447 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.196:40101 I0703 01:40:22.472167 13610 heartbeater.cc:467] Master 127.5.240.254:50751 was elected leader, sending a full tablet report... WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:40:22.610255 13980 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:40:22.610466 13980 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:40:22.612530 13980 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:40:22.612671 13980 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.197 I0703 01:40:22.619096 13980 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.197:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb --webserver_interface=127.5.240.197 --webserver_port=0 --tserver_master_addrs=127.5.240.254:50751 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.197 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:40:22.625649 13980 tablet_server_main.cc:85] Initializing tablet server... I0703 01:40:22.626766 13980 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 164708us I0703 01:40:22.627708 13980 fs_manager.cc:261] Metadata directory not provided I0703 01:40:22.627761 13980 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:40:22.627974 13980 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:40:22.628033 13980 server_base.cc:433] Attempting to create new FS layout instead I0703 01:40:22.632078 13980 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/instance: uuid: "42bc35d124544280bcfdeca6d6f6049b" format_stamp: "Formatted at 2018-07-03 08:40:22 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:22.632568 13980 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal/instance: uuid: "42bc35d124544280bcfdeca6d6f6049b" format_stamp: "Formatted at 2018-07-03 08:40:22 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:22.639817 13980 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s I0703 01:40:22.640987 13980 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:40:22.641355 13980 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:40:22.645018 13980 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0703 01:40:22.645067 13980 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal uuid: "42bc35d124544280bcfdeca6d6f6049b" format_stamp: "Formatted at 2018-07-03 08:40:22 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:22.645175 13980 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:40:22.673693 13980 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:40:22.675940 13980 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:40:22.676283 13980 tablet_server_main.cc:90] Starting tablet server... I0703 01:40:22.761265 13980 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.197:54817 I0703 01:40:22.761734 13980 webserver.cc:175] Starting webserver on 127.5.240.197:0 I0703 01:40:22.761772 13980 webserver.cc:186] Document root disabled I0703 01:40:22.762461 13980 webserver.cc:313] Webserver started. Bound to: http://127.5.240.197:45074/ I0703 01:40:22.763003 13980 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb I0703 01:40:22.764698 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 13980 I0703 01:40:22.765527 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data --block_manager=log --rpc_bind_addresses=127.5.240.198:0 --local_ip_for_outbound_sockets=127.5.240.198 --webserver_interface=127.5.240.198 --webserver_port=0 --tserver_master_addrs=127.5.240.254:50751 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0703 01:40:22.769737 13980 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:40:22.777711 14099 heartbeater.cc:346] Connected to a master server at 127.5.240.254:50751 I0703 01:40:22.777856 14099 heartbeater.cc:426] Registering TS with master... I0703 01:40:22.778648 14099 heartbeater.cc:475] Master 127.5.240.254:50751 requested a full tablet report, sending... I0703 01:40:22.779667 13447 ts_manager.cc:98] Registered new tserver with Master: 42bc35d124544280bcfdeca6d6f6049b (127.5.240.197:54817) I0703 01:40:22.780484 13447 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.197:51961 I0703 01:40:22.794194 13733 heartbeater.cc:467] Master 127.5.240.254:50751 was elected leader, sending a full tablet report... WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:40:22.924870 14102 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:40:22.925076 14102 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:40:22.927109 14102 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:40:22.927197 14102 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.198 I0703 01:40:22.933699 14102 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.198:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data/info.pb --webserver_interface=127.5.240.198 --webserver_port=0 --tserver_master_addrs=127.5.240.254:50751 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.198 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:40:22.940125 14102 tablet_server_main.cc:85] Initializing tablet server... I0703 01:40:22.941218 14102 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 164708us I0703 01:40:22.942154 14102 fs_manager.cc:261] Metadata directory not provided I0703 01:40:22.942209 14102 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:40:22.942826 14102 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:40:22.942883 14102 server_base.cc:433] Attempting to create new FS layout instead I0703 01:40:22.946904 14102 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data/instance: uuid: "48755546d06649e78728d26d3b2bd82d" format_stamp: "Formatted at 2018-07-03 08:40:22 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:22.947384 14102 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/wal/instance: uuid: "48755546d06649e78728d26d3b2bd82d" format_stamp: "Formatted at 2018-07-03 08:40:22 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:22.955173 14102 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.004s sys 0.002s I0703 01:40:22.956409 14102 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:40:22.956813 14102 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:40:22.959843 14102 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0703 01:40:22.959894 14102 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/wal uuid: "48755546d06649e78728d26d3b2bd82d" format_stamp: "Formatted at 2018-07-03 08:40:22 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:22.960000 14102 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:40:22.992015 14102 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:40:22.994216 14102 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:40:22.994549 14102 tablet_server_main.cc:90] Starting tablet server... I0703 01:40:23.082269 14102 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.198:34012 I0703 01:40:23.082703 14102 webserver.cc:175] Starting webserver on 127.5.240.198:0 I0703 01:40:23.082736 14102 webserver.cc:186] Document root disabled I0703 01:40:23.083421 14102 webserver.cc:313] Webserver started. Bound to: http://127.5.240.198:53852/ I0703 01:40:23.083971 14102 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data/info.pb I0703 01:40:23.085808 14102 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:40:23.088824 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 14102 I0703 01:40:23.089426 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data --block_manager=log --rpc_bind_addresses=127.5.240.199:0 --local_ip_for_outbound_sockets=127.5.240.199 --webserver_interface=127.5.240.199 --webserver_port=0 --tserver_master_addrs=127.5.240.254:50751 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0703 01:40:23.097929 14221 heartbeater.cc:346] Connected to a master server at 127.5.240.254:50751 I0703 01:40:23.098088 14221 heartbeater.cc:426] Registering TS with master... I0703 01:40:23.098908 14221 heartbeater.cc:475] Master 127.5.240.254:50751 requested a full tablet report, sending... I0703 01:40:23.100013 13447 ts_manager.cc:98] Registered new tserver with Master: 48755546d06649e78728d26d3b2bd82d (127.5.240.198:34012) I0703 01:40:23.100888 13447 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.198:40135 I0703 01:40:23.158712 13855 heartbeater.cc:467] Master 127.5.240.254:50751 was elected leader, sending a full tablet report... WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:40:23.249212 14224 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:40:23.249431 14224 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:40:23.251405 14224 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:40:23.251497 14224 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.199 I0703 01:40:23.258502 14224 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.199:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data/info.pb --webserver_interface=127.5.240.199 --webserver_port=0 --tserver_master_addrs=127.5.240.254:50751 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.199 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:40:23.265018 14224 tablet_server_main.cc:85] Initializing tablet server... I0703 01:40:23.266136 14224 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 165208us I0703 01:40:23.267082 14224 fs_manager.cc:261] Metadata directory not provided I0703 01:40:23.267135 14224 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:40:23.267341 14224 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:40:23.267398 14224 server_base.cc:433] Attempting to create new FS layout instead I0703 01:40:23.271471 14224 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data/instance: uuid: "136de19b9fc544e9a3bddf205b65360d" format_stamp: "Formatted at 2018-07-03 08:40:23 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:23.271976 14224 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/wal/instance: uuid: "136de19b9fc544e9a3bddf205b65360d" format_stamp: "Formatted at 2018-07-03 08:40:23 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:23.279295 14224 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.004s sys 0.002s I0703 01:40:23.280455 14224 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:40:23.280840 14224 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:40:23.283870 14224 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0703 01:40:23.283922 14224 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/wal uuid: "136de19b9fc544e9a3bddf205b65360d" format_stamp: "Formatted at 2018-07-03 08:40:23 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:23.284029 14224 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:40:23.334323 14224 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:40:23.336122 14224 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:40:23.336532 14224 tablet_server_main.cc:90] Starting tablet server... I0703 01:40:23.423631 14224 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.199:40102 I0703 01:40:23.424015 14224 webserver.cc:175] Starting webserver on 127.5.240.199:0 I0703 01:40:23.424047 14224 webserver.cc:186] Document root disabled I0703 01:40:23.425185 14224 webserver.cc:313] Webserver started. Bound to: http://127.5.240.199:43815/ I0703 01:40:23.425716 14224 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data/info.pb I0703 01:40:23.427672 14224 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:40:23.431807 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 14224 I0703 01:40:23.432399 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-7/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-7/data --block_manager=log --rpc_bind_addresses=127.5.240.200:0 --local_ip_for_outbound_sockets=127.5.240.200 --webserver_interface=127.5.240.200 --webserver_port=0 --tserver_master_addrs=127.5.240.254:50751 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-7/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-7/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0703 01:40:23.438762 14343 heartbeater.cc:346] Connected to a master server at 127.5.240.254:50751 I0703 01:40:23.438917 14343 heartbeater.cc:426] Registering TS with master... I0703 01:40:23.439718 14343 heartbeater.cc:475] Master 127.5.240.254:50751 requested a full tablet report, sending... I0703 01:40:23.440758 13447 ts_manager.cc:98] Registered new tserver with Master: 136de19b9fc544e9a3bddf205b65360d (127.5.240.199:40102) I0703 01:40:23.441617 13447 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.199:34953 I0703 01:40:23.469566 13977 heartbeater.cc:467] Master 127.5.240.254:50751 was elected leader, sending a full tablet report... WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:40:23.589375 14346 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:40:23.589598 14346 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:40:23.591629 14346 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:40:23.591724 14346 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.200 I0703 01:40:23.598767 14346 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-7/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-7/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.200:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-7/data/info.pb --webserver_interface=127.5.240.200 --webserver_port=0 --tserver_master_addrs=127.5.240.254:50751 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.200 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-7/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:40:23.605240 14346 tablet_server_main.cc:85] Initializing tablet server... I0703 01:40:23.606343 14346 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 165208us I0703 01:40:23.607286 14346 fs_manager.cc:261] Metadata directory not provided I0703 01:40:23.607338 14346 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:40:23.607547 14346 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:40:23.607628 14346 server_base.cc:433] Attempting to create new FS layout instead I0703 01:40:23.611657 14346 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-7/data/instance: uuid: "0b034930462648f9b3f72485a0131667" format_stamp: "Formatted at 2018-07-03 08:40:23 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:23.612193 14346 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-7/wal/instance: uuid: "0b034930462648f9b3f72485a0131667" format_stamp: "Formatted at 2018-07-03 08:40:23 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:23.619410 14346 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.003s sys 0.003s I0703 01:40:23.620589 14346 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:40:23.620972 14346 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:40:23.624528 14346 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s I0703 01:40:23.624583 14346 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-7/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-7/wal uuid: "0b034930462648f9b3f72485a0131667" format_stamp: "Formatted at 2018-07-03 08:40:23 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:23.624708 14346 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-7/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-7/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-7/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:40:23.685081 14346 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:40:23.687616 14346 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:40:23.687989 14346 tablet_server_main.cc:90] Starting tablet server... I0703 01:40:23.772349 14346 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.200:52404 I0703 01:40:23.772784 14346 webserver.cc:175] Starting webserver on 127.5.240.200:0 I0703 01:40:23.772817 14346 webserver.cc:186] Document root disabled I0703 01:40:23.773443 14346 webserver.cc:313] Webserver started. Bound to: http://127.5.240.200:47315/ I0703 01:40:23.773999 14346 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-7/data/info.pb I0703 01:40:23.775423 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 14346 I0703 01:40:23.776329 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-8/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-8/data --block_manager=log --rpc_bind_addresses=127.5.240.201:0 --local_ip_for_outbound_sockets=127.5.240.201 --webserver_interface=127.5.240.201 --webserver_port=0 --tserver_master_addrs=127.5.240.254:50751 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-8/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-8/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0703 01:40:23.781044 14346 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:40:23.782739 14099 heartbeater.cc:467] Master 127.5.240.254:50751 was elected leader, sending a full tablet report... I0703 01:40:23.790386 14465 heartbeater.cc:346] Connected to a master server at 127.5.240.254:50751 I0703 01:40:23.790582 14465 heartbeater.cc:426] Registering TS with master... I0703 01:40:23.791431 14465 heartbeater.cc:475] Master 127.5.240.254:50751 requested a full tablet report, sending... I0703 01:40:23.792853 13447 ts_manager.cc:98] Registered new tserver with Master: 0b034930462648f9b3f72485a0131667 (127.5.240.200:52404) I0703 01:40:23.793720 13447 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.200:39037 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:40:23.935034 14468 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:40:23.935246 14468 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:40:23.937213 14468 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:40:23.937306 14468 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.201 I0703 01:40:23.944291 14468 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-8/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-8/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.201:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-8/data/info.pb --webserver_interface=127.5.240.201 --webserver_port=0 --tserver_master_addrs=127.5.240.254:50751 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.201 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-8/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:40:23.950917 14468 tablet_server_main.cc:85] Initializing tablet server... I0703 01:40:23.952064 14468 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 165208us I0703 01:40:23.953022 14468 fs_manager.cc:261] Metadata directory not provided I0703 01:40:23.953076 14468 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:40:23.953305 14468 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:40:23.953366 14468 server_base.cc:433] Attempting to create new FS layout instead I0703 01:40:23.957476 14468 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-8/data/instance: uuid: "f519569b3d3b42fdb0a434061155ac38" format_stamp: "Formatted at 2018-07-03 08:40:23 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:23.957979 14468 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-8/wal/instance: uuid: "f519569b3d3b42fdb0a434061155ac38" format_stamp: "Formatted at 2018-07-03 08:40:23 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:23.965287 14468 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.003s sys 0.003s I0703 01:40:23.966469 14468 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:40:23.966862 14468 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:40:23.970455 14468 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0703 01:40:23.970506 14468 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-8/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-8/wal uuid: "f519569b3d3b42fdb0a434061155ac38" format_stamp: "Formatted at 2018-07-03 08:40:23 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:23.970643 14468 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-8/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-8/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-8/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:40:23.996199 14468 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:40:23.997990 14468 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:40:23.998289 14468 tablet_server_main.cc:90] Starting tablet server... I0703 01:40:24.087556 14468 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.201:52406 I0703 01:40:24.087973 14468 webserver.cc:175] Starting webserver on 127.5.240.201:0 I0703 01:40:24.088007 14468 webserver.cc:186] Document root disabled I0703 01:40:24.089165 14468 webserver.cc:313] Webserver started. Bound to: http://127.5.240.201:60645/ I0703 01:40:24.089706 14468 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/ts-8/data/info.pb I0703 01:40:24.091089 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 14468 I0703 01:40:24.092306 14468 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:40:24.102666 14221 heartbeater.cc:467] Master 127.5.240.254:50751 was elected leader, sending a full tablet report... I0703 01:40:24.105535 14587 heartbeater.cc:346] Connected to a master server at 127.5.240.254:50751 I0703 01:40:24.105684 14587 heartbeater.cc:426] Registering TS with master... I0703 01:40:24.106436 14587 heartbeater.cc:475] Master 127.5.240.254:50751 requested a full tablet report, sending... I0703 01:40:24.107367 13446 ts_manager.cc:98] Registered new tserver with Master: f519569b3d3b42fdb0a434061155ac38 (127.5.240.201:52406) I0703 01:40:24.108209 13446 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.201:57983 I0703 01:40:24.109299 6083 external_mini_cluster.cc:490] 9 TS(s) registered with all masters I0703 01:40:24.119904 13446 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.5.240.254:57465: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 5 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0703 01:40:24.155979 14402 tablet_service.cc:759] Processing CreateTablet for tablet 3ea56191fad44303b1a8ce10e51741ed (table=TestTable [id=33bcc4be0f704acdab70821e538ab9f9]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:40:24.156728 14402 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 3ea56191fad44303b1a8ce10e51741ed. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:40:24.157075 13792 tablet_service.cc:759] Processing CreateTablet for tablet 3ea56191fad44303b1a8ce10e51741ed (table=TestTable [id=33bcc4be0f704acdab70821e538ab9f9]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:40:24.157822 13792 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 3ea56191fad44303b1a8ce10e51741ed. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:40:24.164183 14036 tablet_service.cc:759] Processing CreateTablet for tablet 3ea56191fad44303b1a8ce10e51741ed (table=TestTable [id=33bcc4be0f704acdab70821e538ab9f9]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:40:24.165138 14036 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 3ea56191fad44303b1a8ce10e51741ed. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:40:24.165797 14524 tablet_service.cc:759] Processing CreateTablet for tablet 3ea56191fad44303b1a8ce10e51741ed (table=TestTable [id=33bcc4be0f704acdab70821e538ab9f9]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:40:24.165797 14158 tablet_service.cc:759] Processing CreateTablet for tablet 3ea56191fad44303b1a8ce10e51741ed (table=TestTable [id=33bcc4be0f704acdab70821e538ab9f9]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:40:24.166555 14158 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 3ea56191fad44303b1a8ce10e51741ed. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:40:24.166579 14524 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 3ea56191fad44303b1a8ce10e51741ed. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:40:24.176435 13792 ts_tablet_manager.cc:1173] T 3ea56191fad44303b1a8ce10e51741ed P 01f745abbee546cc83f2570e2628db1a: Registered tablet (data state: TABLET_DATA_READY) I0703 01:40:24.178644 14036 ts_tablet_manager.cc:1173] T 3ea56191fad44303b1a8ce10e51741ed P 42bc35d124544280bcfdeca6d6f6049b: Registered tablet (data state: TABLET_DATA_READY) I0703 01:40:24.181736 14611 ts_tablet_manager.cc:1028] T 3ea56191fad44303b1a8ce10e51741ed P 01f745abbee546cc83f2570e2628db1a: Bootstrapping tablet I0703 01:40:24.182651 14611 tablet_bootstrap.cc:427] T 3ea56191fad44303b1a8ce10e51741ed P 01f745abbee546cc83f2570e2628db1a: Bootstrap starting. I0703 01:40:24.183787 14612 ts_tablet_manager.cc:1028] T 3ea56191fad44303b1a8ce10e51741ed P 42bc35d124544280bcfdeca6d6f6049b: Bootstrapping tablet I0703 01:40:24.183830 14524 ts_tablet_manager.cc:1173] T 3ea56191fad44303b1a8ce10e51741ed P f519569b3d3b42fdb0a434061155ac38: Registered tablet (data state: TABLET_DATA_READY) I0703 01:40:24.184036 14612 tablet_bootstrap.cc:427] T 3ea56191fad44303b1a8ce10e51741ed P 42bc35d124544280bcfdeca6d6f6049b: Bootstrap starting. I0703 01:40:24.186022 14611 tablet_bootstrap.cc:578] T 3ea56191fad44303b1a8ce10e51741ed P 01f745abbee546cc83f2570e2628db1a: No blocks or log segments found. Creating new log. I0703 01:40:24.187392 14158 ts_tablet_manager.cc:1173] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d: Registered tablet (data state: TABLET_DATA_READY) I0703 01:40:24.187662 14612 tablet_bootstrap.cc:578] T 3ea56191fad44303b1a8ce10e51741ed P 42bc35d124544280bcfdeca6d6f6049b: No blocks or log segments found. Creating new log. I0703 01:40:24.187733 14611 log.cc:525] T 3ea56191fad44303b1a8ce10e51741ed P 01f745abbee546cc83f2570e2628db1a: Log is configured to *not* fsync() on all Append() calls I0703 01:40:24.188371 14402 ts_tablet_manager.cc:1173] T 3ea56191fad44303b1a8ce10e51741ed P 0b034930462648f9b3f72485a0131667: Registered tablet (data state: TABLET_DATA_READY) I0703 01:40:24.188500 14612 log.cc:525] T 3ea56191fad44303b1a8ce10e51741ed P 42bc35d124544280bcfdeca6d6f6049b: Log is configured to *not* fsync() on all Append() calls I0703 01:40:24.189733 14613 ts_tablet_manager.cc:1028] T 3ea56191fad44303b1a8ce10e51741ed P f519569b3d3b42fdb0a434061155ac38: Bootstrapping tablet I0703 01:40:24.189982 14614 ts_tablet_manager.cc:1028] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d: Bootstrapping tablet I0703 01:40:24.190290 14614 tablet_bootstrap.cc:427] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d: Bootstrap starting. I0703 01:40:24.191614 14613 tablet_bootstrap.cc:427] T 3ea56191fad44303b1a8ce10e51741ed P f519569b3d3b42fdb0a434061155ac38: Bootstrap starting. I0703 01:40:24.192950 14616 ts_tablet_manager.cc:1028] T 3ea56191fad44303b1a8ce10e51741ed P 0b034930462648f9b3f72485a0131667: Bootstrapping tablet I0703 01:40:24.193958 14616 tablet_bootstrap.cc:427] T 3ea56191fad44303b1a8ce10e51741ed P 0b034930462648f9b3f72485a0131667: Bootstrap starting. I0703 01:40:24.194859 14614 tablet_bootstrap.cc:578] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d: No blocks or log segments found. Creating new log. I0703 01:40:24.195116 14613 tablet_bootstrap.cc:578] T 3ea56191fad44303b1a8ce10e51741ed P f519569b3d3b42fdb0a434061155ac38: No blocks or log segments found. Creating new log. I0703 01:40:24.195804 14614 log.cc:525] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d: Log is configured to *not* fsync() on all Append() calls I0703 01:40:24.195951 14611 tablet_bootstrap.cc:427] T 3ea56191fad44303b1a8ce10e51741ed P 01f745abbee546cc83f2570e2628db1a: No bootstrap required, opened a new log I0703 01:40:24.196175 14611 ts_tablet_manager.cc:1045] T 3ea56191fad44303b1a8ce10e51741ed P 01f745abbee546cc83f2570e2628db1a: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.005s I0703 01:40:24.196226 14613 log.cc:525] T 3ea56191fad44303b1a8ce10e51741ed P f519569b3d3b42fdb0a434061155ac38: Log is configured to *not* fsync() on all Append() calls I0703 01:40:24.197041 14616 tablet_bootstrap.cc:578] T 3ea56191fad44303b1a8ce10e51741ed P 0b034930462648f9b3f72485a0131667: No blocks or log segments found. Creating new log. I0703 01:40:24.197880 14616 log.cc:525] T 3ea56191fad44303b1a8ce10e51741ed P 0b034930462648f9b3f72485a0131667: Log is configured to *not* fsync() on all Append() calls I0703 01:40:24.200260 14612 tablet_bootstrap.cc:427] T 3ea56191fad44303b1a8ce10e51741ed P 42bc35d124544280bcfdeca6d6f6049b: No bootstrap required, opened a new log I0703 01:40:24.200387 14613 tablet_bootstrap.cc:427] T 3ea56191fad44303b1a8ce10e51741ed P f519569b3d3b42fdb0a434061155ac38: No bootstrap required, opened a new log I0703 01:40:24.200460 14612 ts_tablet_manager.cc:1045] T 3ea56191fad44303b1a8ce10e51741ed P 42bc35d124544280bcfdeca6d6f6049b: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.002s I0703 01:40:24.200584 14613 ts_tablet_manager.cc:1045] T 3ea56191fad44303b1a8ce10e51741ed P f519569b3d3b42fdb0a434061155ac38: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s I0703 01:40:24.202600 14616 tablet_bootstrap.cc:427] T 3ea56191fad44303b1a8ce10e51741ed P 0b034930462648f9b3f72485a0131667: No bootstrap required, opened a new log I0703 01:40:24.202805 14616 ts_tablet_manager.cc:1045] T 3ea56191fad44303b1a8ce10e51741ed P 0b034930462648f9b3f72485a0131667: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s I0703 01:40:24.205041 14611 raft_consensus.cc:325] T 3ea56191fad44303b1a8ce10e51741ed P 01f745abbee546cc83f2570e2628db1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42bc35d124544280bcfdeca6d6f6049b" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54817 } } peers { permanent_uuid: "01f745abbee546cc83f2570e2628db1a" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 34366 } } peers { permanent_uuid: "0b034930462648f9b3f72485a0131667" member_type: VOTER last_known_addr { host: "127.5.240.200" port: 52404 } } peers { permanent_uuid: "f519569b3d3b42fdb0a434061155ac38" member_type: VOTER last_known_addr { host: "127.5.240.201" port: 52406 } } peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } I0703 01:40:24.205472 14611 raft_consensus.cc:351] T 3ea56191fad44303b1a8ce10e51741ed P 01f745abbee546cc83f2570e2628db1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:40:24.205557 14611 raft_consensus.cc:628] T 3ea56191fad44303b1a8ce10e51741ed P 01f745abbee546cc83f2570e2628db1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01f745abbee546cc83f2570e2628db1a, State: Initialized, Role: FOLLOWER I0703 01:40:24.206081 14611 consensus_queue.cc:227] T 3ea56191fad44303b1a8ce10e51741ed P 01f745abbee546cc83f2570e2628db1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42bc35d124544280bcfdeca6d6f6049b" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54817 } } peers { permanent_uuid: "01f745abbee546cc83f2570e2628db1a" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 34366 } } peers { permanent_uuid: "0b034930462648f9b3f72485a0131667" member_type: VOTER last_known_addr { host: "127.5.240.200" port: 52404 } } peers { permanent_uuid: "f519569b3d3b42fdb0a434061155ac38" member_type: VOTER last_known_addr { host: "127.5.240.201" port: 52406 } } peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } I0703 01:40:24.209058 14612 raft_consensus.cc:325] T 3ea56191fad44303b1a8ce10e51741ed P 42bc35d124544280bcfdeca6d6f6049b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42bc35d124544280bcfdeca6d6f6049b" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54817 } } peers { permanent_uuid: "01f745abbee546cc83f2570e2628db1a" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 34366 } } peers { permanent_uuid: "0b034930462648f9b3f72485a0131667" member_type: VOTER last_known_addr { host: "127.5.240.200" port: 52404 } } peers { permanent_uuid: "f519569b3d3b42fdb0a434061155ac38" member_type: VOTER last_known_addr { host: "127.5.240.201" port: 52406 } } peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } I0703 01:40:24.209537 14612 raft_consensus.cc:351] T 3ea56191fad44303b1a8ce10e51741ed P 42bc35d124544280bcfdeca6d6f6049b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:40:24.209642 14612 raft_consensus.cc:628] T 3ea56191fad44303b1a8ce10e51741ed P 42bc35d124544280bcfdeca6d6f6049b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42bc35d124544280bcfdeca6d6f6049b, State: Initialized, Role: FOLLOWER I0703 01:40:24.210043 14612 consensus_queue.cc:227] T 3ea56191fad44303b1a8ce10e51741ed P 42bc35d124544280bcfdeca6d6f6049b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42bc35d124544280bcfdeca6d6f6049b" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54817 } } peers { permanent_uuid: "01f745abbee546cc83f2570e2628db1a" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 34366 } } peers { permanent_uuid: "0b034930462648f9b3f72485a0131667" member_type: VOTER last_known_addr { host: "127.5.240.200" port: 52404 } } peers { permanent_uuid: "f519569b3d3b42fdb0a434061155ac38" member_type: VOTER last_known_addr { host: "127.5.240.201" port: 52406 } } peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } I0703 01:40:24.210956 14614 tablet_bootstrap.cc:427] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d: No bootstrap required, opened a new log I0703 01:40:24.210714 14613 raft_consensus.cc:325] T 3ea56191fad44303b1a8ce10e51741ed P f519569b3d3b42fdb0a434061155ac38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42bc35d124544280bcfdeca6d6f6049b" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54817 } } peers { permanent_uuid: "01f745abbee546cc83f2570e2628db1a" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 34366 } } peers { permanent_uuid: "0b034930462648f9b3f72485a0131667" member_type: VOTER last_known_addr { host: "127.5.240.200" port: 52404 } } peers { permanent_uuid: "f519569b3d3b42fdb0a434061155ac38" member_type: VOTER last_known_addr { host: "127.5.240.201" port: 52406 } } peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } I0703 01:40:24.211177 14614 ts_tablet_manager.cc:1045] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.004s I0703 01:40:24.211274 14613 raft_consensus.cc:351] T 3ea56191fad44303b1a8ce10e51741ed P f519569b3d3b42fdb0a434061155ac38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:40:24.211378 14613 raft_consensus.cc:628] T 3ea56191fad44303b1a8ce10e51741ed P f519569b3d3b42fdb0a434061155ac38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f519569b3d3b42fdb0a434061155ac38, State: Initialized, Role: FOLLOWER I0703 01:40:24.211556 14616 raft_consensus.cc:325] T 3ea56191fad44303b1a8ce10e51741ed P 0b034930462648f9b3f72485a0131667 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42bc35d124544280bcfdeca6d6f6049b" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54817 } } peers { permanent_uuid: "01f745abbee546cc83f2570e2628db1a" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 34366 } } peers { permanent_uuid: "0b034930462648f9b3f72485a0131667" member_type: VOTER last_known_addr { host: "127.5.240.200" port: 52404 } } peers { permanent_uuid: "f519569b3d3b42fdb0a434061155ac38" member_type: VOTER last_known_addr { host: "127.5.240.201" port: 52406 } } peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } I0703 01:40:24.212257 14616 raft_consensus.cc:351] T 3ea56191fad44303b1a8ce10e51741ed P 0b034930462648f9b3f72485a0131667 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:40:24.212352 14616 raft_consensus.cc:628] T 3ea56191fad44303b1a8ce10e51741ed P 0b034930462648f9b3f72485a0131667 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b034930462648f9b3f72485a0131667, State: Initialized, Role: FOLLOWER I0703 01:40:24.211910 14613 consensus_queue.cc:227] T 3ea56191fad44303b1a8ce10e51741ed P f519569b3d3b42fdb0a434061155ac38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42bc35d124544280bcfdeca6d6f6049b" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54817 } } peers { permanent_uuid: "01f745abbee546cc83f2570e2628db1a" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 34366 } } peers { permanent_uuid: "0b034930462648f9b3f72485a0131667" member_type: VOTER last_known_addr { host: "127.5.240.200" port: 52404 } } peers { permanent_uuid: "f519569b3d3b42fdb0a434061155ac38" member_type: VOTER last_known_addr { host: "127.5.240.201" port: 52406 } } peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } I0703 01:40:24.212903 14616 consensus_queue.cc:227] T 3ea56191fad44303b1a8ce10e51741ed P 0b034930462648f9b3f72485a0131667 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42bc35d124544280bcfdeca6d6f6049b" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54817 } } peers { permanent_uuid: "01f745abbee546cc83f2570e2628db1a" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 34366 } } peers { permanent_uuid: "0b034930462648f9b3f72485a0131667" member_type: VOTER last_known_addr { host: "127.5.240.200" port: 52404 } } peers { permanent_uuid: "f519569b3d3b42fdb0a434061155ac38" member_type: VOTER last_known_addr { host: "127.5.240.201" port: 52406 } } peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } I0703 01:40:24.213279 14611 ts_tablet_manager.cc:1073] T 3ea56191fad44303b1a8ce10e51741ed P 01f745abbee546cc83f2570e2628db1a: Time spent starting tablet: real 0.017s user 0.009s sys 0.006s I0703 01:40:24.218474 14465 heartbeater.cc:467] Master 127.5.240.254:50751 was elected leader, sending a full tablet report... I0703 01:40:24.219099 14616 ts_tablet_manager.cc:1073] T 3ea56191fad44303b1a8ce10e51741ed P 0b034930462648f9b3f72485a0131667: Time spent starting tablet: real 0.016s user 0.008s sys 0.008s I0703 01:40:24.219863 14587 heartbeater.cc:467] Master 127.5.240.254:50751 was elected leader, sending a full tablet report... I0703 01:40:24.220357 14614 raft_consensus.cc:325] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42bc35d124544280bcfdeca6d6f6049b" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54817 } } peers { permanent_uuid: "01f745abbee546cc83f2570e2628db1a" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 34366 } } peers { permanent_uuid: "0b034930462648f9b3f72485a0131667" member_type: VOTER last_known_addr { host: "127.5.240.200" port: 52404 } } peers { permanent_uuid: "f519569b3d3b42fdb0a434061155ac38" member_type: VOTER last_known_addr { host: "127.5.240.201" port: 52406 } } peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } I0703 01:40:24.221137 14614 raft_consensus.cc:351] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:40:24.221717 14613 ts_tablet_manager.cc:1073] T 3ea56191fad44303b1a8ce10e51741ed P f519569b3d3b42fdb0a434061155ac38: Time spent starting tablet: real 0.021s user 0.009s sys 0.005s I0703 01:40:24.222375 14612 ts_tablet_manager.cc:1073] T 3ea56191fad44303b1a8ce10e51741ed P 42bc35d124544280bcfdeca6d6f6049b: Time spent starting tablet: real 0.022s user 0.007s sys 0.008s I0703 01:40:24.221226 14614 raft_consensus.cc:628] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48755546d06649e78728d26d3b2bd82d, State: Initialized, Role: FOLLOWER I0703 01:40:24.223011 14614 consensus_queue.cc:227] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42bc35d124544280bcfdeca6d6f6049b" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54817 } } peers { permanent_uuid: "01f745abbee546cc83f2570e2628db1a" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 34366 } } peers { permanent_uuid: "0b034930462648f9b3f72485a0131667" member_type: VOTER last_known_addr { host: "127.5.240.200" port: 52404 } } peers { permanent_uuid: "f519569b3d3b42fdb0a434061155ac38" member_type: VOTER last_known_addr { host: "127.5.240.201" port: 52406 } } peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } I0703 01:40:24.225231 14614 ts_tablet_manager.cc:1073] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d: Time spent starting tablet: real 0.014s user 0.007s sys 0.005s W0703 01:40:24.265756 14100 tablet.cc:1678] T 3ea56191fad44303b1a8ce10e51741ed P 42bc35d124544280bcfdeca6d6f6049b: Can't schedule compaction. Clean time has not been advanced past its initial value. W0703 01:40:24.281280 14466 tablet.cc:1678] T 3ea56191fad44303b1a8ce10e51741ed P 0b034930462648f9b3f72485a0131667: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:40:24.287514 14625 raft_consensus.cc:457] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0703 01:40:24.287638 14625 raft_consensus.cc:2730] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d: Snoozing failure detection for 1.849s (starting election) I0703 01:40:24.287752 14625 raft_consensus.cc:479] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42bc35d124544280bcfdeca6d6f6049b" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54817 } } peers { permanent_uuid: "01f745abbee546cc83f2570e2628db1a" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 34366 } } peers { permanent_uuid: "0b034930462648f9b3f72485a0131667" member_type: VOTER last_known_addr { host: "127.5.240.200" port: 52404 } } peers { permanent_uuid: "f519569b3d3b42fdb0a434061155ac38" member_type: VOTER last_known_addr { host: "127.5.240.201" port: 52406 } } peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } I0703 01:40:24.288975 14625 leader_election.cc:243] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 42bc35d124544280bcfdeca6d6f6049b (127.5.240.197:54817) I0703 01:40:24.289330 14625 leader_election.cc:243] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 01f745abbee546cc83f2570e2628db1a (127.5.240.195:34366) I0703 01:40:24.291203 14625 leader_election.cc:243] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 0b034930462648f9b3f72485a0131667 (127.5.240.200:52404) I0703 01:40:24.292677 14625 leader_election.cc:243] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer f519569b3d3b42fdb0a434061155ac38 (127.5.240.201:52406) I0703 01:40:24.305156 14056 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "3ea56191fad44303b1a8ce10e51741ed" candidate_uuid: "48755546d06649e78728d26d3b2bd82d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42bc35d124544280bcfdeca6d6f6049b" is_pre_election: true I0703 01:40:24.305807 14056 raft_consensus.cc:2730] T 3ea56191fad44303b1a8ce10e51741ed P 42bc35d124544280bcfdeca6d6f6049b: Snoozing failure detection for 1.819s (vote granted) I0703 01:40:24.305905 14056 raft_consensus.cc:2243] T 3ea56191fad44303b1a8ce10e51741ed P 42bc35d124544280bcfdeca6d6f6049b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48755546d06649e78728d26d3b2bd82d in term 0. I0703 01:40:24.306036 14422 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "3ea56191fad44303b1a8ce10e51741ed" candidate_uuid: "48755546d06649e78728d26d3b2bd82d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b034930462648f9b3f72485a0131667" is_pre_election: true I0703 01:40:24.306319 14422 raft_consensus.cc:2730] T 3ea56191fad44303b1a8ce10e51741ed P 0b034930462648f9b3f72485a0131667: Snoozing failure detection for 1.678s (vote granted) I0703 01:40:24.306433 14422 raft_consensus.cc:2243] T 3ea56191fad44303b1a8ce10e51741ed P 0b034930462648f9b3f72485a0131667 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48755546d06649e78728d26d3b2bd82d in term 0. I0703 01:40:24.306726 14111 leader_election.cc:379] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [CANDIDATE]: Term 1 pre-election: Vote granted by peer 42bc35d124544280bcfdeca6d6f6049b (127.5.240.197:54817) I0703 01:40:24.307154 14110 leader_election.cc:379] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [CANDIDATE]: Term 1 pre-election: Vote granted by peer 0b034930462648f9b3f72485a0131667 (127.5.240.200:52404) I0703 01:40:24.307212 14110 leader_election.cc:270] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0703 01:40:24.307480 14625 raft_consensus.cc:2730] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d: Snoozing failure detection for 1.702s (election complete) I0703 01:40:24.307554 14625 raft_consensus.cc:2537] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [term 0 FOLLOWER]: Leader pre-election won for term 1 I0703 01:40:24.307634 14625 raft_consensus.cc:457] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0703 01:40:24.307175 13812 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "3ea56191fad44303b1a8ce10e51741ed" candidate_uuid: "48755546d06649e78728d26d3b2bd82d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01f745abbee546cc83f2570e2628db1a" is_pre_election: true I0703 01:40:24.307703 14625 raft_consensus.cc:2730] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d: Snoozing failure detection for 1.635s (starting election) I0703 01:40:24.307763 14625 raft_consensus.cc:2782] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [term 0 FOLLOWER]: Advancing to term 1 I0703 01:40:24.307850 13812 raft_consensus.cc:2730] T 3ea56191fad44303b1a8ce10e51741ed P 01f745abbee546cc83f2570e2628db1a: Snoozing failure detection for 2.018s (vote granted) I0703 01:40:24.307955 13812 raft_consensus.cc:2243] T 3ea56191fad44303b1a8ce10e51741ed P 01f745abbee546cc83f2570e2628db1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48755546d06649e78728d26d3b2bd82d in term 0. I0703 01:40:24.308446 14110 leader_election.cc:379] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [CANDIDATE]: Term 1 pre-election: Vote granted by peer 01f745abbee546cc83f2570e2628db1a (127.5.240.195:34366) I0703 01:40:24.308503 14544 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "3ea56191fad44303b1a8ce10e51741ed" candidate_uuid: "48755546d06649e78728d26d3b2bd82d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f519569b3d3b42fdb0a434061155ac38" is_pre_election: true I0703 01:40:24.309224 14544 raft_consensus.cc:2730] T 3ea56191fad44303b1a8ce10e51741ed P f519569b3d3b42fdb0a434061155ac38: Snoozing failure detection for 2.053s (vote granted) I0703 01:40:24.309327 14544 raft_consensus.cc:2243] T 3ea56191fad44303b1a8ce10e51741ed P f519569b3d3b42fdb0a434061155ac38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48755546d06649e78728d26d3b2bd82d in term 0. I0703 01:40:24.309952 14110 leader_election.cc:379] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [CANDIDATE]: Term 1 pre-election: Vote granted by peer f519569b3d3b42fdb0a434061155ac38 (127.5.240.201:52406) I0703 01:40:24.310868 14625 raft_consensus.cc:479] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42bc35d124544280bcfdeca6d6f6049b" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54817 } } peers { permanent_uuid: "01f745abbee546cc83f2570e2628db1a" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 34366 } } peers { permanent_uuid: "0b034930462648f9b3f72485a0131667" member_type: VOTER last_known_addr { host: "127.5.240.200" port: 52404 } } peers { permanent_uuid: "f519569b3d3b42fdb0a434061155ac38" member_type: VOTER last_known_addr { host: "127.5.240.201" port: 52406 } } peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } I0703 01:40:24.311377 14625 leader_election.cc:243] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [CANDIDATE]: Term 1 election: Requesting vote from peer 42bc35d124544280bcfdeca6d6f6049b (127.5.240.197:54817) I0703 01:40:24.311517 14625 leader_election.cc:243] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [CANDIDATE]: Term 1 election: Requesting vote from peer 01f745abbee546cc83f2570e2628db1a (127.5.240.195:34366) I0703 01:40:24.311754 14625 leader_election.cc:243] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [CANDIDATE]: Term 1 election: Requesting vote from peer 0b034930462648f9b3f72485a0131667 (127.5.240.200:52404) I0703 01:40:24.312108 14056 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "3ea56191fad44303b1a8ce10e51741ed" candidate_uuid: "48755546d06649e78728d26d3b2bd82d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42bc35d124544280bcfdeca6d6f6049b" I0703 01:40:24.312166 14625 leader_election.cc:243] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [CANDIDATE]: Term 1 election: Requesting vote from peer f519569b3d3b42fdb0a434061155ac38 (127.5.240.201:52406) I0703 01:40:24.312183 13812 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "3ea56191fad44303b1a8ce10e51741ed" candidate_uuid: "48755546d06649e78728d26d3b2bd82d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01f745abbee546cc83f2570e2628db1a" I0703 01:40:24.312247 14056 raft_consensus.cc:2782] T 3ea56191fad44303b1a8ce10e51741ed P 42bc35d124544280bcfdeca6d6f6049b [term 0 FOLLOWER]: Advancing to term 1 I0703 01:40:24.312316 13812 raft_consensus.cc:2782] T 3ea56191fad44303b1a8ce10e51741ed P 01f745abbee546cc83f2570e2628db1a [term 0 FOLLOWER]: Advancing to term 1 I0703 01:40:24.312343 14056 raft_consensus.cc:2730] T 3ea56191fad44303b1a8ce10e51741ed P 42bc35d124544280bcfdeca6d6f6049b: Snoozing failure detection for 1.611s (vote granted) I0703 01:40:24.312420 13812 raft_consensus.cc:2730] T 3ea56191fad44303b1a8ce10e51741ed P 01f745abbee546cc83f2570e2628db1a: Snoozing failure detection for 2.102s (vote granted) I0703 01:40:24.312989 14422 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "3ea56191fad44303b1a8ce10e51741ed" candidate_uuid: "48755546d06649e78728d26d3b2bd82d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b034930462648f9b3f72485a0131667" I0703 01:40:24.313014 14544 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "3ea56191fad44303b1a8ce10e51741ed" candidate_uuid: "48755546d06649e78728d26d3b2bd82d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f519569b3d3b42fdb0a434061155ac38" I0703 01:40:24.313130 14422 raft_consensus.cc:2782] T 3ea56191fad44303b1a8ce10e51741ed P 0b034930462648f9b3f72485a0131667 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:40:24.313148 14544 raft_consensus.cc:2782] T 3ea56191fad44303b1a8ce10e51741ed P f519569b3d3b42fdb0a434061155ac38 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:40:24.313212 14422 raft_consensus.cc:2730] T 3ea56191fad44303b1a8ce10e51741ed P 0b034930462648f9b3f72485a0131667: Snoozing failure detection for 1.708s (vote granted) I0703 01:40:24.313244 14544 raft_consensus.cc:2730] T 3ea56191fad44303b1a8ce10e51741ed P f519569b3d3b42fdb0a434061155ac38: Snoozing failure detection for 1.730s (vote granted) I0703 01:40:24.314781 14056 raft_consensus.cc:2243] T 3ea56191fad44303b1a8ce10e51741ed P 42bc35d124544280bcfdeca6d6f6049b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48755546d06649e78728d26d3b2bd82d in term 1. I0703 01:40:24.314966 13812 raft_consensus.cc:2243] T 3ea56191fad44303b1a8ce10e51741ed P 01f745abbee546cc83f2570e2628db1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48755546d06649e78728d26d3b2bd82d in term 1. I0703 01:40:24.315232 14111 leader_election.cc:379] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [CANDIDATE]: Term 1 election: Vote granted by peer 42bc35d124544280bcfdeca6d6f6049b (127.5.240.197:54817) I0703 01:40:24.315527 14110 leader_election.cc:379] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [CANDIDATE]: Term 1 election: Vote granted by peer 01f745abbee546cc83f2570e2628db1a (127.5.240.195:34366) I0703 01:40:24.315613 14110 leader_election.cc:270] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:40:24.315610 14544 raft_consensus.cc:2243] T 3ea56191fad44303b1a8ce10e51741ed P f519569b3d3b42fdb0a434061155ac38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48755546d06649e78728d26d3b2bd82d in term 1. I0703 01:40:24.315771 14625 raft_consensus.cc:2730] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d: Snoozing failure detection for 1.927s (election complete) I0703 01:40:24.315852 14625 raft_consensus.cc:2537] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:40:24.316195 14110 leader_election.cc:379] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [CANDIDATE]: Term 1 election: Vote granted by peer f519569b3d3b42fdb0a434061155ac38 (127.5.240.201:52406) I0703 01:40:24.316917 14422 raft_consensus.cc:2243] T 3ea56191fad44303b1a8ce10e51741ed P 0b034930462648f9b3f72485a0131667 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48755546d06649e78728d26d3b2bd82d in term 1. I0703 01:40:24.317571 14110 leader_election.cc:379] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [CANDIDATE]: Term 1 election: Vote granted by peer 0b034930462648f9b3f72485a0131667 (127.5.240.200:52404) I0703 01:40:24.317603 14625 raft_consensus.cc:593] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [term 1 LEADER]: Becoming Leader. State: Replica: 48755546d06649e78728d26d3b2bd82d, State: Running, Role: LEADER I0703 01:40:24.318105 14625 consensus_queue.cc:204] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42bc35d124544280bcfdeca6d6f6049b" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54817 } } peers { permanent_uuid: "01f745abbee546cc83f2570e2628db1a" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 34366 } } peers { permanent_uuid: "0b034930462648f9b3f72485a0131667" member_type: VOTER last_known_addr { host: "127.5.240.200" port: 52404 } } peers { permanent_uuid: "f519569b3d3b42fdb0a434061155ac38" member_type: VOTER last_known_addr { host: "127.5.240.201" port: 52406 } } peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } I0703 01:40:24.322671 13447 catalog_manager.cc:3798] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d reported cstate change: term changed from 0 to 1, leader changed from to 48755546d06649e78728d26d3b2bd82d (127.5.240.198). New cstate: current_term: 1 leader_uuid: "48755546d06649e78728d26d3b2bd82d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42bc35d124544280bcfdeca6d6f6049b" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01f745abbee546cc83f2570e2628db1a" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 34366 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b034930462648f9b3f72485a0131667" member_type: VOTER last_known_addr { host: "127.5.240.200" port: 52404 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f519569b3d3b42fdb0a434061155ac38" member_type: VOTER last_known_addr { host: "127.5.240.201" port: 52406 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } health_report { overall_health: HEALTHY } } } I0703 01:40:24.335391 6083 external_mini_cluster.cc:490] 9 TS(s) registered with all masters W0703 01:40:24.336570 14222 tablet.cc:1678] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:40:24.336622 6083 ts_itest-base.cc:233] Waiting for 0 tablets on tserver ff379419dfbc4624adc84127282b52d4 to finish bootstrapping W0703 01:40:24.342458 14588 tablet.cc:1678] T 3ea56191fad44303b1a8ce10e51741ed P f519569b3d3b42fdb0a434061155ac38: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:40:24.343044 6083 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 586291a7317b401c9f2f4bd48fe107e2 to finish bootstrapping I0703 01:40:24.349352 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 01f745abbee546cc83f2570e2628db1a to finish bootstrapping I0703 01:40:24.354333 6083 ts_itest-base.cc:233] Waiting for 0 tablets on tserver e6994a142c374c4ba8577054aeaccbd7 to finish bootstrapping I0703 01:40:24.360772 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 42bc35d124544280bcfdeca6d6f6049b to finish bootstrapping I0703 01:40:24.366349 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 48755546d06649e78728d26d3b2bd82d to finish bootstrapping I0703 01:40:24.370712 6083 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 136de19b9fc544e9a3bddf205b65360d to finish bootstrapping I0703 01:40:24.377002 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 0b034930462648f9b3f72485a0131667 to finish bootstrapping I0703 01:40:24.382283 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver f519569b3d3b42fdb0a434061155ac38 to finish bootstrapping I0703 01:40:24.387017 6083 kudu-admin-test.cc:1059] Waiting for Master to see the current replicas... I0703 01:40:24.388377 6083 kudu-admin-test.cc:1066] Tablet locations: tablet_id: "3ea56191fad44303b1a8ce10e51741ed" replicas { ts_info { permanent_uuid: "42bc35d124544280bcfdeca6d6f6049b" rpc_addresses { host: "127.5.240.197" port: 54817 } } role: FOLLOWER } replicas { ts_info { permanent_uuid: "01f745abbee546cc83f2570e2628db1a" rpc_addresses { host: "127.5.240.195" port: 34366 } } role: FOLLOWER } replicas { ts_info { permanent_uuid: "0b034930462648f9b3f72485a0131667" rpc_addresses { host: "127.5.240.200" port: 52404 } } role: FOLLOWER } replicas { ts_info { permanent_uuid: "f519569b3d3b42fdb0a434061155ac38" rpc_addresses { host: "127.5.240.201" port: 52406 } } role: FOLLOWER } replicas { ts_info { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" rpc_addresses { host: "127.5.240.198" port: 34012 } } role: LEADER } DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } W0703 01:40:24.396289 13856 tablet.cc:1678] T 3ea56191fad44303b1a8ce10e51741ed P 01f745abbee546cc83f2570e2628db1a: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:40:24.408923 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 13980 I0703 01:40:24.409437 14096 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:40:24.443567 14343 heartbeater.cc:467] Master 127.5.240.254:50751 was elected leader, sending a full tablet report... I0703 01:40:24.452908 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 13980 W0703 01:40:24.460382 13431 connection.cc:512] server connection from 127.5.240.197:51961 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.197:51961): Connection reset by peer (error 104) W0703 01:40:24.460956 14111 connection.cc:512] client connection to 127.5.240.197:54817 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.197:54817): Connection reset by peer (error 104) I0703 01:40:24.461100 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 14346 I0703 01:40:24.461830 14462 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:40:24.504576 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 14346 W0703 01:40:24.512701 14110 connection.cc:512] client connection to 127.5.240.200:52404 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.200:52404): Connection reset by peer (error 104) I0703 01:40:24.512903 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 13736 W0703 01:40:24.513025 13428 connection.cc:512] server connection from 127.5.240.200:39037 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.200:39037): Connection reset by peer (error 104) I0703 01:40:24.513658 13852 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:40:24.556008 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 13736 W0703 01:40:24.564399 14110 connection.cc:512] client connection to 127.5.240.195:34366 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.195:34366): Connection reset by peer (error 104) W0703 01:40:24.564517 13429 connection.cc:512] server connection from 127.5.240.195:38956 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.195:38956): Connection reset by peer (error 104) I0703 01:40:24.564672 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 14468 I0703 01:40:24.565431 14584 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:40:24.608425 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 14468 W0703 01:40:24.616372 14110 connection.cc:512] client connection to 127.5.240.201:52406 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.201:52406): Connection reset by peer (error 104) W0703 01:40:24.616482 13429 connection.cc:512] server connection from 127.5.240.201:57983 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.201:57983): Connection reset by peer (error 104) I0703 01:40:24.616670 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master pid 13421 I0703 01:40:24.617410 13477 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:40:24.682046 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master with pid 13421 W0703 01:40:24.695603 13624 connection.cc:512] client connection to 127.5.240.254:50751 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.254:50751): Connection reset by peer (error 104) W0703 01:40:24.695575 13501 connection.cc:512] client connection to 127.5.240.254:50751 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.254:50751): Connection reset by peer (error 104) W0703 01:40:24.695660 13868 connection.cc:512] client connection to 127.5.240.254:50751 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.254:50751): Connection reset by peer (error 104) W0703 01:40:24.696255 14112 connection.cc:512] client connection to 127.5.240.254:50751 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.254:50751): Connection reset by peer (error 104) W0703 01:40:24.696292 14234 connection.cc:512] client connection to 127.5.240.254:50751 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.254:50751): Connection reset by peer (error 104) W0703 01:40:24.763906 14110 consensus_peers.cc:433] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d -> Peer f519569b3d3b42fdb0a434061155ac38 (127.5.240.201:52406): Couldn't send request to peer f519569b3d3b42fdb0a434061155ac38 for tablet 3ea56191fad44303b1a8ce10e51741ed. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.201:52406: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. W0703 01:40:24.798429 13733 heartbeater.cc:587] Failed to heartbeat to 127.5.240.254:50751: Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.240.254:50751: connect: Connection refused (error 111) W0703 01:40:24.816849 14111 consensus_peers.cc:433] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d -> Peer 42bc35d124544280bcfdeca6d6f6049b (127.5.240.197:54817): Couldn't send request to peer 42bc35d124544280bcfdeca6d6f6049b for tablet 3ea56191fad44303b1a8ce10e51741ed. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.197:54817: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. W0703 01:40:24.824654 14110 consensus_peers.cc:433] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d -> Peer 0b034930462648f9b3f72485a0131667 (127.5.240.200:52404): Couldn't send request to peer 0b034930462648f9b3f72485a0131667 for tablet 3ea56191fad44303b1a8ce10e51741ed. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.200:52404: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. W0703 01:40:24.859055 14110 consensus_peers.cc:433] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d -> Peer 01f745abbee546cc83f2570e2628db1a (127.5.240.195:34366): Couldn't send request to peer 01f745abbee546cc83f2570e2628db1a for tablet 3ea56191fad44303b1a8ce10e51741ed. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.195:34366: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. W0703 01:40:24.910136 14642 flags.cc:406] Enabled unsafe flag: --never_fsync=true I0703 01:40:24.958734 14178 tablet_service.cc:1077] Received UnsafeChangeConfig RPC: dest_uuid: "48755546d06649e78728d26d3b2bd82d" tablet_id: "3ea56191fad44303b1a8ce10e51741ed" caller_id: "kudu-tools" new_config { peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" } peers { permanent_uuid: "42bc35d124544280bcfdeca6d6f6049b" } peers { permanent_uuid: "0b034930462648f9b3f72485a0131667" } peers { permanent_uuid: "01f745abbee546cc83f2570e2628db1a" } peers { permanent_uuid: "f519569b3d3b42fdb0a434061155ac38" } } from {username='jenkins'} at 127.5.240.198:45241 W0703 01:40:24.959182 14178 raft_consensus.cc:2017] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [term 1 LEADER]: PROCEEDING WITH UNSAFE CONFIG CHANGE ON THIS SERVER, COMMITTED CONFIG: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42bc35d124544280bcfdeca6d6f6049b" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54817 } } peers { permanent_uuid: "01f745abbee546cc83f2570e2628db1a" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 34366 } } peers { permanent_uuid: "0b034930462648f9b3f72485a0131667" member_type: VOTER last_known_addr { host: "127.5.240.200" port: 52404 } } peers { permanent_uuid: "f519569b3d3b42fdb0a434061155ac38" member_type: VOTER last_known_addr { host: "127.5.240.201" port: 52406 } } peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } }NEW CONFIG: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42bc35d124544280bcfdeca6d6f6049b" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54817 } } peers { permanent_uuid: "01f745abbee546cc83f2570e2628db1a" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 34366 } } peers { permanent_uuid: "0b034930462648f9b3f72485a0131667" member_type: VOTER last_known_addr { host: "127.5.240.200" port: 52404 } } peers { permanent_uuid: "f519569b3d3b42fdb0a434061155ac38" member_type: VOTER last_known_addr { host: "127.5.240.201" port: 52406 } } peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } unsafe_config_change: true I0703 01:40:24.959627 14178 raft_consensus.cc:2777] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [term 1 LEADER]: Stepping down as leader of term 1 I0703 01:40:24.959673 14178 raft_consensus.cc:628] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 48755546d06649e78728d26d3b2bd82d, State: Running, Role: LEADER I0703 01:40:24.959918 14178 consensus_queue.cc:227] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42bc35d124544280bcfdeca6d6f6049b" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54817 } } peers { permanent_uuid: "01f745abbee546cc83f2570e2628db1a" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 34366 } } peers { permanent_uuid: "0b034930462648f9b3f72485a0131667" member_type: VOTER last_known_addr { host: "127.5.240.200" port: 52404 } } peers { permanent_uuid: "f519569b3d3b42fdb0a434061155ac38" member_type: VOTER last_known_addr { host: "127.5.240.201" port: 52406 } } peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } I0703 01:40:24.960093 14178 consensus_peers.cc:455] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d -> Peer 0b034930462648f9b3f72485a0131667 (127.5.240.200:52404): Closing peer: 0b034930462648f9b3f72485a0131667 I0703 01:40:24.960155 14178 consensus_peers.cc:455] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d -> Peer f519569b3d3b42fdb0a434061155ac38 (127.5.240.201:52406): Closing peer: f519569b3d3b42fdb0a434061155ac38 I0703 01:40:24.960204 14178 consensus_peers.cc:455] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d -> Peer 01f745abbee546cc83f2570e2628db1a (127.5.240.195:34366): Closing peer: 01f745abbee546cc83f2570e2628db1a I0703 01:40:24.960264 14178 consensus_peers.cc:455] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d -> Peer 42bc35d124544280bcfdeca6d6f6049b (127.5.240.197:54817): Closing peer: 42bc35d124544280bcfdeca6d6f6049b I0703 01:40:24.960436 14178 raft_consensus.cc:2782] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [term 1 FOLLOWER]: Advancing to term 2 W0703 01:40:24.963717 14221 heartbeater.cc:587] Failed to heartbeat to 127.5.240.254:50751: Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.240.254:50751: connect: Connection refused (error 111) W0703 01:40:25.221061 14657 flags.cc:406] Enabled unsafe flag: --never_fsync=true I0703 01:40:25.269927 14178 tablet_service.cc:1077] Received UnsafeChangeConfig RPC: dest_uuid: "48755546d06649e78728d26d3b2bd82d" tablet_id: "3ea56191fad44303b1a8ce10e51741ed" caller_id: "kudu-tools" new_config { peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" } peers { permanent_uuid: "42bc35d124544280bcfdeca6d6f6049b" } peers { permanent_uuid: "0b034930462648f9b3f72485a0131667" } peers { permanent_uuid: "01f745abbee546cc83f2570e2628db1a" } } from {username='jenkins'} at 127.5.240.198:45243 W0703 01:40:25.270138 14178 raft_consensus.cc:1918] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [term 2 FOLLOWER]: Replica has a pending config, but the new config will be unsafely changed anyway. Currently pending config on the node: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42bc35d124544280bcfdeca6d6f6049b" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54817 } } peers { permanent_uuid: "01f745abbee546cc83f2570e2628db1a" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 34366 } } peers { permanent_uuid: "0b034930462648f9b3f72485a0131667" member_type: VOTER last_known_addr { host: "127.5.240.200" port: 52404 } } peers { permanent_uuid: "f519569b3d3b42fdb0a434061155ac38" member_type: VOTER last_known_addr { host: "127.5.240.201" port: 52406 } } peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } unsafe_config_change: true W0703 01:40:25.270489 14178 raft_consensus.cc:2017] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [term 2 FOLLOWER]: PROCEEDING WITH UNSAFE CONFIG CHANGE ON THIS SERVER, COMMITTED CONFIG: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42bc35d124544280bcfdeca6d6f6049b" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54817 } } peers { permanent_uuid: "01f745abbee546cc83f2570e2628db1a" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 34366 } } peers { permanent_uuid: "0b034930462648f9b3f72485a0131667" member_type: VOTER last_known_addr { host: "127.5.240.200" port: 52404 } } peers { permanent_uuid: "f519569b3d3b42fdb0a434061155ac38" member_type: VOTER last_known_addr { host: "127.5.240.201" port: 52406 } } peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } }NEW CONFIG: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42bc35d124544280bcfdeca6d6f6049b" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54817 } } peers { permanent_uuid: "01f745abbee546cc83f2570e2628db1a" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 34366 } } peers { permanent_uuid: "0b034930462648f9b3f72485a0131667" member_type: VOTER last_known_addr { host: "127.5.240.200" port: 52404 } } peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } unsafe_config_change: true I0703 01:40:25.270799 14178 raft_consensus.cc:2782] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [term 2 FOLLOWER]: Advancing to term 3 I0703 01:40:25.272445 14178 raft_consensus.cc:2849] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [term 3 FOLLOWER]: Allowing unsafe config change even though there is a pending config! Existing pending config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42bc35d124544280bcfdeca6d6f6049b" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54817 } } peers { permanent_uuid: "01f745abbee546cc83f2570e2628db1a" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 34366 } } peers { permanent_uuid: "0b034930462648f9b3f72485a0131667" member_type: VOTER last_known_addr { host: "127.5.240.200" port: 52404 } } peers { permanent_uuid: "f519569b3d3b42fdb0a434061155ac38" member_type: VOTER last_known_addr { host: "127.5.240.201" port: 52406 } } peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } unsafe_config_change: true; New pending config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42bc35d124544280bcfdeca6d6f6049b" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54817 } } peers { permanent_uuid: "01f745abbee546cc83f2570e2628db1a" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 34366 } } peers { permanent_uuid: "0b034930462648f9b3f72485a0131667" member_type: VOTER last_known_addr { host: "127.5.240.200" port: 52404 } } peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } unsafe_config_change: true W0703 01:40:25.273180 14221 heartbeater.cc:587] Failed to heartbeat to 127.5.240.254:50751: Network error: Failed to ping master at 127.5.240.254:50751: Client connection negotiation failed: client connection to 127.5.240.254:50751: connect: Connection refused (error 111) W0703 01:40:25.446914 14343 heartbeater.cc:587] Failed to heartbeat to 127.5.240.254:50751: Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.240.254:50751: connect: Connection refused (error 111) W0703 01:40:25.474586 13977 heartbeater.cc:587] Failed to heartbeat to 127.5.240.254:50751: Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.240.254:50751: connect: Connection refused (error 111) W0703 01:40:25.478060 13610 heartbeater.cc:587] Failed to heartbeat to 127.5.240.254:50751: Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.240.254:50751: connect: Connection refused (error 111) W0703 01:40:25.530167 14670 flags.cc:406] Enabled unsafe flag: --never_fsync=true I0703 01:40:25.578210 14178 tablet_service.cc:1077] Received UnsafeChangeConfig RPC: dest_uuid: "48755546d06649e78728d26d3b2bd82d" tablet_id: "3ea56191fad44303b1a8ce10e51741ed" caller_id: "kudu-tools" new_config { peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" } peers { permanent_uuid: "42bc35d124544280bcfdeca6d6f6049b" } peers { permanent_uuid: "0b034930462648f9b3f72485a0131667" } } from {username='jenkins'} at 127.5.240.198:45245 W0703 01:40:25.578418 14178 raft_consensus.cc:1918] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [term 3 FOLLOWER]: Replica has a pending config, but the new config will be unsafely changed anyway. Currently pending config on the node: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42bc35d124544280bcfdeca6d6f6049b" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54817 } } peers { permanent_uuid: "01f745abbee546cc83f2570e2628db1a" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 34366 } } peers { permanent_uuid: "0b034930462648f9b3f72485a0131667" member_type: VOTER last_known_addr { host: "127.5.240.200" port: 52404 } } peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } unsafe_config_change: true W0703 01:40:25.578752 14178 raft_consensus.cc:2017] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [term 3 FOLLOWER]: PROCEEDING WITH UNSAFE CONFIG CHANGE ON THIS SERVER, COMMITTED CONFIG: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42bc35d124544280bcfdeca6d6f6049b" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54817 } } peers { permanent_uuid: "01f745abbee546cc83f2570e2628db1a" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 34366 } } peers { permanent_uuid: "0b034930462648f9b3f72485a0131667" member_type: VOTER last_known_addr { host: "127.5.240.200" port: 52404 } } peers { permanent_uuid: "f519569b3d3b42fdb0a434061155ac38" member_type: VOTER last_known_addr { host: "127.5.240.201" port: 52406 } } peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } }NEW CONFIG: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42bc35d124544280bcfdeca6d6f6049b" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54817 } } peers { permanent_uuid: "0b034930462648f9b3f72485a0131667" member_type: VOTER last_known_addr { host: "127.5.240.200" port: 52404 } } peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } unsafe_config_change: true I0703 01:40:25.579030 14178 raft_consensus.cc:2782] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [term 3 FOLLOWER]: Advancing to term 4 I0703 01:40:25.580586 14178 raft_consensus.cc:2849] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [term 4 FOLLOWER]: Allowing unsafe config change even though there is a pending config! Existing pending config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42bc35d124544280bcfdeca6d6f6049b" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54817 } } peers { permanent_uuid: "01f745abbee546cc83f2570e2628db1a" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 34366 } } peers { permanent_uuid: "0b034930462648f9b3f72485a0131667" member_type: VOTER last_known_addr { host: "127.5.240.200" port: 52404 } } peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } unsafe_config_change: true; New pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42bc35d124544280bcfdeca6d6f6049b" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54817 } } peers { permanent_uuid: "0b034930462648f9b3f72485a0131667" member_type: VOTER last_known_addr { host: "127.5.240.200" port: 52404 } } peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } unsafe_config_change: true W0703 01:40:25.581468 14221 heartbeater.cc:587] Failed to heartbeat to 127.5.240.254:50751: Network error: Failed to ping master at 127.5.240.254:50751: Client connection negotiation failed: client connection to 127.5.240.254:50751: connect: Connection refused (error 111) W0703 01:40:25.799752 13733 heartbeater.cc:587] Failed to heartbeat to 127.5.240.254:50751: Network error: Failed to ping master at 127.5.240.254:50751: Client connection negotiation failed: client connection to 127.5.240.254:50751: connect: Connection refused (error 111) W0703 01:40:25.836060 14686 flags.cc:406] Enabled unsafe flag: --never_fsync=true I0703 01:40:25.884392 14178 tablet_service.cc:1077] Received UnsafeChangeConfig RPC: dest_uuid: "48755546d06649e78728d26d3b2bd82d" tablet_id: "3ea56191fad44303b1a8ce10e51741ed" caller_id: "kudu-tools" new_config { peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" } peers { permanent_uuid: "42bc35d124544280bcfdeca6d6f6049b" } } from {username='jenkins'} at 127.5.240.198:45247 W0703 01:40:25.884621 14178 raft_consensus.cc:1918] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [term 4 FOLLOWER]: Replica has a pending config, but the new config will be unsafely changed anyway. Currently pending config on the node: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42bc35d124544280bcfdeca6d6f6049b" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54817 } } peers { permanent_uuid: "0b034930462648f9b3f72485a0131667" member_type: VOTER last_known_addr { host: "127.5.240.200" port: 52404 } } peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } unsafe_config_change: true W0703 01:40:25.884928 14178 raft_consensus.cc:2017] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [term 4 FOLLOWER]: PROCEEDING WITH UNSAFE CONFIG CHANGE ON THIS SERVER, COMMITTED CONFIG: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42bc35d124544280bcfdeca6d6f6049b" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54817 } } peers { permanent_uuid: "01f745abbee546cc83f2570e2628db1a" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 34366 } } peers { permanent_uuid: "0b034930462648f9b3f72485a0131667" member_type: VOTER last_known_addr { host: "127.5.240.200" port: 52404 } } peers { permanent_uuid: "f519569b3d3b42fdb0a434061155ac38" member_type: VOTER last_known_addr { host: "127.5.240.201" port: 52406 } } peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } }NEW CONFIG: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "42bc35d124544280bcfdeca6d6f6049b" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54817 } } peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } unsafe_config_change: true I0703 01:40:25.885190 14178 raft_consensus.cc:2782] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [term 4 FOLLOWER]: Advancing to term 5 I0703 01:40:25.886762 14178 raft_consensus.cc:2849] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [term 5 FOLLOWER]: Allowing unsafe config change even though there is a pending config! Existing pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42bc35d124544280bcfdeca6d6f6049b" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54817 } } peers { permanent_uuid: "0b034930462648f9b3f72485a0131667" member_type: VOTER last_known_addr { host: "127.5.240.200" port: 52404 } } peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } unsafe_config_change: true; New pending config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "42bc35d124544280bcfdeca6d6f6049b" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54817 } } peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } unsafe_config_change: true W0703 01:40:25.887521 14221 heartbeater.cc:587] Failed to heartbeat to 127.5.240.254:50751: Network error: Failed to ping master at 127.5.240.254:50751: Client connection negotiation failed: client connection to 127.5.240.254:50751: connect: Connection refused (error 111) W0703 01:40:26.141754 14700 flags.cc:406] Enabled unsafe flag: --never_fsync=true I0703 01:40:26.189306 14178 tablet_service.cc:1077] Received UnsafeChangeConfig RPC: dest_uuid: "48755546d06649e78728d26d3b2bd82d" tablet_id: "3ea56191fad44303b1a8ce10e51741ed" caller_id: "kudu-tools" new_config { peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" } } from {username='jenkins'} at 127.5.240.198:45249 W0703 01:40:26.189507 14178 raft_consensus.cc:1918] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [term 5 FOLLOWER]: Replica has a pending config, but the new config will be unsafely changed anyway. Currently pending config on the node: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "42bc35d124544280bcfdeca6d6f6049b" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54817 } } peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } unsafe_config_change: true W0703 01:40:26.189795 14178 raft_consensus.cc:2017] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [term 5 FOLLOWER]: PROCEEDING WITH UNSAFE CONFIG CHANGE ON THIS SERVER, COMMITTED CONFIG: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42bc35d124544280bcfdeca6d6f6049b" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54817 } } peers { permanent_uuid: "01f745abbee546cc83f2570e2628db1a" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 34366 } } peers { permanent_uuid: "0b034930462648f9b3f72485a0131667" member_type: VOTER last_known_addr { host: "127.5.240.200" port: 52404 } } peers { permanent_uuid: "f519569b3d3b42fdb0a434061155ac38" member_type: VOTER last_known_addr { host: "127.5.240.201" port: 52406 } } peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } }NEW CONFIG: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } unsafe_config_change: true I0703 01:40:26.190060 14178 raft_consensus.cc:2782] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [term 5 FOLLOWER]: Advancing to term 6 I0703 01:40:26.191612 14178 raft_consensus.cc:2849] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [term 6 FOLLOWER]: Allowing unsafe config change even though there is a pending config! Existing pending config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "42bc35d124544280bcfdeca6d6f6049b" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54817 } } peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } unsafe_config_change: true; New pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } unsafe_config_change: true W0703 01:40:26.192337 14221 heartbeater.cc:587] Failed to heartbeat to 127.5.240.254:50751: Network error: Failed to ping master at 127.5.240.254:50751: Client connection negotiation failed: client connection to 127.5.240.254:50751: connect: Connection refused (error 111) W0703 01:40:26.448132 14343 heartbeater.cc:587] Failed to heartbeat to 127.5.240.254:50751: Network error: Failed to ping master at 127.5.240.254:50751: Client connection negotiation failed: client connection to 127.5.240.254:50751: connect: Connection refused (error 111) W0703 01:40:26.475849 13977 heartbeater.cc:587] Failed to heartbeat to 127.5.240.254:50751: Network error: Failed to ping master at 127.5.240.254:50751: Client connection negotiation failed: client connection to 127.5.240.254:50751: connect: Connection refused (error 111) W0703 01:40:26.479238 13610 heartbeater.cc:587] Failed to heartbeat to 127.5.240.254:50751: Network error: Failed to ping master at 127.5.240.254:50751: Client connection negotiation failed: client connection to 127.5.240.254:50751: connect: Connection refused (error 111) W0703 01:40:26.801061 13733 heartbeater.cc:587] Failed to heartbeat to 127.5.240.254:50751: Network error: Failed to ping master at 127.5.240.254:50751: Client connection negotiation failed: client connection to 127.5.240.254:50751: connect: Connection refused (error 111) W0703 01:40:27.193367 14221 heartbeater.cc:587] Failed to heartbeat to 127.5.240.254:50751: Network error: Failed to ping master at 127.5.240.254:50751: Client connection negotiation failed: client connection to 127.5.240.254:50751: connect: Connection refused (error 111) W0703 01:40:27.449373 14343 heartbeater.cc:587] Failed to heartbeat to 127.5.240.254:50751: Network error: Failed to ping master at 127.5.240.254:50751: Client connection negotiation failed: client connection to 127.5.240.254:50751: connect: Connection refused (error 111) W0703 01:40:27.476933 13977 heartbeater.cc:587] Failed to heartbeat to 127.5.240.254:50751: Network error: Failed to ping master at 127.5.240.254:50751: Client connection negotiation failed: client connection to 127.5.240.254:50751: connect: Connection refused (error 111) W0703 01:40:27.480267 13610 heartbeater.cc:587] Failed to heartbeat to 127.5.240.254:50751: Network error: Failed to ping master at 127.5.240.254:50751: Client connection negotiation failed: client connection to 127.5.240.254:50751: connect: Connection refused (error 111) I0703 01:40:27.692255 14721 raft_consensus.cc:457] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader kudu-tools) I0703 01:40:27.692332 14721 raft_consensus.cc:2730] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d: Snoozing failure detection for 2.019s (starting election) I0703 01:40:27.692411 14721 raft_consensus.cc:479] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } unsafe_config_change: true I0703 01:40:27.692596 14721 leader_election.cc:270] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. I0703 01:40:27.693006 14721 raft_consensus.cc:2730] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d: Snoozing failure detection for 1.678s (election complete) I0703 01:40:27.693104 14721 raft_consensus.cc:2537] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [term 6 FOLLOWER]: Leader pre-election won for term 7 I0703 01:40:27.693157 14721 raft_consensus.cc:457] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [term 6 FOLLOWER]: Starting leader election (detected failure of leader kudu-tools) I0703 01:40:27.693210 14721 raft_consensus.cc:2730] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d: Snoozing failure detection for 1.875s (starting election) I0703 01:40:27.693274 14721 raft_consensus.cc:2782] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [term 6 FOLLOWER]: Advancing to term 7 I0703 01:40:27.694650 14721 raft_consensus.cc:479] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } unsafe_config_change: true I0703 01:40:27.694799 14721 leader_election.cc:270] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. I0703 01:40:27.694981 14722 raft_consensus.cc:2730] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d: Snoozing failure detection for 1.656s (election complete) I0703 01:40:27.695045 14722 raft_consensus.cc:2537] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [term 7 FOLLOWER]: Leader election won for term 7 I0703 01:40:27.695169 14722 raft_consensus.cc:593] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [term 7 LEADER]: Becoming Leader. State: Replica: 48755546d06649e78728d26d3b2bd82d, State: Running, Role: LEADER I0703 01:40:27.695448 14722 consensus_queue.cc:204] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 6.6, Last appended by leader: 0, Current term: 7, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } unsafe_config_change: true W0703 01:40:27.696741 14221 heartbeater.cc:587] Failed to heartbeat to 127.5.240.254:50751: Network error: Failed to ping master at 127.5.240.254:50751: Client connection negotiation failed: client connection to 127.5.240.254:50751: connect: Connection refused (error 111) I0703 01:40:27.697356 14722 raft_consensus.cc:2684] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [term 7 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42bc35d124544280bcfdeca6d6f6049b" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54817 } } peers { permanent_uuid: "01f745abbee546cc83f2570e2628db1a" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 34366 } } peers { permanent_uuid: "0b034930462648f9b3f72485a0131667" member_type: VOTER last_known_addr { host: "127.5.240.200" port: 52404 } } peers { permanent_uuid: "f519569b3d3b42fdb0a434061155ac38" member_type: VOTER last_known_addr { host: "127.5.240.201" port: 52406 } } peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } unsafe_config_change: true } I0703 01:40:27.699728 14722 raft_consensus.cc:2684] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [term 7 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER f519569b3d3b42fdb0a434061155ac38 (127.5.240.201) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42bc35d124544280bcfdeca6d6f6049b" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54817 } } peers { permanent_uuid: "01f745abbee546cc83f2570e2628db1a" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 34366 } } peers { permanent_uuid: "0b034930462648f9b3f72485a0131667" member_type: VOTER last_known_addr { host: "127.5.240.200" port: 52404 } } peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } unsafe_config_change: true } W0703 01:40:27.700280 14221 heartbeater.cc:587] Failed to heartbeat to 127.5.240.254:50751: Network error: Failed to ping master at 127.5.240.254:50751: Client connection negotiation failed: client connection to 127.5.240.254:50751: connect: Connection refused (error 111) I0703 01:40:27.701493 14722 raft_consensus.cc:2684] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [term 7 LEADER]: Committing config change with OpId 4.4: config changed from index -1 to 4, VOTER 01f745abbee546cc83f2570e2628db1a (127.5.240.195) evicted, VOTER f519569b3d3b42fdb0a434061155ac38 (127.5.240.201) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42bc35d124544280bcfdeca6d6f6049b" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54817 } } peers { permanent_uuid: "0b034930462648f9b3f72485a0131667" member_type: VOTER last_known_addr { host: "127.5.240.200" port: 52404 } } peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } unsafe_config_change: true } W0703 01:40:27.702044 14221 heartbeater.cc:587] Failed to heartbeat to 127.5.240.254:50751: Network error: Failed to ping master at 127.5.240.254:50751: Client connection negotiation failed: client connection to 127.5.240.254:50751: connect: Connection refused (error 111) I0703 01:40:27.703256 14722 raft_consensus.cc:2684] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [term 7 LEADER]: Committing config change with OpId 5.5: config changed from index -1 to 5, VOTER 01f745abbee546cc83f2570e2628db1a (127.5.240.195) evicted, VOTER 0b034930462648f9b3f72485a0131667 (127.5.240.200) evicted, VOTER f519569b3d3b42fdb0a434061155ac38 (127.5.240.201) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "42bc35d124544280bcfdeca6d6f6049b" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 54817 } } peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } unsafe_config_change: true } W0703 01:40:27.703899 14221 heartbeater.cc:587] Failed to heartbeat to 127.5.240.254:50751: Network error: Failed to ping master at 127.5.240.254:50751: Client connection negotiation failed: client connection to 127.5.240.254:50751: connect: Connection refused (error 111) I0703 01:40:27.705065 14722 raft_consensus.cc:2684] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [term 7 LEADER]: Committing config change with OpId 6.6: config changed from index -1 to 6, VOTER 01f745abbee546cc83f2570e2628db1a (127.5.240.195) evicted, VOTER 0b034930462648f9b3f72485a0131667 (127.5.240.200) evicted, VOTER 42bc35d124544280bcfdeca6d6f6049b (127.5.240.197) evicted, VOTER f519569b3d3b42fdb0a434061155ac38 (127.5.240.201) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } unsafe_config_change: true } W0703 01:40:27.705754 14221 heartbeater.cc:587] Failed to heartbeat to 127.5.240.254:50751: Network error: Failed to ping master at 127.5.240.254:50751: Client connection negotiation failed: client connection to 127.5.240.254:50751: connect: Connection refused (error 111) W0703 01:40:27.707466 14221 heartbeater.cc:587] Failed to heartbeat to 127.5.240.254:50751: Network error: Failed to ping master at 127.5.240.254:50751: Client connection negotiation failed: client connection to 127.5.240.254:50751: connect: Connection refused (error 111) I0703 01:40:27.790918 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:50751 --webserver_interface=127.5.240.254 --webserver_port=36210 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --rpc_reuseport=true with env {} W0703 01:40:27.802325 13733 heartbeater.cc:587] Failed to heartbeat to 127.5.240.254:50751: Network error: Failed to ping master at 127.5.240.254:50751: Client connection negotiation failed: client connection to 127.5.240.254:50751: connect: Connection refused (error 111) WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:40:27.971803 14725 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:40:27.972033 14725 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:40:27.974236 14725 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0703 01:40:27.974282 14725 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:40:27.974340 14725 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0703 01:40:27.974387 14725 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0703 01:40:27.982477 14725 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:50751 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.5.240.254 --webserver_port=36210 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:40:27.988301 14725 master_main.cc:80] Initializing master server... I0703 01:40:27.989394 14725 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 167208us I0703 01:40:27.990497 14725 fs_manager.cc:261] Metadata directory not provided I0703 01:40:27.990568 14725 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:40:27.999011 14725 fs_manager.cc:397] Time spent opening directory manager: real 0.006s user 0.001s sys 0.004s I0703 01:40:27.999888 14725 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:40:28.000313 14725 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:40:28.004549 14725 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s I0703 01:40:28.004623 14725 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal uuid: "541b5e7749d34adca822dc46c75b3e52" format_stamp: "Formatted at 2018-07-03 08:40:20 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:28.005477 14725 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:40:28.039093 14725 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:40:28.040470 14725 master_main.cc:83] Starting Master server... I0703 01:40:28.076886 14725 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.254:50751 I0703 01:40:28.077342 14725 webserver.cc:175] Starting webserver on 127.5.240.254:36210 I0703 01:40:28.077386 14725 webserver.cc:186] Document root disabled I0703 01:40:28.078634 14725 webserver.cc:313] Webserver started. Bound to: http://127.5.240.254:36210/ I0703 01:40:28.079293 14725 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb I0703 01:40:28.082638 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master as pid 14725 I0703 01:40:28.087757 14785 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 541b5e7749d34adca822dc46c75b3e52: Bootstrap starting. I0703 01:40:28.100509 14785 log.cc:525] T 00000000000000000000000000000000 P 541b5e7749d34adca822dc46c75b3e52: Log is configured to *not* fsync() on all Append() calls I0703 01:40:28.109058 14785 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 541b5e7749d34adca822dc46c75b3e52: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=1} inserts{seen=4 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:40:28.109468 14785 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 541b5e7749d34adca822dc46c75b3e52: Bootstrap complete. I0703 01:40:28.118793 14785 raft_consensus.cc:325] T 00000000000000000000000000000000 P 541b5e7749d34adca822dc46c75b3e52 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "541b5e7749d34adca822dc46c75b3e52" member_type: VOTER } I0703 01:40:28.119485 14785 raft_consensus.cc:628] T 00000000000000000000000000000000 P 541b5e7749d34adca822dc46c75b3e52 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 541b5e7749d34adca822dc46c75b3e52, State: Initialized, Role: FOLLOWER I0703 01:40:28.119988 14785 consensus_queue.cc:227] T 00000000000000000000000000000000 P 541b5e7749d34adca822dc46c75b3e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "541b5e7749d34adca822dc46c75b3e52" member_type: VOTER } I0703 01:40:28.120123 14785 raft_consensus.cc:365] T 00000000000000000000000000000000 P 541b5e7749d34adca822dc46c75b3e52 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:40:28.120211 14785 raft_consensus.cc:457] T 00000000000000000000000000000000 P 541b5e7749d34adca822dc46c75b3e52 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:40:28.120296 14785 raft_consensus.cc:2730] T 00000000000000000000000000000000 P 541b5e7749d34adca822dc46c75b3e52: Snoozing failure detection for 2.165s (starting election) I0703 01:40:28.120386 14785 raft_consensus.cc:2782] T 00000000000000000000000000000000 P 541b5e7749d34adca822dc46c75b3e52 [term 1 FOLLOWER]: Advancing to term 2 I0703 01:40:28.123867 14785 raft_consensus.cc:479] T 00000000000000000000000000000000 P 541b5e7749d34adca822dc46c75b3e52 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "541b5e7749d34adca822dc46c75b3e52" member_type: VOTER } I0703 01:40:28.124213 14785 leader_election.cc:270] T 00000000000000000000000000000000 P 541b5e7749d34adca822dc46c75b3e52 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0703 01:40:28.125500 14789 raft_consensus.cc:2730] T 00000000000000000000000000000000 P 541b5e7749d34adca822dc46c75b3e52: Snoozing failure detection for 2.096s (election complete) I0703 01:40:28.125985 14789 raft_consensus.cc:2537] T 00000000000000000000000000000000 P 541b5e7749d34adca822dc46c75b3e52 [term 2 FOLLOWER]: Leader election won for term 2 I0703 01:40:28.127732 14789 raft_consensus.cc:593] T 00000000000000000000000000000000 P 541b5e7749d34adca822dc46c75b3e52 [term 2 LEADER]: Becoming Leader. State: Replica: 541b5e7749d34adca822dc46c75b3e52, State: Running, Role: LEADER I0703 01:40:28.128155 14789 consensus_queue.cc:204] T 00000000000000000000000000000000 P 541b5e7749d34adca822dc46c75b3e52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "541b5e7749d34adca822dc46c75b3e52" member_type: VOTER } I0703 01:40:28.129940 14785 sys_catalog.cc:337] T 00000000000000000000000000000000 P 541b5e7749d34adca822dc46c75b3e52 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 2 leader_uuid: "541b5e7749d34adca822dc46c75b3e52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "541b5e7749d34adca822dc46c75b3e52" member_type: VOTER } } I0703 01:40:28.130601 14785 sys_catalog.cc:340] T 00000000000000000000000000000000 P 541b5e7749d34adca822dc46c75b3e52 [sys.catalog]: This master's current role is: LEADER I0703 01:40:28.129936 14791 sys_catalog.cc:337] T 00000000000000000000000000000000 P 541b5e7749d34adca822dc46c75b3e52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 541b5e7749d34adca822dc46c75b3e52. Latest consensus state: current_term: 2 leader_uuid: "541b5e7749d34adca822dc46c75b3e52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "541b5e7749d34adca822dc46c75b3e52" member_type: VOTER } } I0703 01:40:28.130964 14791 sys_catalog.cc:340] T 00000000000000000000000000000000 P 541b5e7749d34adca822dc46c75b3e52 [sys.catalog]: This master's current role is: LEADER I0703 01:40:28.129763 14790 sys_catalog.cc:337] T 00000000000000000000000000000000 P 541b5e7749d34adca822dc46c75b3e52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "541b5e7749d34adca822dc46c75b3e52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "541b5e7749d34adca822dc46c75b3e52" member_type: VOTER } } I0703 01:40:28.131762 14790 sys_catalog.cc:340] T 00000000000000000000000000000000 P 541b5e7749d34adca822dc46c75b3e52 [sys.catalog]: This master's current role is: LEADER I0703 01:40:28.132320 14785 sys_catalog.cc:424] T 00000000000000000000000000000000 P 541b5e7749d34adca822dc46c75b3e52 [sys.catalog]: configured and running, proceeding with master startup. I0703 01:40:28.132582 14793 catalog_manager.cc:1006] Loading table and tablet metadata into memory... I0703 01:40:28.133869 14725 master_main.cc:86] Master server successfully started. I0703 01:40:28.138006 14793 catalog_manager.cc:323] Loaded metadata for table TestTable [id=33bcc4be0f704acdab70821e538ab9f9] I0703 01:40:28.140169 14793 catalog_manager.cc:377] Loaded metadata for tablet 3ea56191fad44303b1a8ce10e51741ed (table TestTable [id=33bcc4be0f704acdab70821e538ab9f9]) I0703 01:40:28.140355 14793 catalog_manager.cc:1017] Initializing Kudu internal certificate authority... I0703 01:40:28.143841 14793 catalog_manager.cc:1026] Loading token signing keys... I0703 01:40:28.144914 14793 catalog_manager.cc:4113] T 00000000000000000000000000000000 P 541b5e7749d34adca822dc46c75b3e52: Loaded TSK: 0 I0703 01:40:28.458282 14343 heartbeater.cc:346] Connected to a master server at 127.5.240.254:50751 I0703 01:40:28.459210 14752 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "136de19b9fc544e9a3bddf205b65360d" instance_seqno: 1530607223407035) as {username='jenkins'} at 127.5.240.199:37852; Asking this server to re-register. I0703 01:40:28.459802 14343 heartbeater.cc:426] Registering TS with master... I0703 01:40:28.459925 14343 heartbeater.cc:475] Master 127.5.240.254:50751 requested a full tablet report, sending... I0703 01:40:28.460801 14752 ts_manager.cc:98] Registered new tserver with Master: 136de19b9fc544e9a3bddf205b65360d (127.5.240.199:40102) I0703 01:40:28.481937 13977 heartbeater.cc:346] Connected to a master server at 127.5.240.254:50751 I0703 01:40:28.482743 14752 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "e6994a142c374c4ba8577054aeaccbd7" instance_seqno: 1530607222431506) as {username='jenkins'} at 127.5.240.196:57473; Asking this server to re-register. I0703 01:40:28.483275 13977 heartbeater.cc:426] Registering TS with master... I0703 01:40:28.483391 13977 heartbeater.cc:475] Master 127.5.240.254:50751 requested a full tablet report, sending... I0703 01:40:28.484061 14752 ts_manager.cc:98] Registered new tserver with Master: e6994a142c374c4ba8577054aeaccbd7 (127.5.240.196:59097) I0703 01:40:28.487268 13610 heartbeater.cc:346] Connected to a master server at 127.5.240.254:50751 I0703 01:40:28.487978 14752 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "ff379419dfbc4624adc84127282b52d4" instance_seqno: 1530607221408883) as {username='jenkins'} at 127.5.240.193:39241; Asking this server to re-register. I0703 01:40:28.488471 13610 heartbeater.cc:426] Registering TS with master... I0703 01:40:28.488586 13610 heartbeater.cc:475] Master 127.5.240.254:50751 requested a full tablet report, sending... I0703 01:40:28.489225 14752 ts_manager.cc:98] Registered new tserver with Master: ff379419dfbc4624adc84127282b52d4 (127.5.240.193:49562) I0703 01:40:28.712177 14221 heartbeater.cc:346] Connected to a master server at 127.5.240.254:50751 I0703 01:40:28.713307 14752 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "48755546d06649e78728d26d3b2bd82d" instance_seqno: 1530607223065179) as {username='jenkins'} at 127.5.240.198:58666; Asking this server to re-register. I0703 01:40:28.713841 14221 heartbeater.cc:426] Registering TS with master... I0703 01:40:28.713956 14221 heartbeater.cc:475] Master 127.5.240.254:50751 requested a full tablet report, sending... I0703 01:40:28.714749 14752 ts_manager.cc:98] Registered new tserver with Master: 48755546d06649e78728d26d3b2bd82d (127.5.240.198:34012) I0703 01:40:28.715179 14752 catalog_manager.cc:3798] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d reported cstate change: config changed from index -1 to 6, term changed from 1 to 7, VOTER 01f745abbee546cc83f2570e2628db1a (127.5.240.195) evicted, VOTER 0b034930462648f9b3f72485a0131667 (127.5.240.200) evicted, VOTER 42bc35d124544280bcfdeca6d6f6049b (127.5.240.197) evicted, VOTER f519569b3d3b42fdb0a434061155ac38 (127.5.240.201) evicted. New cstate: current_term: 7 leader_uuid: "48755546d06649e78728d26d3b2bd82d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } health_report { overall_health: HEALTHY } } unsafe_config_change: true } I0703 01:40:28.722308 14752 catalog_manager.cc:2903] Scheduling retry of 3ea56191fad44303b1a8ce10e51741ed Delete Tablet RPC for TS=42bc35d124544280bcfdeca6d6f6049b with a delay of 26 ms (attempt = 1) W0703 01:40:28.722429 14752 catalog_manager.cc:3957] Failed to send 3ea56191fad44303b1a8ce10e51741ed Delete Tablet RPC for TS=42bc35d124544280bcfdeca6d6f6049b: Not found: failed to reset TS proxy: Could not find TS for UUID 42bc35d124544280bcfdeca6d6f6049b I0703 01:40:28.722555 14752 catalog_manager.cc:2903] Scheduling retry of 3ea56191fad44303b1a8ce10e51741ed Delete Tablet RPC for TS=01f745abbee546cc83f2570e2628db1a with a delay of 21 ms (attempt = 1) W0703 01:40:28.722647 14752 catalog_manager.cc:3957] Failed to send 3ea56191fad44303b1a8ce10e51741ed Delete Tablet RPC for TS=01f745abbee546cc83f2570e2628db1a: Not found: failed to reset TS proxy: Could not find TS for UUID 01f745abbee546cc83f2570e2628db1a I0703 01:40:28.722736 14752 catalog_manager.cc:2903] Scheduling retry of 3ea56191fad44303b1a8ce10e51741ed Delete Tablet RPC for TS=0b034930462648f9b3f72485a0131667 with a delay of 34 ms (attempt = 1) W0703 01:40:28.722839 14752 catalog_manager.cc:3957] Failed to send 3ea56191fad44303b1a8ce10e51741ed Delete Tablet RPC for TS=0b034930462648f9b3f72485a0131667: Not found: failed to reset TS proxy: Could not find TS for UUID 0b034930462648f9b3f72485a0131667 I0703 01:40:28.722914 14752 catalog_manager.cc:2903] Scheduling retry of 3ea56191fad44303b1a8ce10e51741ed Delete Tablet RPC for TS=f519569b3d3b42fdb0a434061155ac38 with a delay of 58 ms (attempt = 1) W0703 01:40:28.722995 14752 catalog_manager.cc:3957] Failed to send 3ea56191fad44303b1a8ce10e51741ed Delete Tablet RPC for TS=f519569b3d3b42fdb0a434061155ac38: Not found: failed to reset TS proxy: Could not find TS for UUID f519569b3d3b42fdb0a434061155ac38 I0703 01:40:28.723348 14752 catalog_manager.cc:3437] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 3ea56191fad44303b1a8ce10e51741ed (attempt 1) I0703 01:40:28.728809 14178 consensus_queue.cc:204] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 0, Current term: 7, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } peers { permanent_uuid: "e6994a142c374c4ba8577054aeaccbd7" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 59097 } attrs { promote: true } } unsafe_config_change: true I0703 01:40:28.730690 14810 raft_consensus.cc:2684] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [term 7 LEADER]: Committing config change with OpId 7.8: config changed from index 6 to 8, NON_VOTER e6994a142c374c4ba8577054aeaccbd7 (127.5.240.196) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } peers { permanent_uuid: "e6994a142c374c4ba8577054aeaccbd7" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 59097 } attrs { promote: true } } unsafe_config_change: true } I0703 01:40:28.733976 14734 catalog_manager.cc:3352] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3ea56191fad44303b1a8ce10e51741ed with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0703 01:40:28.734699 14752 catalog_manager.cc:3798] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d reported cstate change: config changed from index 6 to 8, NON_VOTER e6994a142c374c4ba8577054aeaccbd7 (127.5.240.196) added. New cstate: current_term: 7 leader_uuid: "48755546d06649e78728d26d3b2bd82d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6994a142c374c4ba8577054aeaccbd7" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 59097 } attrs { promote: true } health_report { overall_health: UNKNOWN } } unsafe_config_change: true } I0703 01:40:28.736753 14752 catalog_manager.cc:3437] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 3ea56191fad44303b1a8ce10e51741ed (attempt 1) I0703 01:40:28.737803 14178 consensus_peers.cc:455] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d -> Peer e6994a142c374c4ba8577054aeaccbd7 (127.5.240.196:59097): Closing peer: e6994a142c374c4ba8577054aeaccbd7 I0703 01:40:28.737962 14178 consensus_queue.cc:204] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 0, Current term: 7, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } peers { permanent_uuid: "e6994a142c374c4ba8577054aeaccbd7" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 59097 } attrs { promote: true } } peers { permanent_uuid: "ff379419dfbc4624adc84127282b52d4" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 49562 } attrs { promote: true } } unsafe_config_change: true I0703 01:40:28.739478 14810 raft_consensus.cc:2684] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [term 7 LEADER]: Committing config change with OpId 7.9: config changed from index 8 to 9, NON_VOTER ff379419dfbc4624adc84127282b52d4 (127.5.240.193) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } peers { permanent_uuid: "e6994a142c374c4ba8577054aeaccbd7" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 59097 } attrs { promote: true } } peers { permanent_uuid: "ff379419dfbc4624adc84127282b52d4" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 49562 } attrs { promote: true } } unsafe_config_change: true } W0703 01:40:28.740443 14111 consensus_peers.cc:433] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d -> Peer e6994a142c374c4ba8577054aeaccbd7 (127.5.240.196:59097): Couldn't send request to peer e6994a142c374c4ba8577054aeaccbd7 for tablet 3ea56191fad44303b1a8ce10e51741ed. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3ea56191fad44303b1a8ce10e51741ed. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:40:28.742810 14734 catalog_manager.cc:3352] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3ea56191fad44303b1a8ce10e51741ed with cas_config_opid_index 8: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0703 01:40:28.743751 14752 catalog_manager.cc:3798] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d reported cstate change: config changed from index 8 to 9, NON_VOTER ff379419dfbc4624adc84127282b52d4 (127.5.240.193) added. New cstate: current_term: 7 leader_uuid: "48755546d06649e78728d26d3b2bd82d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6994a142c374c4ba8577054aeaccbd7" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 59097 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff379419dfbc4624adc84127282b52d4" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 49562 } attrs { promote: true } health_report { overall_health: UNKNOWN } } unsafe_config_change: true } I0703 01:40:28.744261 14735 catalog_manager.cc:2903] Scheduling retry of 3ea56191fad44303b1a8ce10e51741ed Delete Tablet RPC for TS=01f745abbee546cc83f2570e2628db1a with a delay of 78 ms (attempt = 2) W0703 01:40:28.744380 14735 catalog_manager.cc:2922] Async tablet task 3ea56191fad44303b1a8ce10e51741ed Delete Tablet RPC for TS=01f745abbee546cc83f2570e2628db1a failed: Not found: failed to reset TS proxy: Could not find TS for UUID 01f745abbee546cc83f2570e2628db1a I0703 01:40:28.745810 14752 catalog_manager.cc:3437] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 3ea56191fad44303b1a8ce10e51741ed (attempt 1) I0703 01:40:28.746562 14178 consensus_peers.cc:455] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d -> Peer ff379419dfbc4624adc84127282b52d4 (127.5.240.193:49562): Closing peer: ff379419dfbc4624adc84127282b52d4 I0703 01:40:28.746626 14178 consensus_peers.cc:455] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d -> Peer e6994a142c374c4ba8577054aeaccbd7 (127.5.240.196:59097): Closing peer: e6994a142c374c4ba8577054aeaccbd7 I0703 01:40:28.746861 14178 consensus_queue.cc:204] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 0, Current term: 7, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } peers { permanent_uuid: "e6994a142c374c4ba8577054aeaccbd7" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 59097 } attrs { promote: true } } peers { permanent_uuid: "ff379419dfbc4624adc84127282b52d4" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 49562 } attrs { promote: true } } peers { permanent_uuid: "136de19b9fc544e9a3bddf205b65360d" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 40102 } attrs { promote: true } } unsafe_config_change: true I0703 01:40:28.748474 14733 catalog_manager.cc:2903] Scheduling retry of 3ea56191fad44303b1a8ce10e51741ed Delete Tablet RPC for TS=42bc35d124544280bcfdeca6d6f6049b with a delay of 61 ms (attempt = 2) W0703 01:40:28.748541 14733 catalog_manager.cc:2922] Async tablet task 3ea56191fad44303b1a8ce10e51741ed Delete Tablet RPC for TS=42bc35d124544280bcfdeca6d6f6049b failed: Not found: failed to reset TS proxy: Could not find TS for UUID 42bc35d124544280bcfdeca6d6f6049b I0703 01:40:28.748427 14811 raft_consensus.cc:2684] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [term 7 LEADER]: Committing config change with OpId 7.10: config changed from index 9 to 10, NON_VOTER 136de19b9fc544e9a3bddf205b65360d (127.5.240.199) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } peers { permanent_uuid: "e6994a142c374c4ba8577054aeaccbd7" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 59097 } attrs { promote: true } } peers { permanent_uuid: "ff379419dfbc4624adc84127282b52d4" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 49562 } attrs { promote: true } } peers { permanent_uuid: "136de19b9fc544e9a3bddf205b65360d" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 40102 } attrs { promote: true } } unsafe_config_change: true } W0703 01:40:28.749457 14111 consensus_peers.cc:433] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d -> Peer e6994a142c374c4ba8577054aeaccbd7 (127.5.240.196:59097): Couldn't send request to peer e6994a142c374c4ba8577054aeaccbd7 for tablet 3ea56191fad44303b1a8ce10e51741ed. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3ea56191fad44303b1a8ce10e51741ed. Retrying in the next heartbeat period. Already tried 1 times. W0703 01:40:28.752754 14112 consensus_peers.cc:433] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d -> Peer ff379419dfbc4624adc84127282b52d4 (127.5.240.193:49562): Couldn't send request to peer ff379419dfbc4624adc84127282b52d4 for tablet 3ea56191fad44303b1a8ce10e51741ed. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3ea56191fad44303b1a8ce10e51741ed. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:40:28.752936 14734 catalog_manager.cc:3352] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3ea56191fad44303b1a8ce10e51741ed with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1) I0703 01:40:28.754060 14752 catalog_manager.cc:3798] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d reported cstate change: config changed from index 9 to 10, NON_VOTER 136de19b9fc544e9a3bddf205b65360d (127.5.240.199) added. New cstate: current_term: 7 leader_uuid: "48755546d06649e78728d26d3b2bd82d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6994a142c374c4ba8577054aeaccbd7" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 59097 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff379419dfbc4624adc84127282b52d4" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 49562 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "136de19b9fc544e9a3bddf205b65360d" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 40102 } attrs { promote: true } health_report { overall_health: UNKNOWN } } unsafe_config_change: true } I0703 01:40:28.755810 14752 catalog_manager.cc:3437] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 3ea56191fad44303b1a8ce10e51741ed (attempt 1) W0703 01:40:28.756014 14752 catalog_manager.cc:3486] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3ea56191fad44303b1a8ce10e51741ed with cas_config_opid_index 10: no candidate replacement replica found for tablet 3ea56191fad44303b1a8ce10e51741ed (table TestTable [id=33bcc4be0f704acdab70821e538ab9f9]); the total number of registered tablet servers (4) does not allow for replacement of the failed replica: at least 6 tablet servers are required I0703 01:40:28.756069 14752 catalog_manager.cc:2903] Scheduling retry of ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3ea56191fad44303b1a8ce10e51741ed with cas_config_opid_index 10 with a delay of 37 ms (attempt = 1) I0703 01:40:28.757547 14736 catalog_manager.cc:2903] Scheduling retry of 3ea56191fad44303b1a8ce10e51741ed Delete Tablet RPC for TS=0b034930462648f9b3f72485a0131667 with a delay of 66 ms (attempt = 2) W0703 01:40:28.757647 14736 catalog_manager.cc:2922] Async tablet task 3ea56191fad44303b1a8ce10e51741ed Delete Tablet RPC for TS=0b034930462648f9b3f72485a0131667 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 0b034930462648f9b3f72485a0131667 W0703 01:40:28.757786 14110 consensus_peers.cc:433] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d -> Peer 136de19b9fc544e9a3bddf205b65360d (127.5.240.199:40102): Couldn't send request to peer 136de19b9fc544e9a3bddf205b65360d for tablet 3ea56191fad44303b1a8ce10e51741ed. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3ea56191fad44303b1a8ce10e51741ed. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:40:28.781746 14733 catalog_manager.cc:2903] Scheduling retry of 3ea56191fad44303b1a8ce10e51741ed Delete Tablet RPC for TS=f519569b3d3b42fdb0a434061155ac38 with a delay of 54 ms (attempt = 2) W0703 01:40:28.781800 14733 catalog_manager.cc:2922] Async tablet task 3ea56191fad44303b1a8ce10e51741ed Delete Tablet RPC for TS=f519569b3d3b42fdb0a434061155ac38 failed: Not found: failed to reset TS proxy: Could not find TS for UUID f519569b3d3b42fdb0a434061155ac38 I0703 01:40:28.793488 14735 catalog_manager.cc:3437] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 3ea56191fad44303b1a8ce10e51741ed (attempt 2) I0703 01:40:28.793634 14735 catalog_manager.cc:2903] Scheduling retry of ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3ea56191fad44303b1a8ce10e51741ed with cas_config_opid_index 10 with a delay of 59 ms (attempt = 2) I0703 01:40:28.807149 13733 heartbeater.cc:346] Connected to a master server at 127.5.240.254:50751 I0703 01:40:28.807885 14752 master_service.cc:205] Got heartbeat from unknown tserver (permanent_uuid: "586291a7317b401c9f2f4bd48fe107e2" instance_seqno: 1530607221755569) as {username='jenkins'} at 127.5.240.194:41935; Asking this server to re-register. I0703 01:40:28.808406 13733 heartbeater.cc:426] Registering TS with master... I0703 01:40:28.808519 13733 heartbeater.cc:475] Master 127.5.240.254:50751 requested a full tablet report, sending... I0703 01:40:28.809176 14752 ts_manager.cc:98] Registered new tserver with Master: 586291a7317b401c9f2f4bd48fe107e2 (127.5.240.194:53793) I0703 01:40:28.810174 14733 catalog_manager.cc:2903] Scheduling retry of 3ea56191fad44303b1a8ce10e51741ed Delete Tablet RPC for TS=42bc35d124544280bcfdeca6d6f6049b with a delay of 98 ms (attempt = 3) W0703 01:40:28.810227 14733 catalog_manager.cc:2922] Async tablet task 3ea56191fad44303b1a8ce10e51741ed Delete Tablet RPC for TS=42bc35d124544280bcfdeca6d6f6049b failed: Not found: failed to reset TS proxy: Could not find TS for UUID 42bc35d124544280bcfdeca6d6f6049b I0703 01:40:28.823165 14735 catalog_manager.cc:2903] Scheduling retry of 3ea56191fad44303b1a8ce10e51741ed Delete Tablet RPC for TS=01f745abbee546cc83f2570e2628db1a with a delay of 104 ms (attempt = 3) W0703 01:40:28.823218 14735 catalog_manager.cc:2922] Async tablet task 3ea56191fad44303b1a8ce10e51741ed Delete Tablet RPC for TS=01f745abbee546cc83f2570e2628db1a failed: Not found: failed to reset TS proxy: Could not find TS for UUID 01f745abbee546cc83f2570e2628db1a I0703 01:40:28.824309 14736 catalog_manager.cc:2903] Scheduling retry of 3ea56191fad44303b1a8ce10e51741ed Delete Tablet RPC for TS=0b034930462648f9b3f72485a0131667 with a delay of 97 ms (attempt = 3) W0703 01:40:28.824362 14736 catalog_manager.cc:2922] Async tablet task 3ea56191fad44303b1a8ce10e51741ed Delete Tablet RPC for TS=0b034930462648f9b3f72485a0131667 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 0b034930462648f9b3f72485a0131667 I0703 01:40:28.836846 14733 catalog_manager.cc:2903] Scheduling retry of 3ea56191fad44303b1a8ce10e51741ed Delete Tablet RPC for TS=f519569b3d3b42fdb0a434061155ac38 with a delay of 86 ms (attempt = 3) W0703 01:40:28.836895 14733 catalog_manager.cc:2922] Async tablet task 3ea56191fad44303b1a8ce10e51741ed Delete Tablet RPC for TS=f519569b3d3b42fdb0a434061155ac38 failed: Not found: failed to reset TS proxy: Could not find TS for UUID f519569b3d3b42fdb0a434061155ac38 I0703 01:40:28.853585 14735 catalog_manager.cc:3437] Sending ChangeConfig:ADD_PEER:NON_VOTER on tablet 3ea56191fad44303b1a8ce10e51741ed (attempt 3) I0703 01:40:28.854378 14178 consensus_peers.cc:455] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d -> Peer ff379419dfbc4624adc84127282b52d4 (127.5.240.193:49562): Closing peer: ff379419dfbc4624adc84127282b52d4 I0703 01:40:28.854449 14178 consensus_peers.cc:455] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d -> Peer 136de19b9fc544e9a3bddf205b65360d (127.5.240.199:40102): Closing peer: 136de19b9fc544e9a3bddf205b65360d I0703 01:40:28.854528 14178 consensus_peers.cc:455] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d -> Peer e6994a142c374c4ba8577054aeaccbd7 (127.5.240.196:59097): Closing peer: e6994a142c374c4ba8577054aeaccbd7 I0703 01:40:28.854755 14178 consensus_queue.cc:204] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 7.10, Last appended by leader: 0, Current term: 7, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } peers { permanent_uuid: "e6994a142c374c4ba8577054aeaccbd7" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 59097 } attrs { promote: true } } peers { permanent_uuid: "ff379419dfbc4624adc84127282b52d4" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 49562 } attrs { promote: true } } peers { permanent_uuid: "136de19b9fc544e9a3bddf205b65360d" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 40102 } attrs { promote: true } } peers { permanent_uuid: "586291a7317b401c9f2f4bd48fe107e2" member_type: NON_VOTER last_known_addr { host: "127.5.240.194" port: 53793 } attrs { promote: true } } unsafe_config_change: true W0703 01:40:28.858628 14111 consensus_peers.cc:433] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d -> Peer e6994a142c374c4ba8577054aeaccbd7 (127.5.240.196:59097): Couldn't send request to peer e6994a142c374c4ba8577054aeaccbd7 for tablet 3ea56191fad44303b1a8ce10e51741ed. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3ea56191fad44303b1a8ce10e51741ed. Retrying in the next heartbeat period. Already tried 1 times. W0703 01:40:28.858887 14112 consensus_peers.cc:433] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d -> Peer ff379419dfbc4624adc84127282b52d4 (127.5.240.193:49562): Couldn't send request to peer ff379419dfbc4624adc84127282b52d4 for tablet 3ea56191fad44303b1a8ce10e51741ed. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3ea56191fad44303b1a8ce10e51741ed. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:40:28.858288 14811 raft_consensus.cc:2684] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [term 7 LEADER]: Committing config change with OpId 7.11: config changed from index 10 to 11, NON_VOTER 586291a7317b401c9f2f4bd48fe107e2 (127.5.240.194) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } peers { permanent_uuid: "e6994a142c374c4ba8577054aeaccbd7" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 59097 } attrs { promote: true } } peers { permanent_uuid: "ff379419dfbc4624adc84127282b52d4" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 49562 } attrs { promote: true } } peers { permanent_uuid: "136de19b9fc544e9a3bddf205b65360d" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 40102 } attrs { promote: true } } peers { permanent_uuid: "586291a7317b401c9f2f4bd48fe107e2" member_type: NON_VOTER last_known_addr { host: "127.5.240.194" port: 53793 } attrs { promote: true } } unsafe_config_change: true } W0703 01:40:28.859248 14110 consensus_peers.cc:433] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d -> Peer 136de19b9fc544e9a3bddf205b65360d (127.5.240.199:40102): Couldn't send request to peer 136de19b9fc544e9a3bddf205b65360d for tablet 3ea56191fad44303b1a8ce10e51741ed. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3ea56191fad44303b1a8ce10e51741ed. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:40:28.862325 14734 catalog_manager.cc:3352] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3ea56191fad44303b1a8ce10e51741ed with cas_config_opid_index 10: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 3) I0703 01:40:28.863554 14752 catalog_manager.cc:3798] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d reported cstate change: config changed from index 10 to 11, NON_VOTER 586291a7317b401c9f2f4bd48fe107e2 (127.5.240.194) added. New cstate: current_term: 7 leader_uuid: "48755546d06649e78728d26d3b2bd82d" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6994a142c374c4ba8577054aeaccbd7" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 59097 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff379419dfbc4624adc84127282b52d4" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 49562 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "136de19b9fc544e9a3bddf205b65360d" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 40102 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "586291a7317b401c9f2f4bd48fe107e2" member_type: NON_VOTER last_known_addr { host: "127.5.240.194" port: 53793 } attrs { promote: true } health_report { overall_health: UNKNOWN } } unsafe_config_change: true } W0703 01:40:28.865958 14112 consensus_peers.cc:433] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d -> Peer 586291a7317b401c9f2f4bd48fe107e2 (127.5.240.194:53793): Couldn't send request to peer 586291a7317b401c9f2f4bd48fe107e2 for tablet 3ea56191fad44303b1a8ce10e51741ed. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3ea56191fad44303b1a8ce10e51741ed. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:40:28.909235 14733 catalog_manager.cc:2903] Scheduling retry of 3ea56191fad44303b1a8ce10e51741ed Delete Tablet RPC for TS=42bc35d124544280bcfdeca6d6f6049b with a delay of 176 ms (attempt = 4) W0703 01:40:28.909283 14733 catalog_manager.cc:2922] Async tablet task 3ea56191fad44303b1a8ce10e51741ed Delete Tablet RPC for TS=42bc35d124544280bcfdeca6d6f6049b failed: Not found: failed to reset TS proxy: Could not find TS for UUID 42bc35d124544280bcfdeca6d6f6049b I0703 01:40:28.922034 14736 catalog_manager.cc:2903] Scheduling retry of 3ea56191fad44303b1a8ce10e51741ed Delete Tablet RPC for TS=0b034930462648f9b3f72485a0131667 with a delay of 158 ms (attempt = 4) W0703 01:40:28.922087 14736 catalog_manager.cc:2922] Async tablet task 3ea56191fad44303b1a8ce10e51741ed Delete Tablet RPC for TS=0b034930462648f9b3f72485a0131667 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 0b034930462648f9b3f72485a0131667 I0703 01:40:28.923918 14733 catalog_manager.cc:2903] Scheduling retry of 3ea56191fad44303b1a8ce10e51741ed Delete Tablet RPC for TS=f519569b3d3b42fdb0a434061155ac38 with a delay of 141 ms (attempt = 4) W0703 01:40:28.923974 14733 catalog_manager.cc:2922] Async tablet task 3ea56191fad44303b1a8ce10e51741ed Delete Tablet RPC for TS=f519569b3d3b42fdb0a434061155ac38 failed: Not found: failed to reset TS proxy: Could not find TS for UUID f519569b3d3b42fdb0a434061155ac38 I0703 01:40:28.928335 14735 catalog_manager.cc:2903] Scheduling retry of 3ea56191fad44303b1a8ce10e51741ed Delete Tablet RPC for TS=01f745abbee546cc83f2570e2628db1a with a delay of 175 ms (attempt = 4) W0703 01:40:28.928391 14735 catalog_manager.cc:2922] Async tablet task 3ea56191fad44303b1a8ce10e51741ed Delete Tablet RPC for TS=01f745abbee546cc83f2570e2628db1a failed: Not found: failed to reset TS proxy: Could not find TS for UUID 01f745abbee546cc83f2570e2628db1a I0703 01:40:29.065542 14733 catalog_manager.cc:2903] Scheduling retry of 3ea56191fad44303b1a8ce10e51741ed Delete Tablet RPC for TS=f519569b3d3b42fdb0a434061155ac38 with a delay of 256 ms (attempt = 5) W0703 01:40:29.065618 14733 catalog_manager.cc:2922] Async tablet task 3ea56191fad44303b1a8ce10e51741ed Delete Tablet RPC for TS=f519569b3d3b42fdb0a434061155ac38 failed: Not found: failed to reset TS proxy: Could not find TS for UUID f519569b3d3b42fdb0a434061155ac38 I0703 01:40:29.080662 14736 catalog_manager.cc:2903] Scheduling retry of 3ea56191fad44303b1a8ce10e51741ed Delete Tablet RPC for TS=0b034930462648f9b3f72485a0131667 with a delay of 286 ms (attempt = 5) W0703 01:40:29.080713 14736 catalog_manager.cc:2922] Async tablet task 3ea56191fad44303b1a8ce10e51741ed Delete Tablet RPC for TS=0b034930462648f9b3f72485a0131667 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 0b034930462648f9b3f72485a0131667 I0703 01:40:29.085983 14733 catalog_manager.cc:2903] Scheduling retry of 3ea56191fad44303b1a8ce10e51741ed Delete Tablet RPC for TS=42bc35d124544280bcfdeca6d6f6049b with a delay of 279 ms (attempt = 5) W0703 01:40:29.086038 14733 catalog_manager.cc:2922] Async tablet task 3ea56191fad44303b1a8ce10e51741ed Delete Tablet RPC for TS=42bc35d124544280bcfdeca6d6f6049b failed: Not found: failed to reset TS proxy: Could not find TS for UUID 42bc35d124544280bcfdeca6d6f6049b I0703 01:40:29.104003 14735 catalog_manager.cc:2903] Scheduling retry of 3ea56191fad44303b1a8ce10e51741ed Delete Tablet RPC for TS=01f745abbee546cc83f2570e2628db1a with a delay of 270 ms (attempt = 5) W0703 01:40:29.104063 14735 catalog_manager.cc:2922] Async tablet task 3ea56191fad44303b1a8ce10e51741ed Delete Tablet RPC for TS=01f745abbee546cc83f2570e2628db1a failed: Not found: failed to reset TS proxy: Could not find TS for UUID 01f745abbee546cc83f2570e2628db1a I0703 01:40:29.308696 14820 ts_tablet_manager.cc:669] T 3ea56191fad44303b1a8ce10e51741ed P 586291a7317b401c9f2f4bd48fe107e2: Initiating tablet copy from peer 48755546d06649e78728d26d3b2bd82d (127.5.240.198:34012) I0703 01:40:29.308956 14820 tablet_copy_client.cc:239] T 3ea56191fad44303b1a8ce10e51741ed P 586291a7317b401c9f2f4bd48fe107e2: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.198:34012 I0703 01:40:29.314440 14198 tablet_copy_service.cc:135] P 48755546d06649e78728d26d3b2bd82d: Received BeginTabletCopySession request for tablet 3ea56191fad44303b1a8ce10e51741ed from peer 586291a7317b401c9f2f4bd48fe107e2 ({username='jenkins'} at 127.5.240.194:51732) I0703 01:40:29.314574 14198 tablet_copy_service.cc:156] P 48755546d06649e78728d26d3b2bd82d: Beginning new tablet copy session on tablet 3ea56191fad44303b1a8ce10e51741ed from peer 586291a7317b401c9f2f4bd48fe107e2 at {username='jenkins'} at 127.5.240.194:51732: session id = 586291a7317b401c9f2f4bd48fe107e2-3ea56191fad44303b1a8ce10e51741ed I0703 01:40:29.316876 14198 tablet_copy_source_session.cc:202] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:40:29.320765 14820 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 3ea56191fad44303b1a8ce10e51741ed. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:40:29.322345 14733 catalog_manager.cc:2903] Scheduling retry of 3ea56191fad44303b1a8ce10e51741ed Delete Tablet RPC for TS=f519569b3d3b42fdb0a434061155ac38 with a delay of 521 ms (attempt = 6) W0703 01:40:29.322415 14733 catalog_manager.cc:2922] Async tablet task 3ea56191fad44303b1a8ce10e51741ed Delete Tablet RPC for TS=f519569b3d3b42fdb0a434061155ac38 failed: Not found: failed to reset TS proxy: Could not find TS for UUID f519569b3d3b42fdb0a434061155ac38 I0703 01:40:29.330600 14820 ts_tablet_manager.cc:1173] T 3ea56191fad44303b1a8ce10e51741ed P 586291a7317b401c9f2f4bd48fe107e2: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:40:29.330863 14820 tablet_copy_client.cc:566] T 3ea56191fad44303b1a8ce10e51741ed P 586291a7317b401c9f2f4bd48fe107e2: tablet copy: Starting download of 0 data blocks... I0703 01:40:29.331084 14820 tablet_copy_client.cc:529] T 3ea56191fad44303b1a8ce10e51741ed P 586291a7317b401c9f2f4bd48fe107e2: tablet copy: Starting download of 1 WAL segments... I0703 01:40:29.334393 14820 tablet_copy_client.cc:405] T 3ea56191fad44303b1a8ce10e51741ed P 586291a7317b401c9f2f4bd48fe107e2: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:40:29.336089 14820 ts_tablet_manager.cc:1028] T 3ea56191fad44303b1a8ce10e51741ed P 586291a7317b401c9f2f4bd48fe107e2: Bootstrapping tablet I0703 01:40:29.336302 14820 tablet_bootstrap.cc:427] T 3ea56191fad44303b1a8ce10e51741ed P 586291a7317b401c9f2f4bd48fe107e2: Bootstrap starting. I0703 01:40:29.341234 14820 log.cc:525] T 3ea56191fad44303b1a8ce10e51741ed P 586291a7317b401c9f2f4bd48fe107e2: Log is configured to *not* fsync() on all Append() calls I0703 01:40:29.346434 14824 ts_tablet_manager.cc:669] T 3ea56191fad44303b1a8ce10e51741ed P 136de19b9fc544e9a3bddf205b65360d: Initiating tablet copy from peer 48755546d06649e78728d26d3b2bd82d (127.5.240.198:34012) I0703 01:40:29.346694 14824 tablet_copy_client.cc:239] T 3ea56191fad44303b1a8ce10e51741ed P 136de19b9fc544e9a3bddf205b65360d: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.198:34012 I0703 01:40:29.347502 14820 tablet_bootstrap.cc:427] T 3ea56191fad44303b1a8ce10e51741ed P 586291a7317b401c9f2f4bd48fe107e2: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:40:29.347959 14820 tablet_bootstrap.cc:427] T 3ea56191fad44303b1a8ce10e51741ed P 586291a7317b401c9f2f4bd48fe107e2: Bootstrap complete. I0703 01:40:29.348253 14820 ts_tablet_manager.cc:1045] T 3ea56191fad44303b1a8ce10e51741ed P 586291a7317b401c9f2f4bd48fe107e2: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.006s I0703 01:40:29.352128 14198 tablet_copy_service.cc:135] P 48755546d06649e78728d26d3b2bd82d: Received BeginTabletCopySession request for tablet 3ea56191fad44303b1a8ce10e51741ed from peer 136de19b9fc544e9a3bddf205b65360d ({username='jenkins'} at 127.5.240.199:43474) I0703 01:40:29.352246 14198 tablet_copy_service.cc:156] P 48755546d06649e78728d26d3b2bd82d: Beginning new tablet copy session on tablet 3ea56191fad44303b1a8ce10e51741ed from peer 136de19b9fc544e9a3bddf205b65360d at {username='jenkins'} at 127.5.240.199:43474: session id = 136de19b9fc544e9a3bddf205b65360d-3ea56191fad44303b1a8ce10e51741ed I0703 01:40:29.354668 14198 tablet_copy_source_session.cc:202] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:40:29.354902 14820 raft_consensus.cc:325] T 3ea56191fad44303b1a8ce10e51741ed P 586291a7317b401c9f2f4bd48fe107e2 [term 7 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } peers { permanent_uuid: "e6994a142c374c4ba8577054aeaccbd7" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 59097 } attrs { promote: true } } peers { permanent_uuid: "ff379419dfbc4624adc84127282b52d4" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 49562 } attrs { promote: true } } peers { permanent_uuid: "136de19b9fc544e9a3bddf205b65360d" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 40102 } attrs { promote: true } } peers { permanent_uuid: "586291a7317b401c9f2f4bd48fe107e2" member_type: NON_VOTER last_known_addr { host: "127.5.240.194" port: 53793 } attrs { promote: true } } unsafe_config_change: true I0703 01:40:29.355249 14820 raft_consensus.cc:628] T 3ea56191fad44303b1a8ce10e51741ed P 586291a7317b401c9f2f4bd48fe107e2 [term 7 LEARNER]: Becoming Follower/Learner. State: Replica: 586291a7317b401c9f2f4bd48fe107e2, State: Initialized, Role: LEARNER I0703 01:40:29.355518 14820 consensus_queue.cc:227] T 3ea56191fad44303b1a8ce10e51741ed P 586291a7317b401c9f2f4bd48fe107e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 7.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } peers { permanent_uuid: "e6994a142c374c4ba8577054aeaccbd7" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 59097 } attrs { promote: true } } peers { permanent_uuid: "ff379419dfbc4624adc84127282b52d4" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 49562 } attrs { promote: true } } peers { permanent_uuid: "136de19b9fc544e9a3bddf205b65360d" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 40102 } attrs { promote: true } } peers { permanent_uuid: "586291a7317b401c9f2f4bd48fe107e2" member_type: NON_VOTER last_known_addr { host: "127.5.240.194" port: 53793 } attrs { promote: true } } unsafe_config_change: true I0703 01:40:29.357544 14824 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 3ea56191fad44303b1a8ce10e51741ed. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:40:29.357947 14820 ts_tablet_manager.cc:1073] T 3ea56191fad44303b1a8ce10e51741ed P 586291a7317b401c9f2f4bd48fe107e2: Time spent starting tablet: real 0.010s user 0.008s sys 0.001s I0703 01:40:29.358613 14198 tablet_copy_service.cc:325] P 48755546d06649e78728d26d3b2bd82d: Request end of tablet copy session 586291a7317b401c9f2f4bd48fe107e2-3ea56191fad44303b1a8ce10e51741ed received from {username='jenkins'} at 127.5.240.194:51732 I0703 01:40:29.358680 14198 tablet_copy_service.cc:413] P 48755546d06649e78728d26d3b2bd82d: ending tablet copy session 586291a7317b401c9f2f4bd48fe107e2-3ea56191fad44303b1a8ce10e51741ed on tablet 3ea56191fad44303b1a8ce10e51741ed with peer 586291a7317b401c9f2f4bd48fe107e2 I0703 01:40:29.365818 14733 catalog_manager.cc:2903] Scheduling retry of 3ea56191fad44303b1a8ce10e51741ed Delete Tablet RPC for TS=42bc35d124544280bcfdeca6d6f6049b with a delay of 534 ms (attempt = 6) W0703 01:40:29.365913 14733 catalog_manager.cc:2922] Async tablet task 3ea56191fad44303b1a8ce10e51741ed Delete Tablet RPC for TS=42bc35d124544280bcfdeca6d6f6049b failed: Not found: failed to reset TS proxy: Could not find TS for UUID 42bc35d124544280bcfdeca6d6f6049b I0703 01:40:29.367895 14736 catalog_manager.cc:2903] Scheduling retry of 3ea56191fad44303b1a8ce10e51741ed Delete Tablet RPC for TS=0b034930462648f9b3f72485a0131667 with a delay of 559 ms (attempt = 6) W0703 01:40:29.367959 14736 catalog_manager.cc:2922] Async tablet task 3ea56191fad44303b1a8ce10e51741ed Delete Tablet RPC for TS=0b034930462648f9b3f72485a0131667 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 0b034930462648f9b3f72485a0131667 I0703 01:40:29.368840 14824 ts_tablet_manager.cc:1173] T 3ea56191fad44303b1a8ce10e51741ed P 136de19b9fc544e9a3bddf205b65360d: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:40:29.369102 14824 tablet_copy_client.cc:566] T 3ea56191fad44303b1a8ce10e51741ed P 136de19b9fc544e9a3bddf205b65360d: tablet copy: Starting download of 0 data blocks... I0703 01:40:29.369312 14824 tablet_copy_client.cc:529] T 3ea56191fad44303b1a8ce10e51741ed P 136de19b9fc544e9a3bddf205b65360d: tablet copy: Starting download of 1 WAL segments... I0703 01:40:29.371595 14824 tablet_copy_client.cc:405] T 3ea56191fad44303b1a8ce10e51741ed P 136de19b9fc544e9a3bddf205b65360d: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:40:29.373212 14824 ts_tablet_manager.cc:1028] T 3ea56191fad44303b1a8ce10e51741ed P 136de19b9fc544e9a3bddf205b65360d: Bootstrapping tablet I0703 01:40:29.373419 14824 tablet_bootstrap.cc:427] T 3ea56191fad44303b1a8ce10e51741ed P 136de19b9fc544e9a3bddf205b65360d: Bootstrap starting. I0703 01:40:29.375244 14735 catalog_manager.cc:2903] Scheduling retry of 3ea56191fad44303b1a8ce10e51741ed Delete Tablet RPC for TS=01f745abbee546cc83f2570e2628db1a with a delay of 536 ms (attempt = 6) W0703 01:40:29.375317 14735 catalog_manager.cc:2922] Async tablet task 3ea56191fad44303b1a8ce10e51741ed Delete Tablet RPC for TS=01f745abbee546cc83f2570e2628db1a failed: Not found: failed to reset TS proxy: Could not find TS for UUID 01f745abbee546cc83f2570e2628db1a I0703 01:40:29.378371 14824 log.cc:525] T 3ea56191fad44303b1a8ce10e51741ed P 136de19b9fc544e9a3bddf205b65360d: Log is configured to *not* fsync() on all Append() calls I0703 01:40:29.384464 14824 tablet_bootstrap.cc:427] T 3ea56191fad44303b1a8ce10e51741ed P 136de19b9fc544e9a3bddf205b65360d: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:40:29.384783 14824 tablet_bootstrap.cc:427] T 3ea56191fad44303b1a8ce10e51741ed P 136de19b9fc544e9a3bddf205b65360d: Bootstrap complete. I0703 01:40:29.385013 14824 ts_tablet_manager.cc:1045] T 3ea56191fad44303b1a8ce10e51741ed P 136de19b9fc544e9a3bddf205b65360d: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.005s I0703 01:40:29.389763 14824 raft_consensus.cc:325] T 3ea56191fad44303b1a8ce10e51741ed P 136de19b9fc544e9a3bddf205b65360d [term 7 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } peers { permanent_uuid: "e6994a142c374c4ba8577054aeaccbd7" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 59097 } attrs { promote: true } } peers { permanent_uuid: "ff379419dfbc4624adc84127282b52d4" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 49562 } attrs { promote: true } } peers { permanent_uuid: "136de19b9fc544e9a3bddf205b65360d" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 40102 } attrs { promote: true } } peers { permanent_uuid: "586291a7317b401c9f2f4bd48fe107e2" member_type: NON_VOTER last_known_addr { host: "127.5.240.194" port: 53793 } attrs { promote: true } } unsafe_config_change: true I0703 01:40:29.390058 14824 raft_consensus.cc:628] T 3ea56191fad44303b1a8ce10e51741ed P 136de19b9fc544e9a3bddf205b65360d [term 7 LEARNER]: Becoming Follower/Learner. State: Replica: 136de19b9fc544e9a3bddf205b65360d, State: Initialized, Role: LEARNER I0703 01:40:29.390323 14824 consensus_queue.cc:227] T 3ea56191fad44303b1a8ce10e51741ed P 136de19b9fc544e9a3bddf205b65360d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 7.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } peers { permanent_uuid: "e6994a142c374c4ba8577054aeaccbd7" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 59097 } attrs { promote: true } } peers { permanent_uuid: "ff379419dfbc4624adc84127282b52d4" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 49562 } attrs { promote: true } } peers { permanent_uuid: "136de19b9fc544e9a3bddf205b65360d" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 40102 } attrs { promote: true } } peers { permanent_uuid: "586291a7317b401c9f2f4bd48fe107e2" member_type: NON_VOTER last_known_addr { host: "127.5.240.194" port: 53793 } attrs { promote: true } } unsafe_config_change: true I0703 01:40:29.391631 14824 ts_tablet_manager.cc:1073] T 3ea56191fad44303b1a8ce10e51741ed P 136de19b9fc544e9a3bddf205b65360d: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s I0703 01:40:29.392230 14198 tablet_copy_service.cc:325] P 48755546d06649e78728d26d3b2bd82d: Request end of tablet copy session 136de19b9fc544e9a3bddf205b65360d-3ea56191fad44303b1a8ce10e51741ed received from {username='jenkins'} at 127.5.240.199:43474 I0703 01:40:29.392303 14198 tablet_copy_service.cc:413] P 48755546d06649e78728d26d3b2bd82d: ending tablet copy session 136de19b9fc544e9a3bddf205b65360d-3ea56191fad44303b1a8ce10e51741ed on tablet 3ea56191fad44303b1a8ce10e51741ed with peer 136de19b9fc544e9a3bddf205b65360d W0703 01:40:29.431257 14344 tablet.cc:1678] T 3ea56191fad44303b1a8ce10e51741ed P 136de19b9fc544e9a3bddf205b65360d: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:40:29.457252 14829 ts_tablet_manager.cc:669] T 3ea56191fad44303b1a8ce10e51741ed P e6994a142c374c4ba8577054aeaccbd7: Initiating tablet copy from peer 48755546d06649e78728d26d3b2bd82d (127.5.240.198:34012) I0703 01:40:29.457494 14829 tablet_copy_client.cc:239] T 3ea56191fad44303b1a8ce10e51741ed P e6994a142c374c4ba8577054aeaccbd7: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.198:34012 I0703 01:40:29.459401 14831 ts_tablet_manager.cc:669] T 3ea56191fad44303b1a8ce10e51741ed P ff379419dfbc4624adc84127282b52d4: Initiating tablet copy from peer 48755546d06649e78728d26d3b2bd82d (127.5.240.198:34012) I0703 01:40:29.459602 14831 tablet_copy_client.cc:239] T 3ea56191fad44303b1a8ce10e51741ed P ff379419dfbc4624adc84127282b52d4: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.198:34012 I0703 01:40:29.462182 14198 tablet_copy_service.cc:135] P 48755546d06649e78728d26d3b2bd82d: Received BeginTabletCopySession request for tablet 3ea56191fad44303b1a8ce10e51741ed from peer e6994a142c374c4ba8577054aeaccbd7 ({username='jenkins'} at 127.5.240.196:54245) I0703 01:40:29.462291 14198 tablet_copy_service.cc:156] P 48755546d06649e78728d26d3b2bd82d: Beginning new tablet copy session on tablet 3ea56191fad44303b1a8ce10e51741ed from peer e6994a142c374c4ba8577054aeaccbd7 at {username='jenkins'} at 127.5.240.196:54245: session id = e6994a142c374c4ba8577054aeaccbd7-3ea56191fad44303b1a8ce10e51741ed I0703 01:40:29.464074 14198 tablet_copy_source_session.cc:202] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:40:29.466503 14198 tablet_copy_service.cc:135] P 48755546d06649e78728d26d3b2bd82d: Received BeginTabletCopySession request for tablet 3ea56191fad44303b1a8ce10e51741ed from peer ff379419dfbc4624adc84127282b52d4 ({username='jenkins'} at 127.5.240.193:38218) I0703 01:40:29.466644 14198 tablet_copy_service.cc:156] P 48755546d06649e78728d26d3b2bd82d: Beginning new tablet copy session on tablet 3ea56191fad44303b1a8ce10e51741ed from peer ff379419dfbc4624adc84127282b52d4 at {username='jenkins'} at 127.5.240.193:38218: session id = ff379419dfbc4624adc84127282b52d4-3ea56191fad44303b1a8ce10e51741ed I0703 01:40:29.467290 14829 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 3ea56191fad44303b1a8ce10e51741ed. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:40:29.468196 14198 tablet_copy_source_session.cc:202] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:40:29.470768 14831 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 3ea56191fad44303b1a8ce10e51741ed. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:40:29.476058 14829 ts_tablet_manager.cc:1173] T 3ea56191fad44303b1a8ce10e51741ed P e6994a142c374c4ba8577054aeaccbd7: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:40:29.476330 14829 tablet_copy_client.cc:566] T 3ea56191fad44303b1a8ce10e51741ed P e6994a142c374c4ba8577054aeaccbd7: tablet copy: Starting download of 0 data blocks... I0703 01:40:29.476578 14829 tablet_copy_client.cc:529] T 3ea56191fad44303b1a8ce10e51741ed P e6994a142c374c4ba8577054aeaccbd7: tablet copy: Starting download of 1 WAL segments... I0703 01:40:29.479041 14829 tablet_copy_client.cc:405] T 3ea56191fad44303b1a8ce10e51741ed P e6994a142c374c4ba8577054aeaccbd7: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:40:29.479773 14831 ts_tablet_manager.cc:1173] T 3ea56191fad44303b1a8ce10e51741ed P ff379419dfbc4624adc84127282b52d4: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:40:29.480031 14831 tablet_copy_client.cc:566] T 3ea56191fad44303b1a8ce10e51741ed P ff379419dfbc4624adc84127282b52d4: tablet copy: Starting download of 0 data blocks... I0703 01:40:29.480427 14831 tablet_copy_client.cc:529] T 3ea56191fad44303b1a8ce10e51741ed P ff379419dfbc4624adc84127282b52d4: tablet copy: Starting download of 1 WAL segments... I0703 01:40:29.480765 14829 ts_tablet_manager.cc:1028] T 3ea56191fad44303b1a8ce10e51741ed P e6994a142c374c4ba8577054aeaccbd7: Bootstrapping tablet I0703 01:40:29.480988 14829 tablet_bootstrap.cc:427] T 3ea56191fad44303b1a8ce10e51741ed P e6994a142c374c4ba8577054aeaccbd7: Bootstrap starting. I0703 01:40:29.482765 14831 tablet_copy_client.cc:405] T 3ea56191fad44303b1a8ce10e51741ed P ff379419dfbc4624adc84127282b52d4: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:40:29.484422 14831 ts_tablet_manager.cc:1028] T 3ea56191fad44303b1a8ce10e51741ed P ff379419dfbc4624adc84127282b52d4: Bootstrapping tablet I0703 01:40:29.484870 14831 tablet_bootstrap.cc:427] T 3ea56191fad44303b1a8ce10e51741ed P ff379419dfbc4624adc84127282b52d4: Bootstrap starting. I0703 01:40:29.487229 14829 log.cc:525] T 3ea56191fad44303b1a8ce10e51741ed P e6994a142c374c4ba8577054aeaccbd7: Log is configured to *not* fsync() on all Append() calls I0703 01:40:29.489073 14831 log.cc:525] T 3ea56191fad44303b1a8ce10e51741ed P ff379419dfbc4624adc84127282b52d4: Log is configured to *not* fsync() on all Append() calls I0703 01:40:29.494246 14829 tablet_bootstrap.cc:427] T 3ea56191fad44303b1a8ce10e51741ed P e6994a142c374c4ba8577054aeaccbd7: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:40:29.494565 14829 tablet_bootstrap.cc:427] T 3ea56191fad44303b1a8ce10e51741ed P e6994a142c374c4ba8577054aeaccbd7: Bootstrap complete. I0703 01:40:29.494771 14829 ts_tablet_manager.cc:1045] T 3ea56191fad44303b1a8ce10e51741ed P e6994a142c374c4ba8577054aeaccbd7: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.004s I0703 01:40:29.496043 14831 tablet_bootstrap.cc:427] T 3ea56191fad44303b1a8ce10e51741ed P ff379419dfbc4624adc84127282b52d4: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:40:29.496578 14831 tablet_bootstrap.cc:427] T 3ea56191fad44303b1a8ce10e51741ed P ff379419dfbc4624adc84127282b52d4: Bootstrap complete. I0703 01:40:29.496800 14831 ts_tablet_manager.cc:1045] T 3ea56191fad44303b1a8ce10e51741ed P ff379419dfbc4624adc84127282b52d4: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.003s I0703 01:40:29.499609 14829 raft_consensus.cc:325] T 3ea56191fad44303b1a8ce10e51741ed P e6994a142c374c4ba8577054aeaccbd7 [term 7 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } peers { permanent_uuid: "e6994a142c374c4ba8577054aeaccbd7" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 59097 } attrs { promote: true } } peers { permanent_uuid: "ff379419dfbc4624adc84127282b52d4" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 49562 } attrs { promote: true } } peers { permanent_uuid: "136de19b9fc544e9a3bddf205b65360d" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 40102 } attrs { promote: true } } peers { permanent_uuid: "586291a7317b401c9f2f4bd48fe107e2" member_type: NON_VOTER last_known_addr { host: "127.5.240.194" port: 53793 } attrs { promote: true } } unsafe_config_change: true I0703 01:40:29.499922 14829 raft_consensus.cc:628] T 3ea56191fad44303b1a8ce10e51741ed P e6994a142c374c4ba8577054aeaccbd7 [term 7 LEARNER]: Becoming Follower/Learner. State: Replica: e6994a142c374c4ba8577054aeaccbd7, State: Initialized, Role: LEARNER I0703 01:40:29.500191 14829 consensus_queue.cc:227] T 3ea56191fad44303b1a8ce10e51741ed P e6994a142c374c4ba8577054aeaccbd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 7.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } peers { permanent_uuid: "e6994a142c374c4ba8577054aeaccbd7" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 59097 } attrs { promote: true } } peers { permanent_uuid: "ff379419dfbc4624adc84127282b52d4" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 49562 } attrs { promote: true } } peers { permanent_uuid: "136de19b9fc544e9a3bddf205b65360d" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 40102 } attrs { promote: true } } peers { permanent_uuid: "586291a7317b401c9f2f4bd48fe107e2" member_type: NON_VOTER last_known_addr { host: "127.5.240.194" port: 53793 } attrs { promote: true } } unsafe_config_change: true I0703 01:40:29.501694 14829 ts_tablet_manager.cc:1073] T 3ea56191fad44303b1a8ce10e51741ed P e6994a142c374c4ba8577054aeaccbd7: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s I0703 01:40:29.501667 14831 raft_consensus.cc:325] T 3ea56191fad44303b1a8ce10e51741ed P ff379419dfbc4624adc84127282b52d4 [term 7 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } peers { permanent_uuid: "e6994a142c374c4ba8577054aeaccbd7" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 59097 } attrs { promote: true } } peers { permanent_uuid: "ff379419dfbc4624adc84127282b52d4" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 49562 } attrs { promote: true } } peers { permanent_uuid: "136de19b9fc544e9a3bddf205b65360d" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 40102 } attrs { promote: true } } peers { permanent_uuid: "586291a7317b401c9f2f4bd48fe107e2" member_type: NON_VOTER last_known_addr { host: "127.5.240.194" port: 53793 } attrs { promote: true } } unsafe_config_change: true I0703 01:40:29.501986 14831 raft_consensus.cc:628] T 3ea56191fad44303b1a8ce10e51741ed P ff379419dfbc4624adc84127282b52d4 [term 7 LEARNER]: Becoming Follower/Learner. State: Replica: ff379419dfbc4624adc84127282b52d4, State: Initialized, Role: LEARNER I0703 01:40:29.502240 14831 consensus_queue.cc:227] T 3ea56191fad44303b1a8ce10e51741ed P ff379419dfbc4624adc84127282b52d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 7.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } peers { permanent_uuid: "e6994a142c374c4ba8577054aeaccbd7" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 59097 } attrs { promote: true } } peers { permanent_uuid: "ff379419dfbc4624adc84127282b52d4" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 49562 } attrs { promote: true } } peers { permanent_uuid: "136de19b9fc544e9a3bddf205b65360d" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 40102 } attrs { promote: true } } peers { permanent_uuid: "586291a7317b401c9f2f4bd48fe107e2" member_type: NON_VOTER last_known_addr { host: "127.5.240.194" port: 53793 } attrs { promote: true } } unsafe_config_change: true I0703 01:40:29.502588 14198 tablet_copy_service.cc:325] P 48755546d06649e78728d26d3b2bd82d: Request end of tablet copy session e6994a142c374c4ba8577054aeaccbd7-3ea56191fad44303b1a8ce10e51741ed received from {username='jenkins'} at 127.5.240.196:54245 I0703 01:40:29.502665 14198 tablet_copy_service.cc:413] P 48755546d06649e78728d26d3b2bd82d: ending tablet copy session e6994a142c374c4ba8577054aeaccbd7-3ea56191fad44303b1a8ce10e51741ed on tablet 3ea56191fad44303b1a8ce10e51741ed with peer e6994a142c374c4ba8577054aeaccbd7 I0703 01:40:29.504011 14831 ts_tablet_manager.cc:1073] T 3ea56191fad44303b1a8ce10e51741ed P ff379419dfbc4624adc84127282b52d4: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s I0703 01:40:29.504779 14198 tablet_copy_service.cc:325] P 48755546d06649e78728d26d3b2bd82d: Request end of tablet copy session ff379419dfbc4624adc84127282b52d4-3ea56191fad44303b1a8ce10e51741ed received from {username='jenkins'} at 127.5.240.193:38218 I0703 01:40:29.504843 14198 tablet_copy_service.cc:413] P 48755546d06649e78728d26d3b2bd82d: ending tablet copy session ff379419dfbc4624adc84127282b52d4-3ea56191fad44303b1a8ce10e51741ed on tablet 3ea56191fad44303b1a8ce10e51741ed with peer ff379419dfbc4624adc84127282b52d4 W0703 01:40:29.531103 13734 tablet.cc:1678] T 3ea56191fad44303b1a8ce10e51741ed P 586291a7317b401c9f2f4bd48fe107e2: Can't schedule compaction. Clean time has not been advanced past its initial value. W0703 01:40:29.694407 13611 tablet.cc:1678] T 3ea56191fad44303b1a8ce10e51741ed P ff379419dfbc4624adc84127282b52d4: Can't schedule compaction. Clean time has not been advanced past its initial value. W0703 01:40:29.707303 13978 tablet.cc:1678] T 3ea56191fad44303b1a8ce10e51741ed P e6994a142c374c4ba8577054aeaccbd7: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:40:29.761124 13690 raft_consensus.cc:1026] T 3ea56191fad44303b1a8ce10e51741ed P 586291a7317b401c9f2f4bd48fe107e2 [term 7 LEARNER]: Deduplicated request from leader. Original: 7.10->[7.11-7.11] Dedup: 7.11->[] I0703 01:40:29.844452 14733 catalog_manager.cc:2903] Scheduling retry of 3ea56191fad44303b1a8ce10e51741ed Delete Tablet RPC for TS=f519569b3d3b42fdb0a434061155ac38 with a delay of 1056 ms (attempt = 7) W0703 01:40:29.844516 14733 catalog_manager.cc:2922] Async tablet task 3ea56191fad44303b1a8ce10e51741ed Delete Tablet RPC for TS=f519569b3d3b42fdb0a434061155ac38 failed: Not found: failed to reset TS proxy: Could not find TS for UUID f519569b3d3b42fdb0a434061155ac38 I0703 01:40:29.865272 13934 raft_consensus.cc:1026] T 3ea56191fad44303b1a8ce10e51741ed P e6994a142c374c4ba8577054aeaccbd7 [term 7 LEARNER]: Deduplicated request from leader. Original: 7.10->[7.11-7.11] Dedup: 7.11->[] I0703 01:40:29.868939 14300 raft_consensus.cc:1026] T 3ea56191fad44303b1a8ce10e51741ed P 136de19b9fc544e9a3bddf205b65360d [term 7 LEARNER]: Deduplicated request from leader. Original: 7.10->[7.11-7.11] Dedup: 7.11->[] I0703 01:40:29.900738 14733 catalog_manager.cc:2903] Scheduling retry of 3ea56191fad44303b1a8ce10e51741ed Delete Tablet RPC for TS=42bc35d124544280bcfdeca6d6f6049b with a delay of 1059 ms (attempt = 7) W0703 01:40:29.900794 14733 catalog_manager.cc:2922] Async tablet task 3ea56191fad44303b1a8ce10e51741ed Delete Tablet RPC for TS=42bc35d124544280bcfdeca6d6f6049b failed: Not found: failed to reset TS proxy: Could not find TS for UUID 42bc35d124544280bcfdeca6d6f6049b I0703 01:40:29.911960 14735 catalog_manager.cc:2903] Scheduling retry of 3ea56191fad44303b1a8ce10e51741ed Delete Tablet RPC for TS=01f745abbee546cc83f2570e2628db1a with a delay of 1056 ms (attempt = 7) W0703 01:40:29.912050 14735 catalog_manager.cc:2922] Async tablet task 3ea56191fad44303b1a8ce10e51741ed Delete Tablet RPC for TS=01f745abbee546cc83f2570e2628db1a failed: Not found: failed to reset TS proxy: Could not find TS for UUID 01f745abbee546cc83f2570e2628db1a I0703 01:40:29.927429 14736 catalog_manager.cc:2903] Scheduling retry of 3ea56191fad44303b1a8ce10e51741ed Delete Tablet RPC for TS=0b034930462648f9b3f72485a0131667 with a delay of 1028 ms (attempt = 7) W0703 01:40:29.927480 14736 catalog_manager.cc:2922] Async tablet task 3ea56191fad44303b1a8ce10e51741ed Delete Tablet RPC for TS=0b034930462648f9b3f72485a0131667 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 0b034930462648f9b3f72485a0131667 I0703 01:40:30.016458 13567 raft_consensus.cc:1026] T 3ea56191fad44303b1a8ce10e51741ed P ff379419dfbc4624adc84127282b52d4 [term 7 LEARNER]: Deduplicated request from leader. Original: 7.10->[7.11-7.11] Dedup: 7.11->[] I0703 01:40:30.234666 14839 raft_consensus.cc:890] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d: attempting to promote NON_VOTER 586291a7317b401c9f2f4bd48fe107e2 to VOTER I0703 01:40:30.235020 14839 consensus_peers.cc:455] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d -> Peer ff379419dfbc4624adc84127282b52d4 (127.5.240.193:49562): Closing peer: ff379419dfbc4624adc84127282b52d4 I0703 01:40:30.235090 14839 consensus_peers.cc:455] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d -> Peer 586291a7317b401c9f2f4bd48fe107e2 (127.5.240.194:53793): Closing peer: 586291a7317b401c9f2f4bd48fe107e2 I0703 01:40:30.235155 14839 consensus_peers.cc:455] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d -> Peer 136de19b9fc544e9a3bddf205b65360d (127.5.240.199:40102): Closing peer: 136de19b9fc544e9a3bddf205b65360d I0703 01:40:30.235206 14839 consensus_peers.cc:455] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d -> Peer e6994a142c374c4ba8577054aeaccbd7 (127.5.240.196:59097): Closing peer: e6994a142c374c4ba8577054aeaccbd7 I0703 01:40:30.235466 14839 consensus_queue.cc:204] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 7.11, Last appended by leader: 0, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } peers { permanent_uuid: "e6994a142c374c4ba8577054aeaccbd7" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 59097 } attrs { promote: true } } peers { permanent_uuid: "ff379419dfbc4624adc84127282b52d4" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 49562 } attrs { promote: true } } peers { permanent_uuid: "136de19b9fc544e9a3bddf205b65360d" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 40102 } attrs { promote: true } } peers { permanent_uuid: "586291a7317b401c9f2f4bd48fe107e2" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 53793 } attrs { promote: false } } unsafe_config_change: true I0703 01:40:30.238315 13690 raft_consensus.cc:1081] T 3ea56191fad44303b1a8ce10e51741ed P 586291a7317b401c9f2f4bd48fe107e2 [term 7 LEARNER]: Refusing update from remote peer 48755546d06649e78728d26d3b2bd82d: Log matching property violated. Preceding OpId in replica: term: 7 index: 11. Preceding OpId from leader: term: 7 index: 12. (index mismatch) I0703 01:40:30.238353 13567 raft_consensus.cc:1081] T 3ea56191fad44303b1a8ce10e51741ed P ff379419dfbc4624adc84127282b52d4 [term 7 LEARNER]: Refusing update from remote peer 48755546d06649e78728d26d3b2bd82d: Log matching property violated. Preceding OpId in replica: term: 7 index: 11. Preceding OpId from leader: term: 7 index: 12. (index mismatch) I0703 01:40:30.238574 14300 raft_consensus.cc:1081] T 3ea56191fad44303b1a8ce10e51741ed P 136de19b9fc544e9a3bddf205b65360d [term 7 LEARNER]: Refusing update from remote peer 48755546d06649e78728d26d3b2bd82d: Log matching property violated. Preceding OpId in replica: term: 7 index: 11. Preceding OpId from leader: term: 7 index: 12. (index mismatch) I0703 01:40:30.238786 13934 raft_consensus.cc:1081] T 3ea56191fad44303b1a8ce10e51741ed P e6994a142c374c4ba8577054aeaccbd7 [term 7 LEARNER]: Refusing update from remote peer 48755546d06649e78728d26d3b2bd82d: Log matching property violated. Preceding OpId in replica: term: 7 index: 11. Preceding OpId from leader: term: 7 index: 12. (index mismatch) I0703 01:40:30.239270 14842 consensus_queue.cc:962] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff379419dfbc4624adc84127282b52d4" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 49562 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s I0703 01:40:30.239830 14838 consensus_queue.cc:962] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [LEADER]: Connected to new peer: Peer: permanent_uuid: "136de19b9fc544e9a3bddf205b65360d" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 40102 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s I0703 01:40:30.240216 14842 consensus_queue.cc:962] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6994a142c374c4ba8577054aeaccbd7" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 59097 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s I0703 01:40:30.240440 14839 consensus_queue.cc:962] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [LEADER]: Connected to new peer: Peer: permanent_uuid: "586291a7317b401c9f2f4bd48fe107e2" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 53793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s I0703 01:40:30.246240 14838 raft_consensus.cc:2684] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [term 7 LEADER]: Committing config change with OpId 7.12: config changed from index 11 to 12, 586291a7317b401c9f2f4bd48fe107e2 (127.5.240.194) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } peers { permanent_uuid: "e6994a142c374c4ba8577054aeaccbd7" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 59097 } attrs { promote: true } } peers { permanent_uuid: "ff379419dfbc4624adc84127282b52d4" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 49562 } attrs { promote: true } } peers { permanent_uuid: "136de19b9fc544e9a3bddf205b65360d" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 40102 } attrs { promote: true } } peers { permanent_uuid: "586291a7317b401c9f2f4bd48fe107e2" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 53793 } attrs { promote: false } } unsafe_config_change: true } I0703 01:40:30.247391 13934 raft_consensus.cc:2684] T 3ea56191fad44303b1a8ce10e51741ed P e6994a142c374c4ba8577054aeaccbd7 [term 7 LEARNER]: Committing config change with OpId 7.12: config changed from index 11 to 12, 586291a7317b401c9f2f4bd48fe107e2 (127.5.240.194) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } peers { permanent_uuid: "e6994a142c374c4ba8577054aeaccbd7" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 59097 } attrs { promote: true } } peers { permanent_uuid: "ff379419dfbc4624adc84127282b52d4" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 49562 } attrs { promote: true } } peers { permanent_uuid: "136de19b9fc544e9a3bddf205b65360d" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 40102 } attrs { promote: true } } peers { permanent_uuid: "586291a7317b401c9f2f4bd48fe107e2" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 53793 } attrs { promote: false } } unsafe_config_change: true } I0703 01:40:30.250250 14815 raft_consensus.cc:890] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d: attempting to promote NON_VOTER e6994a142c374c4ba8577054aeaccbd7 to VOTER I0703 01:40:30.250589 14815 consensus_peers.cc:455] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d -> Peer ff379419dfbc4624adc84127282b52d4 (127.5.240.193:49562): Closing peer: ff379419dfbc4624adc84127282b52d4 I0703 01:40:30.250689 14815 consensus_peers.cc:455] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d -> Peer 586291a7317b401c9f2f4bd48fe107e2 (127.5.240.194:53793): Closing peer: 586291a7317b401c9f2f4bd48fe107e2 I0703 01:40:30.249996 13690 raft_consensus.cc:2684] T 3ea56191fad44303b1a8ce10e51741ed P 586291a7317b401c9f2f4bd48fe107e2 [term 7 FOLLOWER]: Committing config change with OpId 7.12: config changed from index 11 to 12, 586291a7317b401c9f2f4bd48fe107e2 (127.5.240.194) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } peers { permanent_uuid: "e6994a142c374c4ba8577054aeaccbd7" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 59097 } attrs { promote: true } } peers { permanent_uuid: "ff379419dfbc4624adc84127282b52d4" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 49562 } attrs { promote: true } } peers { permanent_uuid: "136de19b9fc544e9a3bddf205b65360d" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 40102 } attrs { promote: true } } peers { permanent_uuid: "586291a7317b401c9f2f4bd48fe107e2" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 53793 } attrs { promote: false } } unsafe_config_change: true } I0703 01:40:30.250748 14815 consensus_peers.cc:455] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d -> Peer 136de19b9fc544e9a3bddf205b65360d (127.5.240.199:40102): Closing peer: 136de19b9fc544e9a3bddf205b65360d I0703 01:40:30.250800 14815 consensus_peers.cc:455] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d -> Peer e6994a142c374c4ba8577054aeaccbd7 (127.5.240.196:59097): Closing peer: e6994a142c374c4ba8577054aeaccbd7 I0703 01:40:30.250977 14815 consensus_queue.cc:204] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 7.12, Last appended by leader: 0, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } peers { permanent_uuid: "e6994a142c374c4ba8577054aeaccbd7" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 59097 } attrs { promote: false } } peers { permanent_uuid: "ff379419dfbc4624adc84127282b52d4" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 49562 } attrs { promote: true } } peers { permanent_uuid: "136de19b9fc544e9a3bddf205b65360d" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 40102 } attrs { promote: true } } peers { permanent_uuid: "586291a7317b401c9f2f4bd48fe107e2" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 53793 } attrs { promote: false } } unsafe_config_change: true I0703 01:40:30.251360 14751 catalog_manager.cc:3798] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d reported cstate change: config changed from index 11 to 12, 586291a7317b401c9f2f4bd48fe107e2 (127.5.240.194) changed from NON_VOTER to VOTER. New cstate: current_term: 7 leader_uuid: "48755546d06649e78728d26d3b2bd82d" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6994a142c374c4ba8577054aeaccbd7" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 59097 } attrs { promote: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff379419dfbc4624adc84127282b52d4" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 49562 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "136de19b9fc544e9a3bddf205b65360d" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 40102 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "586291a7317b401c9f2f4bd48fe107e2" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 53793 } attrs { promote: false } health_report { overall_health: HEALTHY } } unsafe_config_change: true } I0703 01:40:30.254626 14300 raft_consensus.cc:1081] T 3ea56191fad44303b1a8ce10e51741ed P 136de19b9fc544e9a3bddf205b65360d [term 7 LEARNER]: Refusing update from remote peer 48755546d06649e78728d26d3b2bd82d: Log matching property violated. Preceding OpId in replica: term: 7 index: 12. Preceding OpId from leader: term: 7 index: 13. (index mismatch) I0703 01:40:30.255154 13567 raft_consensus.cc:1081] T 3ea56191fad44303b1a8ce10e51741ed P ff379419dfbc4624adc84127282b52d4 [term 7 LEARNER]: Refusing update from remote peer 48755546d06649e78728d26d3b2bd82d: Log matching property violated. Preceding OpId in replica: term: 7 index: 12. Preceding OpId from leader: term: 7 index: 13. (index mismatch) I0703 01:40:30.255388 13689 raft_consensus.cc:1081] T 3ea56191fad44303b1a8ce10e51741ed P 586291a7317b401c9f2f4bd48fe107e2 [term 7 FOLLOWER]: Refusing update from remote peer 48755546d06649e78728d26d3b2bd82d: Log matching property violated. Preceding OpId in replica: term: 7 index: 12. Preceding OpId from leader: term: 7 index: 13. (index mismatch) I0703 01:40:30.256232 14842 consensus_queue.cc:962] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff379419dfbc4624adc84127282b52d4" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 49562 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s I0703 01:40:30.256575 14839 consensus_queue.cc:962] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [LEADER]: Connected to new peer: Peer: permanent_uuid: "586291a7317b401c9f2f4bd48fe107e2" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 53793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s I0703 01:40:30.257455 14815 consensus_queue.cc:962] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [LEADER]: Connected to new peer: Peer: permanent_uuid: "136de19b9fc544e9a3bddf205b65360d" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 40102 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s I0703 01:40:30.257439 13567 raft_consensus.cc:2684] T 3ea56191fad44303b1a8ce10e51741ed P ff379419dfbc4624adc84127282b52d4 [term 7 LEARNER]: Committing config change with OpId 7.12: config changed from index 11 to 12, 586291a7317b401c9f2f4bd48fe107e2 (127.5.240.194) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } peers { permanent_uuid: "e6994a142c374c4ba8577054aeaccbd7" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 59097 } attrs { promote: true } } peers { permanent_uuid: "ff379419dfbc4624adc84127282b52d4" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 49562 } attrs { promote: true } } peers { permanent_uuid: "136de19b9fc544e9a3bddf205b65360d" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 40102 } attrs { promote: true } } peers { permanent_uuid: "586291a7317b401c9f2f4bd48fe107e2" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 53793 } attrs { promote: false } } unsafe_config_change: true } I0703 01:40:30.259289 14842 raft_consensus.cc:2684] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [term 7 LEADER]: Committing config change with OpId 7.13: config changed from index 12 to 13, e6994a142c374c4ba8577054aeaccbd7 (127.5.240.196) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } peers { permanent_uuid: "e6994a142c374c4ba8577054aeaccbd7" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 59097 } attrs { promote: false } } peers { permanent_uuid: "ff379419dfbc4624adc84127282b52d4" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 49562 } attrs { promote: true } } peers { permanent_uuid: "136de19b9fc544e9a3bddf205b65360d" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 40102 } attrs { promote: true } } peers { permanent_uuid: "586291a7317b401c9f2f4bd48fe107e2" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 53793 } attrs { promote: false } } unsafe_config_change: true } I0703 01:40:30.259268 14300 raft_consensus.cc:2684] T 3ea56191fad44303b1a8ce10e51741ed P 136de19b9fc544e9a3bddf205b65360d [term 7 LEARNER]: Committing config change with OpId 7.12: config changed from index 11 to 12, 586291a7317b401c9f2f4bd48fe107e2 (127.5.240.194) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } peers { permanent_uuid: "e6994a142c374c4ba8577054aeaccbd7" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 59097 } attrs { promote: true } } peers { permanent_uuid: "ff379419dfbc4624adc84127282b52d4" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 49562 } attrs { promote: true } } peers { permanent_uuid: "136de19b9fc544e9a3bddf205b65360d" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 40102 } attrs { promote: true } } peers { permanent_uuid: "586291a7317b401c9f2f4bd48fe107e2" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 53793 } attrs { promote: false } } unsafe_config_change: true } I0703 01:40:30.260078 13689 raft_consensus.cc:2684] T 3ea56191fad44303b1a8ce10e51741ed P 586291a7317b401c9f2f4bd48fe107e2 [term 7 FOLLOWER]: Committing config change with OpId 7.13: config changed from index 12 to 13, e6994a142c374c4ba8577054aeaccbd7 (127.5.240.196) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } peers { permanent_uuid: "e6994a142c374c4ba8577054aeaccbd7" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 59097 } attrs { promote: false } } peers { permanent_uuid: "ff379419dfbc4624adc84127282b52d4" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 49562 } attrs { promote: true } } peers { permanent_uuid: "136de19b9fc544e9a3bddf205b65360d" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 40102 } attrs { promote: true } } peers { permanent_uuid: "586291a7317b401c9f2f4bd48fe107e2" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 53793 } attrs { promote: false } } unsafe_config_change: true } I0703 01:40:30.262382 14842 raft_consensus.cc:890] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d: attempting to promote NON_VOTER ff379419dfbc4624adc84127282b52d4 to VOTER I0703 01:40:30.262526 13567 raft_consensus.cc:2684] T 3ea56191fad44303b1a8ce10e51741ed P ff379419dfbc4624adc84127282b52d4 [term 7 LEARNER]: Committing config change with OpId 7.13: config changed from index 12 to 13, e6994a142c374c4ba8577054aeaccbd7 (127.5.240.196) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } peers { permanent_uuid: "e6994a142c374c4ba8577054aeaccbd7" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 59097 } attrs { promote: false } } peers { permanent_uuid: "ff379419dfbc4624adc84127282b52d4" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 49562 } attrs { promote: true } } peers { permanent_uuid: "136de19b9fc544e9a3bddf205b65360d" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 40102 } attrs { promote: true } } peers { permanent_uuid: "586291a7317b401c9f2f4bd48fe107e2" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 53793 } attrs { promote: false } } unsafe_config_change: true } I0703 01:40:30.262823 14842 consensus_peers.cc:455] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d -> Peer ff379419dfbc4624adc84127282b52d4 (127.5.240.193:49562): Closing peer: ff379419dfbc4624adc84127282b52d4 I0703 01:40:30.262899 14842 consensus_peers.cc:455] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d -> Peer 586291a7317b401c9f2f4bd48fe107e2 (127.5.240.194:53793): Closing peer: 586291a7317b401c9f2f4bd48fe107e2 I0703 01:40:30.262954 14842 consensus_peers.cc:455] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d -> Peer 136de19b9fc544e9a3bddf205b65360d (127.5.240.199:40102): Closing peer: 136de19b9fc544e9a3bddf205b65360d I0703 01:40:30.263017 14842 consensus_peers.cc:455] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d -> Peer e6994a142c374c4ba8577054aeaccbd7 (127.5.240.196:59097): Closing peer: e6994a142c374c4ba8577054aeaccbd7 I0703 01:40:30.263404 14842 consensus_queue.cc:204] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 7.13, Last appended by leader: 0, Current term: 7, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } peers { permanent_uuid: "e6994a142c374c4ba8577054aeaccbd7" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 59097 } attrs { promote: false } } peers { permanent_uuid: "ff379419dfbc4624adc84127282b52d4" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 49562 } attrs { promote: false } } peers { permanent_uuid: "136de19b9fc544e9a3bddf205b65360d" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 40102 } attrs { promote: true } } peers { permanent_uuid: "586291a7317b401c9f2f4bd48fe107e2" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 53793 } attrs { promote: false } } unsafe_config_change: true I0703 01:40:30.264423 13934 raft_consensus.cc:1081] T 3ea56191fad44303b1a8ce10e51741ed P e6994a142c374c4ba8577054aeaccbd7 [term 7 LEARNER]: Refusing update from remote peer 48755546d06649e78728d26d3b2bd82d: Log matching property violated. Preceding OpId in replica: term: 7 index: 12. Preceding OpId from leader: term: 7 index: 13. (index mismatch) I0703 01:40:30.267864 14751 catalog_manager.cc:3798] T 3ea56191fad44303b1a8ce10e51741ed P ff379419dfbc4624adc84127282b52d4 reported cstate change: config changed from index 12 to 13, e6994a142c374c4ba8577054aeaccbd7 (127.5.240.196) changed from NON_VOTER to VOTER. New cstate: current_term: 7 leader_uuid: "48755546d06649e78728d26d3b2bd82d" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } peers { permanent_uuid: "e6994a142c374c4ba8577054aeaccbd7" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 59097 } attrs { promote: false } } peers { permanent_uuid: "ff379419dfbc4624adc84127282b52d4" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 49562 } attrs { promote: true } } peers { permanent_uuid: "136de19b9fc544e9a3bddf205b65360d" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 40102 } attrs { promote: true } } peers { permanent_uuid: "586291a7317b401c9f2f4bd48fe107e2" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 53793 } attrs { promote: false } } unsafe_config_change: true } I0703 01:40:30.269327 13567 raft_consensus.cc:1081] T 3ea56191fad44303b1a8ce10e51741ed P ff379419dfbc4624adc84127282b52d4 [term 7 LEARNER]: Refusing update from remote peer 48755546d06649e78728d26d3b2bd82d: Log matching property violated. Preceding OpId in replica: term: 7 index: 13. Preceding OpId from leader: term: 7 index: 14. (index mismatch) I0703 01:40:30.269438 13689 raft_consensus.cc:1081] T 3ea56191fad44303b1a8ce10e51741ed P 586291a7317b401c9f2f4bd48fe107e2 [term 7 FOLLOWER]: Refusing update from remote peer 48755546d06649e78728d26d3b2bd82d: Log matching property violated. Preceding OpId in replica: term: 7 index: 13. Preceding OpId from leader: term: 7 index: 14. (index mismatch) I0703 01:40:30.270298 14300 raft_consensus.cc:1081] T 3ea56191fad44303b1a8ce10e51741ed P 136de19b9fc544e9a3bddf205b65360d [term 7 LEARNER]: Refusing update from remote peer 48755546d06649e78728d26d3b2bd82d: Log matching property violated. Preceding OpId in replica: term: 7 index: 13. Preceding OpId from leader: term: 7 index: 14. (index mismatch) I0703 01:40:30.270578 13934 raft_consensus.cc:1081] T 3ea56191fad44303b1a8ce10e51741ed P e6994a142c374c4ba8577054aeaccbd7 [term 7 LEARNER]: Refusing update from remote peer 48755546d06649e78728d26d3b2bd82d: Log matching property violated. Preceding OpId in replica: term: 7 index: 12. Preceding OpId from leader: term: 7 index: 14. (index mismatch) I0703 01:40:30.271517 14842 consensus_queue.cc:962] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff379419dfbc4624adc84127282b52d4" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 49562 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s I0703 01:40:30.271716 14815 consensus_queue.cc:962] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [LEADER]: Connected to new peer: Peer: permanent_uuid: "136de19b9fc544e9a3bddf205b65360d" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 40102 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s I0703 01:40:30.271931 14838 consensus_queue.cc:962] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [LEADER]: Connected to new peer: Peer: permanent_uuid: "586291a7317b401c9f2f4bd48fe107e2" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 53793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s I0703 01:40:30.272164 14847 consensus_queue.cc:962] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6994a142c374c4ba8577054aeaccbd7" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 59097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s I0703 01:40:30.273941 14300 raft_consensus.cc:2684] T 3ea56191fad44303b1a8ce10e51741ed P 136de19b9fc544e9a3bddf205b65360d [term 7 LEARNER]: Committing config change with OpId 7.13: config changed from index 12 to 13, e6994a142c374c4ba8577054aeaccbd7 (127.5.240.196) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } peers { permanent_uuid: "e6994a142c374c4ba8577054aeaccbd7" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 59097 } attrs { promote: false } } peers { permanent_uuid: "ff379419dfbc4624adc84127282b52d4" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 49562 } attrs { promote: true } } peers { permanent_uuid: "136de19b9fc544e9a3bddf205b65360d" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 40102 } attrs { promote: true } } peers { permanent_uuid: "586291a7317b401c9f2f4bd48fe107e2" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 53793 } attrs { promote: false } } unsafe_config_change: true } I0703 01:40:30.274142 13934 raft_consensus.cc:2849] T 3ea56191fad44303b1a8ce10e51741ed P e6994a142c374c4ba8577054aeaccbd7 [term 7 FOLLOWER]: Allowing unsafe config change even though there is a pending config! Existing pending config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } peers { permanent_uuid: "e6994a142c374c4ba8577054aeaccbd7" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 59097 } attrs { promote: false } } peers { permanent_uuid: "ff379419dfbc4624adc84127282b52d4" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 49562 } attrs { promote: true } } peers { permanent_uuid: "136de19b9fc544e9a3bddf205b65360d" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 40102 } attrs { promote: true } } peers { permanent_uuid: "586291a7317b401c9f2f4bd48fe107e2" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 53793 } attrs { promote: false } } unsafe_config_change: true; New pending config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } peers { permanent_uuid: "e6994a142c374c4ba8577054aeaccbd7" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 59097 } attrs { promote: false } } peers { permanent_uuid: "ff379419dfbc4624adc84127282b52d4" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 49562 } attrs { promote: false } } peers { permanent_uuid: "136de19b9fc544e9a3bddf205b65360d" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 40102 } attrs { promote: true } } peers { permanent_uuid: "586291a7317b401c9f2f4bd48fe107e2" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 53793 } attrs { promote: false } } unsafe_config_change: true I0703 01:40:30.275005 13934 raft_consensus.cc:2684] T 3ea56191fad44303b1a8ce10e51741ed P e6994a142c374c4ba8577054aeaccbd7 [term 7 FOLLOWER]: Committing config change with OpId 7.13: config changed from index 12 to 13, e6994a142c374c4ba8577054aeaccbd7 (127.5.240.196) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } peers { permanent_uuid: "e6994a142c374c4ba8577054aeaccbd7" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 59097 } attrs { promote: false } } peers { permanent_uuid: "ff379419dfbc4624adc84127282b52d4" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 49562 } attrs { promote: true } } peers { permanent_uuid: "136de19b9fc544e9a3bddf205b65360d" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 40102 } attrs { promote: true } } peers { permanent_uuid: "586291a7317b401c9f2f4bd48fe107e2" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 53793 } attrs { promote: false } } unsafe_config_change: true } I0703 01:40:30.275279 14847 raft_consensus.cc:2684] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [term 7 LEADER]: Committing config change with OpId 7.14: config changed from index 13 to 14, ff379419dfbc4624adc84127282b52d4 (127.5.240.193) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } peers { permanent_uuid: "e6994a142c374c4ba8577054aeaccbd7" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 59097 } attrs { promote: false } } peers { permanent_uuid: "ff379419dfbc4624adc84127282b52d4" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 49562 } attrs { promote: false } } peers { permanent_uuid: "136de19b9fc544e9a3bddf205b65360d" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 40102 } attrs { promote: true } } peers { permanent_uuid: "586291a7317b401c9f2f4bd48fe107e2" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 53793 } attrs { promote: false } } unsafe_config_change: true } I0703 01:40:30.279172 14847 raft_consensus.cc:890] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d: attempting to promote NON_VOTER 136de19b9fc544e9a3bddf205b65360d to VOTER I0703 01:40:30.279428 13689 raft_consensus.cc:2684] T 3ea56191fad44303b1a8ce10e51741ed P 586291a7317b401c9f2f4bd48fe107e2 [term 7 FOLLOWER]: Committing config change with OpId 7.14: config changed from index 13 to 14, ff379419dfbc4624adc84127282b52d4 (127.5.240.193) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } peers { permanent_uuid: "e6994a142c374c4ba8577054aeaccbd7" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 59097 } attrs { promote: false } } peers { permanent_uuid: "ff379419dfbc4624adc84127282b52d4" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 49562 } attrs { promote: false } } peers { permanent_uuid: "136de19b9fc544e9a3bddf205b65360d" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 40102 } attrs { promote: true } } peers { permanent_uuid: "586291a7317b401c9f2f4bd48fe107e2" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 53793 } attrs { promote: false } } unsafe_config_change: true } I0703 01:40:30.280186 14750 catalog_manager.cc:3798] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d reported cstate change: config changed from index 13 to 14, ff379419dfbc4624adc84127282b52d4 (127.5.240.193) changed from NON_VOTER to VOTER. New cstate: current_term: 7 leader_uuid: "48755546d06649e78728d26d3b2bd82d" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6994a142c374c4ba8577054aeaccbd7" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 59097 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff379419dfbc4624adc84127282b52d4" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 49562 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "136de19b9fc544e9a3bddf205b65360d" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 40102 } attrs { promote: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "586291a7317b401c9f2f4bd48fe107e2" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 53793 } attrs { promote: false } health_report { overall_health: HEALTHY } } unsafe_config_change: true } I0703 01:40:30.280391 14300 raft_consensus.cc:2684] T 3ea56191fad44303b1a8ce10e51741ed P 136de19b9fc544e9a3bddf205b65360d [term 7 LEARNER]: Committing config change with OpId 7.14: config changed from index 13 to 14, ff379419dfbc4624adc84127282b52d4 (127.5.240.193) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } peers { permanent_uuid: "e6994a142c374c4ba8577054aeaccbd7" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 59097 } attrs { promote: false } } peers { permanent_uuid: "ff379419dfbc4624adc84127282b52d4" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 49562 } attrs { promote: false } } peers { permanent_uuid: "136de19b9fc544e9a3bddf205b65360d" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 40102 } attrs { promote: true } } peers { permanent_uuid: "586291a7317b401c9f2f4bd48fe107e2" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 53793 } attrs { promote: false } } unsafe_config_change: true } I0703 01:40:30.280673 13934 raft_consensus.cc:2684] T 3ea56191fad44303b1a8ce10e51741ed P e6994a142c374c4ba8577054aeaccbd7 [term 7 FOLLOWER]: Committing config change with OpId 7.14: config changed from index 13 to 14, ff379419dfbc4624adc84127282b52d4 (127.5.240.193) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } peers { permanent_uuid: "e6994a142c374c4ba8577054aeaccbd7" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 59097 } attrs { promote: false } } peers { permanent_uuid: "ff379419dfbc4624adc84127282b52d4" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 49562 } attrs { promote: false } } peers { permanent_uuid: "136de19b9fc544e9a3bddf205b65360d" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 40102 } attrs { promote: true } } peers { permanent_uuid: "586291a7317b401c9f2f4bd48fe107e2" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 53793 } attrs { promote: false } } unsafe_config_change: true } I0703 01:40:30.281805 14847 consensus_peers.cc:455] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d -> Peer ff379419dfbc4624adc84127282b52d4 (127.5.240.193:49562): Closing peer: ff379419dfbc4624adc84127282b52d4 I0703 01:40:30.281877 14847 consensus_peers.cc:455] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d -> Peer 586291a7317b401c9f2f4bd48fe107e2 (127.5.240.194:53793): Closing peer: 586291a7317b401c9f2f4bd48fe107e2 I0703 01:40:30.281929 14847 consensus_peers.cc:455] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d -> Peer 136de19b9fc544e9a3bddf205b65360d (127.5.240.199:40102): Closing peer: 136de19b9fc544e9a3bddf205b65360d I0703 01:40:30.281985 14847 consensus_peers.cc:455] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d -> Peer e6994a142c374c4ba8577054aeaccbd7 (127.5.240.196:59097): Closing peer: e6994a142c374c4ba8577054aeaccbd7 I0703 01:40:30.282155 14847 consensus_queue.cc:204] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 7.14, Last appended by leader: 0, Current term: 7, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } peers { permanent_uuid: "e6994a142c374c4ba8577054aeaccbd7" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 59097 } attrs { promote: false } } peers { permanent_uuid: "ff379419dfbc4624adc84127282b52d4" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 49562 } attrs { promote: false } } peers { permanent_uuid: "136de19b9fc544e9a3bddf205b65360d" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 40102 } attrs { promote: false } } peers { permanent_uuid: "586291a7317b401c9f2f4bd48fe107e2" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 53793 } attrs { promote: false } } unsafe_config_change: true I0703 01:40:30.285447 14299 raft_consensus.cc:1081] T 3ea56191fad44303b1a8ce10e51741ed P 136de19b9fc544e9a3bddf205b65360d [term 7 LEARNER]: Refusing update from remote peer 48755546d06649e78728d26d3b2bd82d: Log matching property violated. Preceding OpId in replica: term: 7 index: 14. Preceding OpId from leader: term: 7 index: 15. (index mismatch) I0703 01:40:30.285627 13689 raft_consensus.cc:1081] T 3ea56191fad44303b1a8ce10e51741ed P 586291a7317b401c9f2f4bd48fe107e2 [term 7 FOLLOWER]: Refusing update from remote peer 48755546d06649e78728d26d3b2bd82d: Log matching property violated. Preceding OpId in replica: term: 7 index: 14. Preceding OpId from leader: term: 7 index: 15. (index mismatch) I0703 01:40:30.285477 13567 raft_consensus.cc:2684] T 3ea56191fad44303b1a8ce10e51741ed P ff379419dfbc4624adc84127282b52d4 [term 7 FOLLOWER]: Committing config change with OpId 7.14: config changed from index 13 to 14, ff379419dfbc4624adc84127282b52d4 (127.5.240.193) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } peers { permanent_uuid: "e6994a142c374c4ba8577054aeaccbd7" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 59097 } attrs { promote: false } } peers { permanent_uuid: "ff379419dfbc4624adc84127282b52d4" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 49562 } attrs { promote: false } } peers { permanent_uuid: "136de19b9fc544e9a3bddf205b65360d" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 40102 } attrs { promote: true } } peers { permanent_uuid: "586291a7317b401c9f2f4bd48fe107e2" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 53793 } attrs { promote: false } } unsafe_config_change: true } I0703 01:40:30.286218 13934 raft_consensus.cc:1081] T 3ea56191fad44303b1a8ce10e51741ed P e6994a142c374c4ba8577054aeaccbd7 [term 7 FOLLOWER]: Refusing update from remote peer 48755546d06649e78728d26d3b2bd82d: Log matching property violated. Preceding OpId in replica: term: 7 index: 14. Preceding OpId from leader: term: 7 index: 15. (index mismatch) I0703 01:40:30.287150 14839 consensus_queue.cc:962] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [LEADER]: Connected to new peer: Peer: permanent_uuid: "136de19b9fc544e9a3bddf205b65360d" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 40102 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s I0703 01:40:30.287416 14815 consensus_queue.cc:962] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6994a142c374c4ba8577054aeaccbd7" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 59097 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s I0703 01:40:30.287928 14839 consensus_queue.cc:962] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [LEADER]: Connected to new peer: Peer: permanent_uuid: "586291a7317b401c9f2f4bd48fe107e2" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 53793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s I0703 01:40:30.289085 13566 raft_consensus.cc:1081] T 3ea56191fad44303b1a8ce10e51741ed P ff379419dfbc4624adc84127282b52d4 [term 7 FOLLOWER]: Refusing update from remote peer 48755546d06649e78728d26d3b2bd82d: Log matching property violated. Preceding OpId in replica: term: 7 index: 14. Preceding OpId from leader: term: 7 index: 15. (index mismatch) I0703 01:40:30.292325 14839 raft_consensus.cc:2684] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [term 7 LEADER]: Committing config change with OpId 7.15: config changed from index 14 to 15, 136de19b9fc544e9a3bddf205b65360d (127.5.240.199) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } peers { permanent_uuid: "e6994a142c374c4ba8577054aeaccbd7" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 59097 } attrs { promote: false } } peers { permanent_uuid: "ff379419dfbc4624adc84127282b52d4" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 49562 } attrs { promote: false } } peers { permanent_uuid: "136de19b9fc544e9a3bddf205b65360d" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 40102 } attrs { promote: false } } peers { permanent_uuid: "586291a7317b401c9f2f4bd48fe107e2" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 53793 } attrs { promote: false } } unsafe_config_change: true } I0703 01:40:30.292917 14300 raft_consensus.cc:2684] T 3ea56191fad44303b1a8ce10e51741ed P 136de19b9fc544e9a3bddf205b65360d [term 7 FOLLOWER]: Committing config change with OpId 7.15: config changed from index 14 to 15, 136de19b9fc544e9a3bddf205b65360d (127.5.240.199) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } peers { permanent_uuid: "e6994a142c374c4ba8577054aeaccbd7" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 59097 } attrs { promote: false } } peers { permanent_uuid: "ff379419dfbc4624adc84127282b52d4" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 49562 } attrs { promote: false } } peers { permanent_uuid: "136de19b9fc544e9a3bddf205b65360d" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 40102 } attrs { promote: false } } peers { permanent_uuid: "586291a7317b401c9f2f4bd48fe107e2" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 53793 } attrs { promote: false } } unsafe_config_change: true } I0703 01:40:30.295097 14815 consensus_queue.cc:962] T 3ea56191fad44303b1a8ce10e51741ed P 48755546d06649e78728d26d3b2bd82d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff379419dfbc4624adc84127282b52d4" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 49562 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s I0703 01:40:30.295240 13934 raft_consensus.cc:2684] T 3ea56191fad44303b1a8ce10e51741ed P e6994a142c374c4ba8577054aeaccbd7 [term 7 FOLLOWER]: Committing config change with OpId 7.15: config changed from index 14 to 15, 136de19b9fc544e9a3bddf205b65360d (127.5.240.199) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } peers { permanent_uuid: "e6994a142c374c4ba8577054aeaccbd7" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 59097 } attrs { promote: false } } peers { permanent_uuid: "ff379419dfbc4624adc84127282b52d4" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 49562 } attrs { promote: false } } peers { permanent_uuid: "136de19b9fc544e9a3bddf205b65360d" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 40102 } attrs { promote: false } } peers { permanent_uuid: "586291a7317b401c9f2f4bd48fe107e2" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 53793 } attrs { promote: false } } unsafe_config_change: true } I0703 01:40:30.296511 13689 raft_consensus.cc:2684] T 3ea56191fad44303b1a8ce10e51741ed P 586291a7317b401c9f2f4bd48fe107e2 [term 7 FOLLOWER]: Committing config change with OpId 7.15: config changed from index 14 to 15, 136de19b9fc544e9a3bddf205b65360d (127.5.240.199) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } peers { permanent_uuid: "e6994a142c374c4ba8577054aeaccbd7" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 59097 } attrs { promote: false } } peers { permanent_uuid: "ff379419dfbc4624adc84127282b52d4" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 49562 } attrs { promote: false } } peers { permanent_uuid: "136de19b9fc544e9a3bddf205b65360d" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 40102 } attrs { promote: false } } peers { permanent_uuid: "586291a7317b401c9f2f4bd48fe107e2" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 53793 } attrs { promote: false } } unsafe_config_change: true } I0703 01:40:30.296805 14752 catalog_manager.cc:3798] T 3ea56191fad44303b1a8ce10e51741ed P 136de19b9fc544e9a3bddf205b65360d reported cstate change: config changed from index 14 to 15, 136de19b9fc544e9a3bddf205b65360d (127.5.240.199) changed from NON_VOTER to VOTER. New cstate: current_term: 7 leader_uuid: "48755546d06649e78728d26d3b2bd82d" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } peers { permanent_uuid: "e6994a142c374c4ba8577054aeaccbd7" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 59097 } attrs { promote: false } } peers { permanent_uuid: "ff379419dfbc4624adc84127282b52d4" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 49562 } attrs { promote: false } } peers { permanent_uuid: "136de19b9fc544e9a3bddf205b65360d" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 40102 } attrs { promote: false } } peers { permanent_uuid: "586291a7317b401c9f2f4bd48fe107e2" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 53793 } attrs { promote: false } } unsafe_config_change: true } I0703 01:40:30.296835 13567 raft_consensus.cc:2684] T 3ea56191fad44303b1a8ce10e51741ed P ff379419dfbc4624adc84127282b52d4 [term 7 FOLLOWER]: Committing config change with OpId 7.15: config changed from index 14 to 15, 136de19b9fc544e9a3bddf205b65360d (127.5.240.199) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 34012 } } peers { permanent_uuid: "e6994a142c374c4ba8577054aeaccbd7" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 59097 } attrs { promote: false } } peers { permanent_uuid: "ff379419dfbc4624adc84127282b52d4" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 49562 } attrs { promote: false } } peers { permanent_uuid: "136de19b9fc544e9a3bddf205b65360d" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 40102 } attrs { promote: false } } peers { permanent_uuid: "586291a7317b401c9f2f4bd48fe107e2" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 53793 } attrs { promote: false } } unsafe_config_change: true } I0703 01:40:30.418795 6083 kudu-admin-test.cc:1124] Waiting for Master to see new config... I0703 01:40:30.424473 6083 kudu-admin-test.cc:1129] Tablet locations: tablet_id: "3ea56191fad44303b1a8ce10e51741ed" replicas { ts_info { permanent_uuid: "48755546d06649e78728d26d3b2bd82d" rpc_addresses { host: "127.5.240.198" port: 34012 } } role: LEADER } replicas { ts_info { permanent_uuid: "e6994a142c374c4ba8577054aeaccbd7" rpc_addresses { host: "127.5.240.196" port: 59097 } } role: FOLLOWER } replicas { ts_info { permanent_uuid: "ff379419dfbc4624adc84127282b52d4" rpc_addresses { host: "127.5.240.193" port: 49562 } } role: FOLLOWER } replicas { ts_info { permanent_uuid: "136de19b9fc544e9a3bddf205b65360d" rpc_addresses { host: "127.5.240.199" port: 40102 } } role: FOLLOWER } replicas { ts_info { permanent_uuid: "586291a7317b401c9f2f4bd48fe107e2" rpc_addresses { host: "127.5.240.194" port: 53793 } } role: FOLLOWER } DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } I0703 01:40:30.425472 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 13481 I0703 01:40:30.426393 13607 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:40:30.475474 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 13481 W0703 01:40:30.483916 14111 connection.cc:512] server connection from 127.5.240.193:38218 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:38218): Connection reset by peer (error 104) W0703 01:40:30.483919 14736 connection.cc:512] server connection from 127.5.240.193:39241 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:39241): Connection reset by peer (error 104) W0703 01:40:30.484241 14112 connection.cc:512] client connection to 127.5.240.193:49562 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:49562): Connection reset by peer (error 104) I0703 01:40:30.484448 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 13614 I0703 01:40:30.485280 13730 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:40:30.534121 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 13614 W0703 01:40:30.542855 14112 connection.cc:512] client connection to 127.5.240.194:53793 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:53793): Connection reset by peer (error 104) W0703 01:40:30.542855 14735 connection.cc:512] server connection from 127.5.240.194:41935 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:41935): Connection reset by peer (error 104) I0703 01:40:30.543078 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 13858 W0703 01:40:30.543138 14111 connection.cc:512] server connection from 127.5.240.194:51732 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:51732): Connection reset by peer (error 104) I0703 01:40:30.543915 13974 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:40:30.588413 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 13858 W0703 01:40:30.596437 14111 connection.cc:512] client connection to 127.5.240.196:59097 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.196:59097): Connection reset by peer (error 104) W0703 01:40:30.596446 14735 connection.cc:512] server connection from 127.5.240.196:57473 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.196:57473): Connection reset by peer (error 104) W0703 01:40:30.596891 14109 connection.cc:512] server connection from 127.5.240.196:54245 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.196:54245): Connection reset by peer (error 104) I0703 01:40:30.596941 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 14102 I0703 01:40:30.597633 14218 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:40:30.648835 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 14102 W0703 01:40:30.658530 14733 connection.cc:512] server connection from 127.5.240.198:58666 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.198:58666): Connection reset by peer (error 104) W0703 01:40:30.658828 14232 connection.cc:512] server connection from 127.5.240.198:35677 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.198:35677): Connection reset by peer (error 104) I0703 01:40:30.658955 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 14224 W0703 01:40:30.659143 14232 connection.cc:512] client connection to 127.5.240.198:34012 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.198:34012): Connection reset by peer (error 104) I0703 01:40:30.659713 14340 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:40:30.703101 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 14224 W0703 01:40:30.711249 14735 connection.cc:512] server connection from 127.5.240.199:37852 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.199:37852): Connection reset by peer (error 104) I0703 01:40:30.711364 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master pid 14725 I0703 01:40:30.712131 14782 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:40:30.776291 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master with pid 14725 [ OK ] AdminCliTest.TestUnsafeChangeConfigWithMultiplePendingConfigs (10017 ms) [ RUN ] AdminCliTest.TestLeaderStepDown I0703 01:40:30.801477 6083 test_util.cc:208] Using random seed: 1145591346 I0703 01:40:30.803218 6083 ts_itest-base.cc:113] Starting cluster with: I0703 01:40:30.803254 6083 ts_itest-base.cc:114] -------------- I0703 01:40:30.803280 6083 ts_itest-base.cc:115] 3 tablet servers I0703 01:40:30.803309 6083 ts_itest-base.cc:116] 3 replicas per TS I0703 01:40:30.803335 6083 ts_itest-base.cc:117] -------------- I0703 01:40:30.804493 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:41705 --webserver_interface=127.5.240.254 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:40:30.983521 14865 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:40:30.984566 14865 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:40:30.986793 14865 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0703 01:40:30.986838 14865 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:40:30.986881 14865 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0703 01:40:30.986927 14865 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0703 01:40:30.994247 14865 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:41705 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.5.240.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:40:30.999866 14865 master_main.cc:80] Initializing master server... I0703 01:40:31.000941 14865 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 168708us I0703 01:40:31.002032 14865 fs_manager.cc:261] Metadata directory not provided I0703 01:40:31.002095 14865 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:40:31.002360 14865 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:40:31.002424 14865 server_base.cc:433] Attempting to create new FS layout instead I0703 01:40:31.007354 14865 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/instance: uuid: "a71583e9997a48a3a6ef003970a2bbe1" format_stamp: "Formatted at 2018-07-03 08:40:31 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:31.007889 14865 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "a71583e9997a48a3a6ef003970a2bbe1" format_stamp: "Formatted at 2018-07-03 08:40:31 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:31.016249 14865 fs_manager.cc:501] Time spent creating directory manager: real 0.008s user 0.003s sys 0.004s I0703 01:40:31.017458 14865 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:40:31.017880 14865 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:40:31.021503 14865 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s I0703 01:40:31.021580 14865 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal uuid: "a71583e9997a48a3a6ef003970a2bbe1" format_stamp: "Formatted at 2018-07-03 08:40:31 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:31.021697 14865 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:40:31.049059 14865 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:40:31.050861 14865 master_main.cc:83] Starting Master server... I0703 01:40:31.088184 14865 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.254:41705 I0703 01:40:31.088666 14865 webserver.cc:175] Starting webserver on 127.5.240.254:0 I0703 01:40:31.088711 14865 webserver.cc:186] Document root disabled I0703 01:40:31.089445 14865 webserver.cc:313] Webserver started. Bound to: http://127.5.240.254:46211/ I0703 01:40:31.090037 14865 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb I0703 01:40:31.093708 14924 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:40:31.095116 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master as pid 14865 I0703 01:40:31.095741 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.5.240.193:0 --local_ip_for_outbound_sockets=127.5.240.193 --webserver_interface=127.5.240.193 --webserver_port=0 --tserver_master_addrs=127.5.240.254:41705 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0703 01:40:31.105303 14924 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P a71583e9997a48a3a6ef003970a2bbe1: Bootstrap starting. I0703 01:40:31.107317 14924 tablet_bootstrap.cc:578] T 00000000000000000000000000000000 P a71583e9997a48a3a6ef003970a2bbe1: No blocks or log segments found. Creating new log. I0703 01:40:31.108081 14924 log.cc:525] T 00000000000000000000000000000000 P a71583e9997a48a3a6ef003970a2bbe1: Log is configured to *not* fsync() on all Append() calls I0703 01:40:31.109868 14924 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P a71583e9997a48a3a6ef003970a2bbe1: No bootstrap required, opened a new log I0703 01:40:31.116058 14924 raft_consensus.cc:325] T 00000000000000000000000000000000 P a71583e9997a48a3a6ef003970a2bbe1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a71583e9997a48a3a6ef003970a2bbe1" member_type: VOTER } I0703 01:40:31.116252 14924 raft_consensus.cc:351] T 00000000000000000000000000000000 P a71583e9997a48a3a6ef003970a2bbe1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:40:31.116313 14924 raft_consensus.cc:628] T 00000000000000000000000000000000 P a71583e9997a48a3a6ef003970a2bbe1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a71583e9997a48a3a6ef003970a2bbe1, State: Initialized, Role: FOLLOWER I0703 01:40:31.116792 14924 consensus_queue.cc:227] T 00000000000000000000000000000000 P a71583e9997a48a3a6ef003970a2bbe1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a71583e9997a48a3a6ef003970a2bbe1" member_type: VOTER } I0703 01:40:31.116967 14924 raft_consensus.cc:365] T 00000000000000000000000000000000 P a71583e9997a48a3a6ef003970a2bbe1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:40:31.117050 14924 raft_consensus.cc:457] T 00000000000000000000000000000000 P a71583e9997a48a3a6ef003970a2bbe1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:40:31.117137 14924 raft_consensus.cc:2730] T 00000000000000000000000000000000 P a71583e9997a48a3a6ef003970a2bbe1: Snoozing failure detection for 2.189s (starting election) I0703 01:40:31.117230 14924 raft_consensus.cc:2782] T 00000000000000000000000000000000 P a71583e9997a48a3a6ef003970a2bbe1 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:40:31.118590 14924 raft_consensus.cc:479] T 00000000000000000000000000000000 P a71583e9997a48a3a6ef003970a2bbe1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a71583e9997a48a3a6ef003970a2bbe1" member_type: VOTER } I0703 01:40:31.118957 14924 leader_election.cc:270] T 00000000000000000000000000000000 P a71583e9997a48a3a6ef003970a2bbe1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:40:31.119825 14927 raft_consensus.cc:2730] T 00000000000000000000000000000000 P a71583e9997a48a3a6ef003970a2bbe1: Snoozing failure detection for 1.595s (election complete) I0703 01:40:31.120146 14927 raft_consensus.cc:2537] T 00000000000000000000000000000000 P a71583e9997a48a3a6ef003970a2bbe1 [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:40:31.122192 14927 raft_consensus.cc:593] T 00000000000000000000000000000000 P a71583e9997a48a3a6ef003970a2bbe1 [term 1 LEADER]: Becoming Leader. State: Replica: a71583e9997a48a3a6ef003970a2bbe1, State: Running, Role: LEADER I0703 01:40:31.122648 14927 consensus_queue.cc:204] T 00000000000000000000000000000000 P a71583e9997a48a3a6ef003970a2bbe1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a71583e9997a48a3a6ef003970a2bbe1" member_type: VOTER } I0703 01:40:31.124783 14924 sys_catalog.cc:337] T 00000000000000000000000000000000 P a71583e9997a48a3a6ef003970a2bbe1 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "a71583e9997a48a3a6ef003970a2bbe1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a71583e9997a48a3a6ef003970a2bbe1" member_type: VOTER } } I0703 01:40:31.125164 14924 sys_catalog.cc:340] T 00000000000000000000000000000000 P a71583e9997a48a3a6ef003970a2bbe1 [sys.catalog]: This master's current role is: LEADER I0703 01:40:31.124382 14929 sys_catalog.cc:337] T 00000000000000000000000000000000 P a71583e9997a48a3a6ef003970a2bbe1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a71583e9997a48a3a6ef003970a2bbe1. Latest consensus state: current_term: 1 leader_uuid: "a71583e9997a48a3a6ef003970a2bbe1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a71583e9997a48a3a6ef003970a2bbe1" member_type: VOTER } } I0703 01:40:31.124776 14928 sys_catalog.cc:337] T 00000000000000000000000000000000 P a71583e9997a48a3a6ef003970a2bbe1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a71583e9997a48a3a6ef003970a2bbe1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a71583e9997a48a3a6ef003970a2bbe1" member_type: VOTER } } I0703 01:40:31.126286 14929 sys_catalog.cc:340] T 00000000000000000000000000000000 P a71583e9997a48a3a6ef003970a2bbe1 [sys.catalog]: This master's current role is: LEADER I0703 01:40:31.126288 14928 sys_catalog.cc:340] T 00000000000000000000000000000000 P a71583e9997a48a3a6ef003970a2bbe1 [sys.catalog]: This master's current role is: LEADER I0703 01:40:31.129006 14924 sys_catalog.cc:424] T 00000000000000000000000000000000 P a71583e9997a48a3a6ef003970a2bbe1 [sys.catalog]: configured and running, proceeding with master startup. I0703 01:40:31.129302 14931 catalog_manager.cc:1006] Loading table and tablet metadata into memory... I0703 01:40:31.130647 14865 master_main.cc:86] Master server successfully started. I0703 01:40:31.135624 14931 catalog_manager.cc:1017] Initializing Kudu internal certificate authority... I0703 01:40:31.228302 14931 catalog_manager.cc:904] Generated new certificate authority record I0703 01:40:31.230576 14931 catalog_manager.cc:1026] Loading token signing keys... I0703 01:40:31.252705 14931 catalog_manager.cc:4102] T 00000000000000000000000000000000 P a71583e9997a48a3a6ef003970a2bbe1: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:40:31.283387 14925 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:40:31.284420 14925 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:40:31.286487 14925 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:40:31.286603 14925 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.193 I0703 01:40:31.293449 14925 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.5.240.193 --webserver_port=0 --tserver_master_addrs=127.5.240.254:41705 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.193 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:40:31.300096 14925 tablet_server_main.cc:85] Initializing tablet server... I0703 01:40:31.301292 14925 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 169208us I0703 01:40:31.302285 14925 fs_manager.cc:261] Metadata directory not provided I0703 01:40:31.302350 14925 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:40:31.303002 14925 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:40:31.303066 14925 server_base.cc:433] Attempting to create new FS layout instead I0703 01:40:31.307216 14925 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "4cbf0a90d2564e67bcd791d9791ca759" format_stamp: "Formatted at 2018-07-03 08:40:31 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:31.307703 14925 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "4cbf0a90d2564e67bcd791d9791ca759" format_stamp: "Formatted at 2018-07-03 08:40:31 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:31.315987 14925 fs_manager.cc:501] Time spent creating directory manager: real 0.008s user 0.003s sys 0.003s I0703 01:40:31.317199 14925 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:40:31.317595 14925 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:40:31.321069 14925 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s I0703 01:40:31.321127 14925 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal uuid: "4cbf0a90d2564e67bcd791d9791ca759" format_stamp: "Formatted at 2018-07-03 08:40:31 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:31.321250 14925 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:40:31.352358 14925 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:40:31.354460 14925 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:40:31.354920 14925 tablet_server_main.cc:90] Starting tablet server... I0703 01:40:31.463194 14925 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.193:41919 I0703 01:40:31.463706 14925 webserver.cc:175] Starting webserver on 127.5.240.193:0 I0703 01:40:31.463745 14925 webserver.cc:186] Document root disabled I0703 01:40:31.464537 14925 webserver.cc:313] Webserver started. Bound to: http://127.5.240.193:34451/ I0703 01:40:31.465082 14925 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb I0703 01:40:31.467885 14925 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:40:31.468092 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 14925 I0703 01:40:31.468686 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.5.240.194:0 --local_ip_for_outbound_sockets=127.5.240.194 --webserver_interface=127.5.240.194 --webserver_port=0 --tserver_master_addrs=127.5.240.254:41705 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0703 01:40:31.496050 15054 heartbeater.cc:346] Connected to a master server at 127.5.240.254:41705 I0703 01:40:31.496204 15054 heartbeater.cc:426] Registering TS with master... I0703 01:40:31.497000 15054 heartbeater.cc:475] Master 127.5.240.254:41705 requested a full tablet report, sending... I0703 01:40:31.498392 14891 ts_manager.cc:98] Registered new tserver with Master: 4cbf0a90d2564e67bcd791d9791ca759 (127.5.240.193:41919) I0703 01:40:31.500195 14891 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.193:33804 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:40:31.633409 15056 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:40:31.634408 15056 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:40:31.636438 15056 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:40:31.636559 15056 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.194 I0703 01:40:31.642662 15056 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.194:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.5.240.194 --webserver_port=0 --tserver_master_addrs=127.5.240.254:41705 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.194 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:40:31.648769 15056 tablet_server_main.cc:85] Initializing tablet server... I0703 01:40:31.649881 15056 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 169208us I0703 01:40:31.650830 15056 fs_manager.cc:261] Metadata directory not provided I0703 01:40:31.650892 15056 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:40:31.651121 15056 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:40:31.651183 15056 server_base.cc:433] Attempting to create new FS layout instead I0703 01:40:31.655572 15056 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "b67d8eb488ac4415a88e177011adb79c" format_stamp: "Formatted at 2018-07-03 08:40:31 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:31.656101 15056 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "b67d8eb488ac4415a88e177011adb79c" format_stamp: "Formatted at 2018-07-03 08:40:31 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:31.663825 15056 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.003s sys 0.004s I0703 01:40:31.665001 15056 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:40:31.665371 15056 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:40:31.668447 15056 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0703 01:40:31.668505 15056 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal uuid: "b67d8eb488ac4415a88e177011adb79c" format_stamp: "Formatted at 2018-07-03 08:40:31 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:31.668648 15056 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:40:31.698252 15056 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:40:31.699617 15056 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:40:31.699932 15056 tablet_server_main.cc:90] Starting tablet server... I0703 01:40:31.784247 15056 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.194:60801 I0703 01:40:31.784714 15056 webserver.cc:175] Starting webserver on 127.5.240.194:0 I0703 01:40:31.784751 15056 webserver.cc:186] Document root disabled I0703 01:40:31.785915 15056 webserver.cc:313] Webserver started. Bound to: http://127.5.240.194:46055/ I0703 01:40:31.786432 15056 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb I0703 01:40:31.788342 15056 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:40:31.792171 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 15056 I0703 01:40:31.792768 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.5.240.195:0 --local_ip_for_outbound_sockets=127.5.240.195 --webserver_interface=127.5.240.195 --webserver_port=0 --tserver_master_addrs=127.5.240.254:41705 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0703 01:40:31.799823 15177 heartbeater.cc:346] Connected to a master server at 127.5.240.254:41705 I0703 01:40:31.800002 15177 heartbeater.cc:426] Registering TS with master... I0703 01:40:31.800947 15177 heartbeater.cc:475] Master 127.5.240.254:41705 requested a full tablet report, sending... I0703 01:40:31.802049 14891 ts_manager.cc:98] Registered new tserver with Master: b67d8eb488ac4415a88e177011adb79c (127.5.240.194:60801) I0703 01:40:31.803380 14891 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.194:40689 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:40:31.949807 15180 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:40:31.950814 15180 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:40:31.952877 15180 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:40:31.952966 15180 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.195 I0703 01:40:31.959889 15180 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.195:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.5.240.195 --webserver_port=0 --tserver_master_addrs=127.5.240.254:41705 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.195 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:40:31.966382 15180 tablet_server_main.cc:85] Initializing tablet server... I0703 01:40:31.967491 15180 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 169208us I0703 01:40:31.968410 15180 fs_manager.cc:261] Metadata directory not provided I0703 01:40:31.968464 15180 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:40:31.968701 15180 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:40:31.968758 15180 server_base.cc:433] Attempting to create new FS layout instead I0703 01:40:31.972775 15180 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "218e9285385744f780c07feed8487b2c" format_stamp: "Formatted at 2018-07-03 08:40:31 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:31.973258 15180 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "218e9285385744f780c07feed8487b2c" format_stamp: "Formatted at 2018-07-03 08:40:31 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:31.980581 15180 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.004s sys 0.002s I0703 01:40:31.981750 15180 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:40:31.982102 15180 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:40:31.985744 15180 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0703 01:40:31.985795 15180 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal uuid: "218e9285385744f780c07feed8487b2c" format_stamp: "Formatted at 2018-07-03 08:40:31 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:31.985905 15180 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:40:32.014925 15180 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:40:32.016705 15180 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:40:32.017042 15180 tablet_server_main.cc:90] Starting tablet server... I0703 01:40:32.102121 15180 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.195:54289 I0703 01:40:32.102587 15180 webserver.cc:175] Starting webserver on 127.5.240.195:0 I0703 01:40:32.102627 15180 webserver.cc:186] Document root disabled I0703 01:40:32.103834 15180 webserver.cc:313] Webserver started. Bound to: http://127.5.240.195:47820/ I0703 01:40:32.104357 15180 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDown.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb I0703 01:40:32.105094 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 15180 I0703 01:40:32.107575 15180 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:40:32.119459 15299 heartbeater.cc:346] Connected to a master server at 127.5.240.254:41705 I0703 01:40:32.119598 15299 heartbeater.cc:426] Registering TS with master... I0703 01:40:32.120406 15299 heartbeater.cc:475] Master 127.5.240.254:41705 requested a full tablet report, sending... I0703 01:40:32.121454 14890 ts_manager.cc:98] Registered new tserver with Master: 218e9285385744f780c07feed8487b2c (127.5.240.195:54289) I0703 01:40:32.122413 14890 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.195:50879 I0703 01:40:32.123085 6083 external_mini_cluster.cc:490] 3 TS(s) registered with all masters I0703 01:40:32.133067 14891 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.5.240.254:46426: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } W0703 01:40:32.137759 14891 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended). I0703 01:40:32.156971 15114 tablet_service.cc:759] Processing CreateTablet for tablet 0774fd0e7d654eacb8328db7fb72970e (table=TestTable [id=3e8191d657ae49b4ae05bd7df6b35635]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:40:32.157837 15114 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 0774fd0e7d654eacb8328db7fb72970e. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:40:32.161586 15236 tablet_service.cc:759] Processing CreateTablet for tablet 0774fd0e7d654eacb8328db7fb72970e (table=TestTable [id=3e8191d657ae49b4ae05bd7df6b35635]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:40:32.162245 15236 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 0774fd0e7d654eacb8328db7fb72970e. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:40:32.163333 14991 tablet_service.cc:759] Processing CreateTablet for tablet 0774fd0e7d654eacb8328db7fb72970e (table=TestTable [id=3e8191d657ae49b4ae05bd7df6b35635]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:40:32.164059 14991 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 0774fd0e7d654eacb8328db7fb72970e. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:40:32.172024 15236 ts_tablet_manager.cc:1173] T 0774fd0e7d654eacb8328db7fb72970e P 218e9285385744f780c07feed8487b2c: Registered tablet (data state: TABLET_DATA_READY) I0703 01:40:32.172024 15114 ts_tablet_manager.cc:1173] T 0774fd0e7d654eacb8328db7fb72970e P b67d8eb488ac4415a88e177011adb79c: Registered tablet (data state: TABLET_DATA_READY) I0703 01:40:32.172919 15318 ts_tablet_manager.cc:1028] T 0774fd0e7d654eacb8328db7fb72970e P 218e9285385744f780c07feed8487b2c: Bootstrapping tablet I0703 01:40:32.173493 15318 tablet_bootstrap.cc:427] T 0774fd0e7d654eacb8328db7fb72970e P 218e9285385744f780c07feed8487b2c: Bootstrap starting. I0703 01:40:32.176540 15318 tablet_bootstrap.cc:578] T 0774fd0e7d654eacb8328db7fb72970e P 218e9285385744f780c07feed8487b2c: No blocks or log segments found. Creating new log. I0703 01:40:32.177323 15318 log.cc:525] T 0774fd0e7d654eacb8328db7fb72970e P 218e9285385744f780c07feed8487b2c: Log is configured to *not* fsync() on all Append() calls I0703 01:40:32.177742 14991 ts_tablet_manager.cc:1173] T 0774fd0e7d654eacb8328db7fb72970e P 4cbf0a90d2564e67bcd791d9791ca759: Registered tablet (data state: TABLET_DATA_READY) I0703 01:40:32.178252 15319 ts_tablet_manager.cc:1028] T 0774fd0e7d654eacb8328db7fb72970e P b67d8eb488ac4415a88e177011adb79c: Bootstrapping tablet I0703 01:40:32.178556 15319 tablet_bootstrap.cc:427] T 0774fd0e7d654eacb8328db7fb72970e P b67d8eb488ac4415a88e177011adb79c: Bootstrap starting. I0703 01:40:32.178746 15321 ts_tablet_manager.cc:1028] T 0774fd0e7d654eacb8328db7fb72970e P 4cbf0a90d2564e67bcd791d9791ca759: Bootstrapping tablet I0703 01:40:32.179805 15318 tablet_bootstrap.cc:427] T 0774fd0e7d654eacb8328db7fb72970e P 218e9285385744f780c07feed8487b2c: No bootstrap required, opened a new log I0703 01:40:32.180003 15318 ts_tablet_manager.cc:1045] T 0774fd0e7d654eacb8328db7fb72970e P 218e9285385744f780c07feed8487b2c: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.002s I0703 01:40:32.181252 15319 tablet_bootstrap.cc:578] T 0774fd0e7d654eacb8328db7fb72970e P b67d8eb488ac4415a88e177011adb79c: No blocks or log segments found. Creating new log. I0703 01:40:32.179515 15321 tablet_bootstrap.cc:427] T 0774fd0e7d654eacb8328db7fb72970e P 4cbf0a90d2564e67bcd791d9791ca759: Bootstrap starting. I0703 01:40:32.181982 15319 log.cc:525] T 0774fd0e7d654eacb8328db7fb72970e P b67d8eb488ac4415a88e177011adb79c: Log is configured to *not* fsync() on all Append() calls I0703 01:40:32.184176 15319 tablet_bootstrap.cc:427] T 0774fd0e7d654eacb8328db7fb72970e P b67d8eb488ac4415a88e177011adb79c: No bootstrap required, opened a new log I0703 01:40:32.184342 15319 ts_tablet_manager.cc:1045] T 0774fd0e7d654eacb8328db7fb72970e P b67d8eb488ac4415a88e177011adb79c: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.002s I0703 01:40:32.184515 15321 tablet_bootstrap.cc:578] T 0774fd0e7d654eacb8328db7fb72970e P 4cbf0a90d2564e67bcd791d9791ca759: No blocks or log segments found. Creating new log. I0703 01:40:32.185411 15321 log.cc:525] T 0774fd0e7d654eacb8328db7fb72970e P 4cbf0a90d2564e67bcd791d9791ca759: Log is configured to *not* fsync() on all Append() calls I0703 01:40:32.187022 15321 tablet_bootstrap.cc:427] T 0774fd0e7d654eacb8328db7fb72970e P 4cbf0a90d2564e67bcd791d9791ca759: No bootstrap required, opened a new log I0703 01:40:32.187218 15321 ts_tablet_manager.cc:1045] T 0774fd0e7d654eacb8328db7fb72970e P 4cbf0a90d2564e67bcd791d9791ca759: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.001s I0703 01:40:32.188290 15318 raft_consensus.cc:325] T 0774fd0e7d654eacb8328db7fb72970e P 218e9285385744f780c07feed8487b2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67d8eb488ac4415a88e177011adb79c" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 60801 } } peers { permanent_uuid: "218e9285385744f780c07feed8487b2c" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 54289 } } peers { permanent_uuid: "4cbf0a90d2564e67bcd791d9791ca759" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 41919 } } I0703 01:40:32.188714 15318 raft_consensus.cc:351] T 0774fd0e7d654eacb8328db7fb72970e P 218e9285385744f780c07feed8487b2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:40:32.188791 15318 raft_consensus.cc:628] T 0774fd0e7d654eacb8328db7fb72970e P 218e9285385744f780c07feed8487b2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 218e9285385744f780c07feed8487b2c, State: Initialized, Role: FOLLOWER I0703 01:40:32.189400 15318 consensus_queue.cc:227] T 0774fd0e7d654eacb8328db7fb72970e P 218e9285385744f780c07feed8487b2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67d8eb488ac4415a88e177011adb79c" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 60801 } } peers { permanent_uuid: "218e9285385744f780c07feed8487b2c" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 54289 } } peers { permanent_uuid: "4cbf0a90d2564e67bcd791d9791ca759" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 41919 } } I0703 01:40:32.190896 15319 raft_consensus.cc:325] T 0774fd0e7d654eacb8328db7fb72970e P b67d8eb488ac4415a88e177011adb79c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67d8eb488ac4415a88e177011adb79c" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 60801 } } peers { permanent_uuid: "218e9285385744f780c07feed8487b2c" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 54289 } } peers { permanent_uuid: "4cbf0a90d2564e67bcd791d9791ca759" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 41919 } } I0703 01:40:32.191298 15299 heartbeater.cc:467] Master 127.5.240.254:41705 was elected leader, sending a full tablet report... I0703 01:40:32.191375 15319 raft_consensus.cc:351] T 0774fd0e7d654eacb8328db7fb72970e P b67d8eb488ac4415a88e177011adb79c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:40:32.191443 15319 raft_consensus.cc:628] T 0774fd0e7d654eacb8328db7fb72970e P b67d8eb488ac4415a88e177011adb79c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b67d8eb488ac4415a88e177011adb79c, State: Initialized, Role: FOLLOWER I0703 01:40:32.192065 15319 consensus_queue.cc:227] T 0774fd0e7d654eacb8328db7fb72970e P b67d8eb488ac4415a88e177011adb79c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67d8eb488ac4415a88e177011adb79c" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 60801 } } peers { permanent_uuid: "218e9285385744f780c07feed8487b2c" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 54289 } } peers { permanent_uuid: "4cbf0a90d2564e67bcd791d9791ca759" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 41919 } } I0703 01:40:32.192894 15177 heartbeater.cc:467] Master 127.5.240.254:41705 was elected leader, sending a full tablet report... I0703 01:40:32.194264 15319 ts_tablet_manager.cc:1073] T 0774fd0e7d654eacb8328db7fb72970e P b67d8eb488ac4415a88e177011adb79c: Time spent starting tablet: real 0.010s user 0.006s sys 0.002s I0703 01:40:32.194449 15318 ts_tablet_manager.cc:1073] T 0774fd0e7d654eacb8328db7fb72970e P 218e9285385744f780c07feed8487b2c: Time spent starting tablet: real 0.014s user 0.007s sys 0.005s I0703 01:40:32.195268 15321 raft_consensus.cc:325] T 0774fd0e7d654eacb8328db7fb72970e P 4cbf0a90d2564e67bcd791d9791ca759 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67d8eb488ac4415a88e177011adb79c" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 60801 } } peers { permanent_uuid: "218e9285385744f780c07feed8487b2c" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 54289 } } peers { permanent_uuid: "4cbf0a90d2564e67bcd791d9791ca759" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 41919 } } I0703 01:40:32.195536 15321 raft_consensus.cc:351] T 0774fd0e7d654eacb8328db7fb72970e P 4cbf0a90d2564e67bcd791d9791ca759 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:40:32.195622 15321 raft_consensus.cc:628] T 0774fd0e7d654eacb8328db7fb72970e P 4cbf0a90d2564e67bcd791d9791ca759 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cbf0a90d2564e67bcd791d9791ca759, State: Initialized, Role: FOLLOWER I0703 01:40:32.196050 15321 consensus_queue.cc:227] T 0774fd0e7d654eacb8328db7fb72970e P 4cbf0a90d2564e67bcd791d9791ca759 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67d8eb488ac4415a88e177011adb79c" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 60801 } } peers { permanent_uuid: "218e9285385744f780c07feed8487b2c" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 54289 } } peers { permanent_uuid: "4cbf0a90d2564e67bcd791d9791ca759" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 41919 } } I0703 01:40:32.197710 15054 heartbeater.cc:467] Master 127.5.240.254:41705 was elected leader, sending a full tablet report... I0703 01:40:32.198395 15321 ts_tablet_manager.cc:1073] T 0774fd0e7d654eacb8328db7fb72970e P 4cbf0a90d2564e67bcd791d9791ca759: Time spent starting tablet: real 0.011s user 0.010s sys 0.001s W0703 01:40:32.218467 15055 tablet.cc:1678] T 0774fd0e7d654eacb8328db7fb72970e P 4cbf0a90d2564e67bcd791d9791ca759: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:40:32.225062 15324 raft_consensus.cc:457] T 0774fd0e7d654eacb8328db7fb72970e P 218e9285385744f780c07feed8487b2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0703 01:40:32.225569 15324 raft_consensus.cc:2730] T 0774fd0e7d654eacb8328db7fb72970e P 218e9285385744f780c07feed8487b2c: Snoozing failure detection for 1.955s (starting election) I0703 01:40:32.225690 15324 raft_consensus.cc:479] T 0774fd0e7d654eacb8328db7fb72970e P 218e9285385744f780c07feed8487b2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67d8eb488ac4415a88e177011adb79c" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 60801 } } peers { permanent_uuid: "218e9285385744f780c07feed8487b2c" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 54289 } } peers { permanent_uuid: "4cbf0a90d2564e67bcd791d9791ca759" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 41919 } } I0703 01:40:32.226817 15324 leader_election.cc:243] T 0774fd0e7d654eacb8328db7fb72970e P 218e9285385744f780c07feed8487b2c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer b67d8eb488ac4415a88e177011adb79c (127.5.240.194:60801) I0703 01:40:32.227130 15324 leader_election.cc:243] T 0774fd0e7d654eacb8328db7fb72970e P 218e9285385744f780c07feed8487b2c [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 4cbf0a90d2564e67bcd791d9791ca759 (127.5.240.193:41919) I0703 01:40:32.233045 15011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "0774fd0e7d654eacb8328db7fb72970e" candidate_uuid: "218e9285385744f780c07feed8487b2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cbf0a90d2564e67bcd791d9791ca759" is_pre_election: true I0703 01:40:32.233279 15011 raft_consensus.cc:2730] T 0774fd0e7d654eacb8328db7fb72970e P 4cbf0a90d2564e67bcd791d9791ca759: Snoozing failure detection for 2.002s (vote granted) I0703 01:40:32.233366 15011 raft_consensus.cc:2243] T 0774fd0e7d654eacb8328db7fb72970e P 4cbf0a90d2564e67bcd791d9791ca759 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 218e9285385744f780c07feed8487b2c in term 0. I0703 01:40:32.233945 15190 leader_election.cc:379] T 0774fd0e7d654eacb8328db7fb72970e P 218e9285385744f780c07feed8487b2c [CANDIDATE]: Term 1 pre-election: Vote granted by peer 4cbf0a90d2564e67bcd791d9791ca759 (127.5.240.193:41919) I0703 01:40:32.234011 15190 leader_election.cc:270] T 0774fd0e7d654eacb8328db7fb72970e P 218e9285385744f780c07feed8487b2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0703 01:40:32.234263 15324 raft_consensus.cc:2730] T 0774fd0e7d654eacb8328db7fb72970e P 218e9285385744f780c07feed8487b2c: Snoozing failure detection for 1.969s (election complete) I0703 01:40:32.234364 15324 raft_consensus.cc:2537] T 0774fd0e7d654eacb8328db7fb72970e P 218e9285385744f780c07feed8487b2c [term 0 FOLLOWER]: Leader pre-election won for term 1 I0703 01:40:32.234478 15324 raft_consensus.cc:457] T 0774fd0e7d654eacb8328db7fb72970e P 218e9285385744f780c07feed8487b2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0703 01:40:32.234681 15324 raft_consensus.cc:2730] T 0774fd0e7d654eacb8328db7fb72970e P 218e9285385744f780c07feed8487b2c: Snoozing failure detection for 1.730s (starting election) I0703 01:40:32.234777 15324 raft_consensus.cc:2782] T 0774fd0e7d654eacb8328db7fb72970e P 218e9285385744f780c07feed8487b2c [term 0 FOLLOWER]: Advancing to term 1 I0703 01:40:32.235374 15134 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "0774fd0e7d654eacb8328db7fb72970e" candidate_uuid: "218e9285385744f780c07feed8487b2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b67d8eb488ac4415a88e177011adb79c" is_pre_election: true I0703 01:40:32.236085 15134 raft_consensus.cc:2730] T 0774fd0e7d654eacb8328db7fb72970e P b67d8eb488ac4415a88e177011adb79c: Snoozing failure detection for 2.025s (vote granted) I0703 01:40:32.236183 15134 raft_consensus.cc:2243] T 0774fd0e7d654eacb8328db7fb72970e P b67d8eb488ac4415a88e177011adb79c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 218e9285385744f780c07feed8487b2c in term 0. I0703 01:40:32.236779 15188 leader_election.cc:379] T 0774fd0e7d654eacb8328db7fb72970e P 218e9285385744f780c07feed8487b2c [CANDIDATE]: Term 1 pre-election: Vote granted by peer b67d8eb488ac4415a88e177011adb79c (127.5.240.194:60801) I0703 01:40:32.237134 15324 raft_consensus.cc:479] T 0774fd0e7d654eacb8328db7fb72970e P 218e9285385744f780c07feed8487b2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67d8eb488ac4415a88e177011adb79c" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 60801 } } peers { permanent_uuid: "218e9285385744f780c07feed8487b2c" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 54289 } } peers { permanent_uuid: "4cbf0a90d2564e67bcd791d9791ca759" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 41919 } } I0703 01:40:32.237537 15324 leader_election.cc:243] T 0774fd0e7d654eacb8328db7fb72970e P 218e9285385744f780c07feed8487b2c [CANDIDATE]: Term 1 election: Requesting vote from peer b67d8eb488ac4415a88e177011adb79c (127.5.240.194:60801) I0703 01:40:32.237720 15324 leader_election.cc:243] T 0774fd0e7d654eacb8328db7fb72970e P 218e9285385744f780c07feed8487b2c [CANDIDATE]: Term 1 election: Requesting vote from peer 4cbf0a90d2564e67bcd791d9791ca759 (127.5.240.193:41919) I0703 01:40:32.238245 15134 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "0774fd0e7d654eacb8328db7fb72970e" candidate_uuid: "218e9285385744f780c07feed8487b2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b67d8eb488ac4415a88e177011adb79c" I0703 01:40:32.238354 15134 raft_consensus.cc:2782] T 0774fd0e7d654eacb8328db7fb72970e P b67d8eb488ac4415a88e177011adb79c [term 0 FOLLOWER]: Advancing to term 1 I0703 01:40:32.238392 15011 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "0774fd0e7d654eacb8328db7fb72970e" candidate_uuid: "218e9285385744f780c07feed8487b2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cbf0a90d2564e67bcd791d9791ca759" I0703 01:40:32.238442 15134 raft_consensus.cc:2730] T 0774fd0e7d654eacb8328db7fb72970e P b67d8eb488ac4415a88e177011adb79c: Snoozing failure detection for 1.810s (vote granted) I0703 01:40:32.238515 15011 raft_consensus.cc:2782] T 0774fd0e7d654eacb8328db7fb72970e P 4cbf0a90d2564e67bcd791d9791ca759 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:40:32.238610 15011 raft_consensus.cc:2730] T 0774fd0e7d654eacb8328db7fb72970e P 4cbf0a90d2564e67bcd791d9791ca759: Snoozing failure detection for 1.693s (vote granted) I0703 01:40:32.239812 15134 raft_consensus.cc:2243] T 0774fd0e7d654eacb8328db7fb72970e P b67d8eb488ac4415a88e177011adb79c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 218e9285385744f780c07feed8487b2c in term 1. I0703 01:40:32.239964 15011 raft_consensus.cc:2243] T 0774fd0e7d654eacb8328db7fb72970e P 4cbf0a90d2564e67bcd791d9791ca759 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 218e9285385744f780c07feed8487b2c in term 1. I0703 01:40:32.240226 15188 leader_election.cc:379] T 0774fd0e7d654eacb8328db7fb72970e P 218e9285385744f780c07feed8487b2c [CANDIDATE]: Term 1 election: Vote granted by peer b67d8eb488ac4415a88e177011adb79c (127.5.240.194:60801) I0703 01:40:32.240326 15188 leader_election.cc:270] T 0774fd0e7d654eacb8328db7fb72970e P 218e9285385744f780c07feed8487b2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:40:32.240425 15190 leader_election.cc:379] T 0774fd0e7d654eacb8328db7fb72970e P 218e9285385744f780c07feed8487b2c [CANDIDATE]: Term 1 election: Vote granted by peer 4cbf0a90d2564e67bcd791d9791ca759 (127.5.240.193:41919) I0703 01:40:32.240531 15324 raft_consensus.cc:2730] T 0774fd0e7d654eacb8328db7fb72970e P 218e9285385744f780c07feed8487b2c: Snoozing failure detection for 1.680s (election complete) I0703 01:40:32.240634 15324 raft_consensus.cc:2537] T 0774fd0e7d654eacb8328db7fb72970e P 218e9285385744f780c07feed8487b2c [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:40:32.241528 15324 raft_consensus.cc:593] T 0774fd0e7d654eacb8328db7fb72970e P 218e9285385744f780c07feed8487b2c [term 1 LEADER]: Becoming Leader. State: Replica: 218e9285385744f780c07feed8487b2c, State: Running, Role: LEADER I0703 01:40:32.241997 15324 consensus_queue.cc:204] T 0774fd0e7d654eacb8328db7fb72970e P 218e9285385744f780c07feed8487b2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67d8eb488ac4415a88e177011adb79c" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 60801 } } peers { permanent_uuid: "218e9285385744f780c07feed8487b2c" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 54289 } } peers { permanent_uuid: "4cbf0a90d2564e67bcd791d9791ca759" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 41919 } } I0703 01:40:32.245771 14891 catalog_manager.cc:3798] T 0774fd0e7d654eacb8328db7fb72970e P 218e9285385744f780c07feed8487b2c reported cstate change: term changed from 0 to 1, leader changed from to 218e9285385744f780c07feed8487b2c (127.5.240.195). New cstate: current_term: 1 leader_uuid: "218e9285385744f780c07feed8487b2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67d8eb488ac4415a88e177011adb79c" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 60801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "218e9285385744f780c07feed8487b2c" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 54289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cbf0a90d2564e67bcd791d9791ca759" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 41919 } health_report { overall_health: UNKNOWN } } } I0703 01:40:32.258565 6083 external_mini_cluster.cc:490] 3 TS(s) registered with all masters I0703 01:40:32.259632 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 4cbf0a90d2564e67bcd791d9791ca759 to finish bootstrapping I0703 01:40:32.264075 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver b67d8eb488ac4415a88e177011adb79c to finish bootstrapping I0703 01:40:32.269183 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 218e9285385744f780c07feed8487b2c to finish bootstrapping W0703 01:40:32.288578 15178 tablet.cc:1678] T 0774fd0e7d654eacb8328db7fb72970e P b67d8eb488ac4415a88e177011adb79c: Can't schedule compaction. Clean time has not been advanced past its initial value. W0703 01:40:32.358160 15300 tablet.cc:1678] T 0774fd0e7d654eacb8328db7fb72970e P 218e9285385744f780c07feed8487b2c: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:40:32.553088 15256 tablet_service.cc:1140] Received LeaderStepDown RPC: tablet_id: "0774fd0e7d654eacb8328db7fb72970e" dest_uuid: "218e9285385744f780c07feed8487b2c" from {username='jenkins'} at 127.5.240.195:58495 I0703 01:40:32.553298 15256 raft_consensus.cc:552] T 0774fd0e7d654eacb8328db7fb72970e P 218e9285385744f780c07feed8487b2c [term 1 LEADER]: Received request to step down I0703 01:40:32.553354 15256 raft_consensus.cc:628] T 0774fd0e7d654eacb8328db7fb72970e P 218e9285385744f780c07feed8487b2c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 218e9285385744f780c07feed8487b2c, State: Running, Role: LEADER I0703 01:40:32.553603 15256 consensus_queue.cc:227] T 0774fd0e7d654eacb8328db7fb72970e P 218e9285385744f780c07feed8487b2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67d8eb488ac4415a88e177011adb79c" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 60801 } } peers { permanent_uuid: "218e9285385744f780c07feed8487b2c" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 54289 } } peers { permanent_uuid: "4cbf0a90d2564e67bcd791d9791ca759" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 41919 } } I0703 01:40:32.553782 15256 consensus_peers.cc:455] T 0774fd0e7d654eacb8328db7fb72970e P 218e9285385744f780c07feed8487b2c -> Peer 4cbf0a90d2564e67bcd791d9791ca759 (127.5.240.193:41919): Closing peer: 4cbf0a90d2564e67bcd791d9791ca759 I0703 01:40:32.553853 15256 consensus_peers.cc:455] T 0774fd0e7d654eacb8328db7fb72970e P 218e9285385744f780c07feed8487b2c -> Peer b67d8eb488ac4415a88e177011adb79c (127.5.240.194:60801): Closing peer: b67d8eb488ac4415a88e177011adb79c I0703 01:40:32.554028 15256 raft_consensus.cc:2730] T 0774fd0e7d654eacb8328db7fb72970e P 218e9285385744f780c07feed8487b2c: Snoozing failure detection for 3.000s (explicit stepdown request) I0703 01:40:33.933750 15344 raft_consensus.cc:457] T 0774fd0e7d654eacb8328db7fb72970e P 4cbf0a90d2564e67bcd791d9791ca759 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0703 01:40:33.933828 15344 raft_consensus.cc:2730] T 0774fd0e7d654eacb8328db7fb72970e P 4cbf0a90d2564e67bcd791d9791ca759: Snoozing failure detection for 1.751s (starting election) I0703 01:40:33.933904 15344 raft_consensus.cc:479] T 0774fd0e7d654eacb8328db7fb72970e P 4cbf0a90d2564e67bcd791d9791ca759 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67d8eb488ac4415a88e177011adb79c" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 60801 } } peers { permanent_uuid: "218e9285385744f780c07feed8487b2c" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 54289 } } peers { permanent_uuid: "4cbf0a90d2564e67bcd791d9791ca759" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 41919 } } I0703 01:40:33.934605 15344 leader_election.cc:243] T 0774fd0e7d654eacb8328db7fb72970e P 4cbf0a90d2564e67bcd791d9791ca759 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer b67d8eb488ac4415a88e177011adb79c (127.5.240.194:60801) I0703 01:40:33.934897 15344 leader_election.cc:243] T 0774fd0e7d654eacb8328db7fb72970e P 4cbf0a90d2564e67bcd791d9791ca759 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 218e9285385744f780c07feed8487b2c (127.5.240.195:54289) I0703 01:40:33.940758 15134 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "0774fd0e7d654eacb8328db7fb72970e" candidate_uuid: "4cbf0a90d2564e67bcd791d9791ca759" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b67d8eb488ac4415a88e177011adb79c" is_pre_election: true I0703 01:40:33.940943 15134 raft_consensus.cc:2730] T 0774fd0e7d654eacb8328db7fb72970e P b67d8eb488ac4415a88e177011adb79c: Snoozing failure detection for 1.760s (vote granted) I0703 01:40:33.941002 15134 raft_consensus.cc:2243] T 0774fd0e7d654eacb8328db7fb72970e P b67d8eb488ac4415a88e177011adb79c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cbf0a90d2564e67bcd791d9791ca759 in term 1. I0703 01:40:33.941305 15256 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "0774fd0e7d654eacb8328db7fb72970e" candidate_uuid: "4cbf0a90d2564e67bcd791d9791ca759" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "218e9285385744f780c07feed8487b2c" is_pre_election: true I0703 01:40:33.941537 14943 leader_election.cc:379] T 0774fd0e7d654eacb8328db7fb72970e P 4cbf0a90d2564e67bcd791d9791ca759 [CANDIDATE]: Term 2 pre-election: Vote granted by peer b67d8eb488ac4415a88e177011adb79c (127.5.240.194:60801) I0703 01:40:33.941622 15256 raft_consensus.cc:2190] T 0774fd0e7d654eacb8328db7fb72970e P 218e9285385744f780c07feed8487b2c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cbf0a90d2564e67bcd791d9791ca759 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0. I0703 01:40:33.941639 14943 leader_election.cc:270] T 0774fd0e7d654eacb8328db7fb72970e P 4cbf0a90d2564e67bcd791d9791ca759 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. I0703 01:40:33.941910 15344 raft_consensus.cc:2730] T 0774fd0e7d654eacb8328db7fb72970e P 4cbf0a90d2564e67bcd791d9791ca759: Snoozing failure detection for 1.862s (election complete) I0703 01:40:33.941967 15344 raft_consensus.cc:2537] T 0774fd0e7d654eacb8328db7fb72970e P 4cbf0a90d2564e67bcd791d9791ca759 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0703 01:40:33.942018 15344 raft_consensus.cc:457] T 0774fd0e7d654eacb8328db7fb72970e P 4cbf0a90d2564e67bcd791d9791ca759 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0703 01:40:33.942080 15344 raft_consensus.cc:2730] T 0774fd0e7d654eacb8328db7fb72970e P 4cbf0a90d2564e67bcd791d9791ca759: Snoozing failure detection for 1.717s (starting election) I0703 01:40:33.942196 15344 raft_consensus.cc:2782] T 0774fd0e7d654eacb8328db7fb72970e P 4cbf0a90d2564e67bcd791d9791ca759 [term 1 FOLLOWER]: Advancing to term 2 I0703 01:40:33.942481 14944 leader_election.cc:393] T 0774fd0e7d654eacb8328db7fb72970e P 4cbf0a90d2564e67bcd791d9791ca759 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 218e9285385744f780c07feed8487b2c (127.5.240.195:54289). Message: Invalid argument: T 0774fd0e7d654eacb8328db7fb72970e P 218e9285385744f780c07feed8487b2c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4cbf0a90d2564e67bcd791d9791ca759 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0. I0703 01:40:33.943929 15344 raft_consensus.cc:479] T 0774fd0e7d654eacb8328db7fb72970e P 4cbf0a90d2564e67bcd791d9791ca759 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67d8eb488ac4415a88e177011adb79c" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 60801 } } peers { permanent_uuid: "218e9285385744f780c07feed8487b2c" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 54289 } } peers { permanent_uuid: "4cbf0a90d2564e67bcd791d9791ca759" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 41919 } } I0703 01:40:33.944267 15344 leader_election.cc:243] T 0774fd0e7d654eacb8328db7fb72970e P 4cbf0a90d2564e67bcd791d9791ca759 [CANDIDATE]: Term 2 election: Requesting vote from peer b67d8eb488ac4415a88e177011adb79c (127.5.240.194:60801) I0703 01:40:33.944425 15344 leader_election.cc:243] T 0774fd0e7d654eacb8328db7fb72970e P 4cbf0a90d2564e67bcd791d9791ca759 [CANDIDATE]: Term 2 election: Requesting vote from peer 218e9285385744f780c07feed8487b2c (127.5.240.195:54289) I0703 01:40:33.944916 15134 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "0774fd0e7d654eacb8328db7fb72970e" candidate_uuid: "4cbf0a90d2564e67bcd791d9791ca759" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b67d8eb488ac4415a88e177011adb79c" I0703 01:40:33.945030 15134 raft_consensus.cc:2782] T 0774fd0e7d654eacb8328db7fb72970e P b67d8eb488ac4415a88e177011adb79c [term 1 FOLLOWER]: Advancing to term 2 I0703 01:40:33.945047 15256 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "0774fd0e7d654eacb8328db7fb72970e" candidate_uuid: "4cbf0a90d2564e67bcd791d9791ca759" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "218e9285385744f780c07feed8487b2c" I0703 01:40:33.945134 15134 raft_consensus.cc:2730] T 0774fd0e7d654eacb8328db7fb72970e P b67d8eb488ac4415a88e177011adb79c: Snoozing failure detection for 1.984s (vote granted) I0703 01:40:33.945152 15256 raft_consensus.cc:2782] T 0774fd0e7d654eacb8328db7fb72970e P 218e9285385744f780c07feed8487b2c [term 1 FOLLOWER]: Advancing to term 2 I0703 01:40:33.946419 15134 raft_consensus.cc:2243] T 0774fd0e7d654eacb8328db7fb72970e P b67d8eb488ac4415a88e177011adb79c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cbf0a90d2564e67bcd791d9791ca759 in term 2. I0703 01:40:33.946501 15256 raft_consensus.cc:2190] T 0774fd0e7d654eacb8328db7fb72970e P 218e9285385744f780c07feed8487b2c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cbf0a90d2564e67bcd791d9791ca759 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0. I0703 01:40:33.946894 14943 leader_election.cc:379] T 0774fd0e7d654eacb8328db7fb72970e P 4cbf0a90d2564e67bcd791d9791ca759 [CANDIDATE]: Term 2 election: Vote granted by peer b67d8eb488ac4415a88e177011adb79c (127.5.240.194:60801) I0703 01:40:33.946975 14944 leader_election.cc:393] T 0774fd0e7d654eacb8328db7fb72970e P 4cbf0a90d2564e67bcd791d9791ca759 [CANDIDATE]: Term 2 election: Vote denied by peer 218e9285385744f780c07feed8487b2c (127.5.240.195:54289). Message: Invalid argument: T 0774fd0e7d654eacb8328db7fb72970e P 218e9285385744f780c07feed8487b2c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4cbf0a90d2564e67bcd791d9791ca759 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0. I0703 01:40:33.947046 14944 leader_election.cc:270] T 0774fd0e7d654eacb8328db7fb72970e P 4cbf0a90d2564e67bcd791d9791ca759 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0703 01:40:33.947229 15344 raft_consensus.cc:2730] T 0774fd0e7d654eacb8328db7fb72970e P 4cbf0a90d2564e67bcd791d9791ca759: Snoozing failure detection for 1.798s (election complete) I0703 01:40:33.947316 15344 raft_consensus.cc:2537] T 0774fd0e7d654eacb8328db7fb72970e P 4cbf0a90d2564e67bcd791d9791ca759 [term 2 FOLLOWER]: Leader election won for term 2 I0703 01:40:33.947731 15344 raft_consensus.cc:593] T 0774fd0e7d654eacb8328db7fb72970e P 4cbf0a90d2564e67bcd791d9791ca759 [term 2 LEADER]: Becoming Leader. State: Replica: 4cbf0a90d2564e67bcd791d9791ca759, State: Running, Role: LEADER I0703 01:40:33.948107 15344 consensus_queue.cc:204] T 0774fd0e7d654eacb8328db7fb72970e P 4cbf0a90d2564e67bcd791d9791ca759 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67d8eb488ac4415a88e177011adb79c" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 60801 } } peers { permanent_uuid: "218e9285385744f780c07feed8487b2c" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 54289 } } peers { permanent_uuid: "4cbf0a90d2564e67bcd791d9791ca759" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 41919 } } I0703 01:40:33.951865 14891 catalog_manager.cc:3798] T 0774fd0e7d654eacb8328db7fb72970e P 4cbf0a90d2564e67bcd791d9791ca759 reported cstate change: term changed from 1 to 2, leader changed from 218e9285385744f780c07feed8487b2c (127.5.240.195) to 4cbf0a90d2564e67bcd791d9791ca759 (127.5.240.193). New cstate: current_term: 2 leader_uuid: "4cbf0a90d2564e67bcd791d9791ca759" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b67d8eb488ac4415a88e177011adb79c" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 60801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "218e9285385744f780c07feed8487b2c" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 54289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cbf0a90d2564e67bcd791d9791ca759" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 41919 } health_report { overall_health: HEALTHY } } } I0703 01:40:34.418674 15134 raft_consensus.cc:1081] T 0774fd0e7d654eacb8328db7fb72970e P b67d8eb488ac4415a88e177011adb79c [term 2 FOLLOWER]: Refusing update from remote peer 4cbf0a90d2564e67bcd791d9791ca759: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch) I0703 01:40:34.419435 15344 consensus_queue.cc:962] T 0774fd0e7d654eacb8328db7fb72970e P 4cbf0a90d2564e67bcd791d9791ca759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b67d8eb488ac4415a88e177011adb79c" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 60801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0703 01:40:34.425896 15256 raft_consensus.cc:1081] T 0774fd0e7d654eacb8328db7fb72970e P 218e9285385744f780c07feed8487b2c [term 2 FOLLOWER]: Refusing update from remote peer 4cbf0a90d2564e67bcd791d9791ca759: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch) I0703 01:40:34.425953 15256 pending_rounds.cc:76] T 0774fd0e7d654eacb8328db7fb72970e P 218e9285385744f780c07feed8487b2c: Aborting all transactions after (but not including) 0 I0703 01:40:34.426000 15256 pending_rounds.cc:97] T 0774fd0e7d654eacb8328db7fb72970e P 218e9285385744f780c07feed8487b2c: Aborting uncommitted NO_OP operation due to leader change: 1.1 I0703 01:40:34.426095 15256 raft_consensus.cc:2620] T 0774fd0e7d654eacb8328db7fb72970e P 218e9285385744f780c07feed8487b2c [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted by new leader I0703 01:40:34.427502 15344 consensus_queue.cc:962] T 0774fd0e7d654eacb8328db7fb72970e P 4cbf0a90d2564e67bcd791d9791ca759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218e9285385744f780c07feed8487b2c" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 54289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0703 01:40:34.650079 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 14925 I0703 01:40:34.653257 15051 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:40:34.698300 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 14925 W0703 01:40:34.707075 15190 connection.cc:512] client connection to 127.5.240.193:41919 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:41919): Connection reset by peer (error 104) W0703 01:40:34.707094 15066 connection.cc:512] server connection from 127.5.240.193:46371 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:46371): Connection reset by peer (error 104) W0703 01:40:34.707113 14872 connection.cc:512] server connection from 127.5.240.193:33804 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:33804): Connection reset by peer (error 104) W0703 01:40:34.707682 15188 connection.cc:512] server connection from 127.5.240.193:43979 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:43979): Connection reset by peer (error 104) I0703 01:40:34.707765 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 15056 I0703 01:40:34.708504 15174 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:40:34.751313 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 15056 W0703 01:40:34.759296 14872 connection.cc:512] server connection from 127.5.240.194:40689 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:40689): Connection reset by peer (error 104) W0703 01:40:34.759317 15188 connection.cc:512] client connection to 127.5.240.194:60801 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:60801): Connection reset by peer (error 104) I0703 01:40:34.759624 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 15180 I0703 01:40:34.760359 15296 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:40:34.803669 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 15180 W0703 01:40:34.811777 14874 connection.cc:512] server connection from 127.5.240.195:50879 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.195:50879): Connection reset by peer (error 104) I0703 01:40:34.811914 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master pid 14865 I0703 01:40:34.812647 14921 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:40:34.875178 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master with pid 14865 [ OK ] AdminCliTest.TestLeaderStepDown (4094 ms) [ RUN ] AdminCliTest.TestLeaderStepDownWhenNotPresent I0703 01:40:34.896065 6083 test_util.cc:208] Using random seed: 1149685940 I0703 01:40:34.897891 6083 ts_itest-base.cc:113] Starting cluster with: I0703 01:40:34.897946 6083 ts_itest-base.cc:114] -------------- I0703 01:40:34.898000 6083 ts_itest-base.cc:115] 3 tablet servers I0703 01:40:34.898025 6083 ts_itest-base.cc:116] 3 replicas per TS I0703 01:40:34.898064 6083 ts_itest-base.cc:117] -------------- I0703 01:40:34.899233 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:39327 --webserver_interface=127.5.240.254 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --catalog_manager_wait_for_new_tablets_to_elect_leader=false --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:40:35.080183 15369 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:40:35.080411 15369 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:40:35.082667 15369 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0703 01:40:35.082712 15369 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:40:35.082770 15369 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0703 01:40:35.082818 15369 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0703 01:40:35.090420 15369 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --catalog_manager_wait_for_new_tablets_to_elect_leader=false --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:39327 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.5.240.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:40:35.096053 15369 master_main.cc:80] Initializing master server... I0703 01:40:35.097141 15369 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 171208us I0703 01:40:35.098208 15369 fs_manager.cc:261] Metadata directory not provided I0703 01:40:35.098273 15369 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:40:35.098512 15369 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:40:35.098616 15369 server_base.cc:433] Attempting to create new FS layout instead I0703 01:40:35.102774 15369 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/instance: uuid: "2aa149bd0dec4ae3a5c3087c0c7f9f2e" format_stamp: "Formatted at 2018-07-03 08:40:35 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:35.103282 15369 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "2aa149bd0dec4ae3a5c3087c0c7f9f2e" format_stamp: "Formatted at 2018-07-03 08:40:35 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:35.111428 15369 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.001s sys 0.005s I0703 01:40:35.112632 15369 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:40:35.113016 15369 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:40:35.116649 15369 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s I0703 01:40:35.116708 15369 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal uuid: "2aa149bd0dec4ae3a5c3087c0c7f9f2e" format_stamp: "Formatted at 2018-07-03 08:40:35 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:35.116817 15369 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:40:35.138238 15369 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:40:35.139698 15369 master_main.cc:83] Starting Master server... I0703 01:40:35.175954 15369 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.254:39327 I0703 01:40:35.176409 15369 webserver.cc:175] Starting webserver on 127.5.240.254:0 I0703 01:40:35.176456 15369 webserver.cc:186] Document root disabled I0703 01:40:35.177667 15369 webserver.cc:313] Webserver started. Bound to: http://127.5.240.254:38111/ I0703 01:40:35.178266 15369 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb I0703 01:40:35.179392 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master as pid 15369 I0703 01:40:35.180030 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.5.240.193:0 --local_ip_for_outbound_sockets=127.5.240.193 --webserver_interface=127.5.240.193 --webserver_port=0 --tserver_master_addrs=127.5.240.254:39327 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --enable_leader_failure_detection=false with env {} I0703 01:40:35.181924 15428 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:40:35.193595 15428 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 2aa149bd0dec4ae3a5c3087c0c7f9f2e: Bootstrap starting. I0703 01:40:35.195586 15428 tablet_bootstrap.cc:578] T 00000000000000000000000000000000 P 2aa149bd0dec4ae3a5c3087c0c7f9f2e: No blocks or log segments found. Creating new log. I0703 01:40:35.196363 15428 log.cc:525] T 00000000000000000000000000000000 P 2aa149bd0dec4ae3a5c3087c0c7f9f2e: Log is configured to *not* fsync() on all Append() calls I0703 01:40:35.198123 15428 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 2aa149bd0dec4ae3a5c3087c0c7f9f2e: No bootstrap required, opened a new log I0703 01:40:35.204300 15428 raft_consensus.cc:325] T 00000000000000000000000000000000 P 2aa149bd0dec4ae3a5c3087c0c7f9f2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aa149bd0dec4ae3a5c3087c0c7f9f2e" member_type: VOTER } I0703 01:40:35.204501 15428 raft_consensus.cc:351] T 00000000000000000000000000000000 P 2aa149bd0dec4ae3a5c3087c0c7f9f2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:40:35.204566 15428 raft_consensus.cc:628] T 00000000000000000000000000000000 P 2aa149bd0dec4ae3a5c3087c0c7f9f2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa149bd0dec4ae3a5c3087c0c7f9f2e, State: Initialized, Role: FOLLOWER I0703 01:40:35.205042 15428 consensus_queue.cc:227] T 00000000000000000000000000000000 P 2aa149bd0dec4ae3a5c3087c0c7f9f2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aa149bd0dec4ae3a5c3087c0c7f9f2e" member_type: VOTER } I0703 01:40:35.205190 15428 raft_consensus.cc:365] T 00000000000000000000000000000000 P 2aa149bd0dec4ae3a5c3087c0c7f9f2e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:40:35.205279 15428 raft_consensus.cc:457] T 00000000000000000000000000000000 P 2aa149bd0dec4ae3a5c3087c0c7f9f2e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:40:35.205366 15428 raft_consensus.cc:2730] T 00000000000000000000000000000000 P 2aa149bd0dec4ae3a5c3087c0c7f9f2e: Snoozing failure detection for 1.933s (starting election) I0703 01:40:35.205457 15428 raft_consensus.cc:2782] T 00000000000000000000000000000000 P 2aa149bd0dec4ae3a5c3087c0c7f9f2e [term 0 FOLLOWER]: Advancing to term 1 I0703 01:40:35.206859 15428 raft_consensus.cc:479] T 00000000000000000000000000000000 P 2aa149bd0dec4ae3a5c3087c0c7f9f2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aa149bd0dec4ae3a5c3087c0c7f9f2e" member_type: VOTER } I0703 01:40:35.207239 15428 leader_election.cc:270] T 00000000000000000000000000000000 P 2aa149bd0dec4ae3a5c3087c0c7f9f2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:40:35.208803 15431 raft_consensus.cc:2730] T 00000000000000000000000000000000 P 2aa149bd0dec4ae3a5c3087c0c7f9f2e: Snoozing failure detection for 1.698s (election complete) I0703 01:40:35.209051 15431 raft_consensus.cc:2537] T 00000000000000000000000000000000 P 2aa149bd0dec4ae3a5c3087c0c7f9f2e [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:40:35.214781 15431 raft_consensus.cc:593] T 00000000000000000000000000000000 P 2aa149bd0dec4ae3a5c3087c0c7f9f2e [term 1 LEADER]: Becoming Leader. State: Replica: 2aa149bd0dec4ae3a5c3087c0c7f9f2e, State: Running, Role: LEADER I0703 01:40:35.215116 15431 consensus_queue.cc:204] T 00000000000000000000000000000000 P 2aa149bd0dec4ae3a5c3087c0c7f9f2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aa149bd0dec4ae3a5c3087c0c7f9f2e" member_type: VOTER } I0703 01:40:35.217736 15428 sys_catalog.cc:337] T 00000000000000000000000000000000 P 2aa149bd0dec4ae3a5c3087c0c7f9f2e [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "2aa149bd0dec4ae3a5c3087c0c7f9f2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aa149bd0dec4ae3a5c3087c0c7f9f2e" member_type: VOTER } } I0703 01:40:35.218497 15428 sys_catalog.cc:340] T 00000000000000000000000000000000 P 2aa149bd0dec4ae3a5c3087c0c7f9f2e [sys.catalog]: This master's current role is: LEADER I0703 01:40:35.217634 15432 sys_catalog.cc:337] T 00000000000000000000000000000000 P 2aa149bd0dec4ae3a5c3087c0c7f9f2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2aa149bd0dec4ae3a5c3087c0c7f9f2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aa149bd0dec4ae3a5c3087c0c7f9f2e" member_type: VOTER } } I0703 01:40:35.218878 15432 sys_catalog.cc:340] T 00000000000000000000000000000000 P 2aa149bd0dec4ae3a5c3087c0c7f9f2e [sys.catalog]: This master's current role is: LEADER I0703 01:40:35.218413 15433 sys_catalog.cc:337] T 00000000000000000000000000000000 P 2aa149bd0dec4ae3a5c3087c0c7f9f2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2aa149bd0dec4ae3a5c3087c0c7f9f2e. Latest consensus state: current_term: 1 leader_uuid: "2aa149bd0dec4ae3a5c3087c0c7f9f2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aa149bd0dec4ae3a5c3087c0c7f9f2e" member_type: VOTER } } I0703 01:40:35.219915 15433 sys_catalog.cc:340] T 00000000000000000000000000000000 P 2aa149bd0dec4ae3a5c3087c0c7f9f2e [sys.catalog]: This master's current role is: LEADER I0703 01:40:35.221246 15428 sys_catalog.cc:424] T 00000000000000000000000000000000 P 2aa149bd0dec4ae3a5c3087c0c7f9f2e [sys.catalog]: configured and running, proceeding with master startup. I0703 01:40:35.221480 15435 catalog_manager.cc:1006] Loading table and tablet metadata into memory... I0703 01:40:35.222954 15369 master_main.cc:86] Master server successfully started. I0703 01:40:35.228077 15435 catalog_manager.cc:1017] Initializing Kudu internal certificate authority... I0703 01:40:35.255815 15435 catalog_manager.cc:904] Generated new certificate authority record I0703 01:40:35.257882 15435 catalog_manager.cc:1026] Loading token signing keys... I0703 01:40:35.265719 15435 catalog_manager.cc:4102] T 00000000000000000000000000000000 P 2aa149bd0dec4ae3a5c3087c0c7f9f2e: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:40:35.346786 15429 flags.cc:406] Enabled unsafe flag: --enable_leader_failure_detection=false W0703 01:40:35.347810 15429 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:40:35.347904 15429 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:40:35.349952 15429 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:40:35.350061 15429 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.193 I0703 01:40:35.356602 15429 tablet_server_main.cc:78] Tablet server non-default flags: --enable_leader_failure_detection=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.5.240.193 --webserver_port=0 --tserver_master_addrs=127.5.240.254:39327 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.193 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:40:35.363142 15429 tablet_server_main.cc:85] Initializing tablet server... I0703 01:40:35.364677 15429 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 171208us I0703 01:40:35.365237 15429 fs_manager.cc:261] Metadata directory not provided I0703 01:40:35.365305 15429 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:40:35.365936 15429 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:40:35.366005 15429 server_base.cc:433] Attempting to create new FS layout instead I0703 01:40:35.370146 15429 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "293b31d65e404f96baa9f3501ea3746e" format_stamp: "Formatted at 2018-07-03 08:40:35 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:35.370648 15429 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "293b31d65e404f96baa9f3501ea3746e" format_stamp: "Formatted at 2018-07-03 08:40:35 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:35.378412 15429 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.002s sys 0.004s I0703 01:40:35.379643 15429 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:40:35.380020 15429 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:40:35.384028 15429 fs_manager.cc:417] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s I0703 01:40:35.384079 15429 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal uuid: "293b31d65e404f96baa9f3501ea3746e" format_stamp: "Formatted at 2018-07-03 08:40:35 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:35.384204 15429 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:40:35.418160 15429 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:40:35.420364 15429 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:40:35.420775 15429 tablet_server_main.cc:90] Starting tablet server... I0703 01:40:35.518471 15429 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.193:57702 I0703 01:40:35.518955 15429 webserver.cc:175] Starting webserver on 127.5.240.193:0 I0703 01:40:35.519002 15429 webserver.cc:186] Document root disabled I0703 01:40:35.519784 15429 webserver.cc:313] Webserver started. Bound to: http://127.5.240.193:55042/ I0703 01:40:35.520310 15429 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb I0703 01:40:35.521841 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 15429 I0703 01:40:35.522555 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.5.240.194:0 --local_ip_for_outbound_sockets=127.5.240.194 --webserver_interface=127.5.240.194 --webserver_port=0 --tserver_master_addrs=127.5.240.254:39327 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --enable_leader_failure_detection=false with env {} I0703 01:40:35.533494 15429 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:40:35.552719 15558 heartbeater.cc:346] Connected to a master server at 127.5.240.254:39327 I0703 01:40:35.552873 15558 heartbeater.cc:426] Registering TS with master... I0703 01:40:35.553678 15558 heartbeater.cc:475] Master 127.5.240.254:39327 requested a full tablet report, sending... I0703 01:40:35.555207 15395 ts_manager.cc:98] Registered new tserver with Master: 293b31d65e404f96baa9f3501ea3746e (127.5.240.193:57702) I0703 01:40:35.557757 15395 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.193:48392 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:40:35.696727 15560 flags.cc:406] Enabled unsafe flag: --enable_leader_failure_detection=false W0703 01:40:35.697763 15560 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:40:35.697865 15560 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:40:35.699903 15560 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:40:35.699996 15560 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.194 I0703 01:40:35.706789 15560 tablet_server_main.cc:78] Tablet server non-default flags: --enable_leader_failure_detection=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.194:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.5.240.194 --webserver_port=0 --tserver_master_addrs=127.5.240.254:39327 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.194 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:40:35.713379 15560 tablet_server_main.cc:85] Initializing tablet server... I0703 01:40:35.714900 15560 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 171208us I0703 01:40:35.715445 15560 fs_manager.cc:261] Metadata directory not provided I0703 01:40:35.715503 15560 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:40:35.715750 15560 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:40:35.715812 15560 server_base.cc:433] Attempting to create new FS layout instead I0703 01:40:35.719954 15560 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "2fda9a8952024790ba8814dc6c8269cf" format_stamp: "Formatted at 2018-07-03 08:40:35 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:35.720454 15560 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "2fda9a8952024790ba8814dc6c8269cf" format_stamp: "Formatted at 2018-07-03 08:40:35 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:35.727888 15560 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.004s sys 0.002s I0703 01:40:35.729044 15560 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:40:35.729405 15560 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:40:35.732962 15560 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0703 01:40:35.733007 15560 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal uuid: "2fda9a8952024790ba8814dc6c8269cf" format_stamp: "Formatted at 2018-07-03 08:40:35 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:35.733119 15560 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:40:35.763841 15560 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:40:35.765730 15560 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:40:35.766050 15560 tablet_server_main.cc:90] Starting tablet server... I0703 01:40:35.850108 15560 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.194:55439 I0703 01:40:35.850569 15560 webserver.cc:175] Starting webserver on 127.5.240.194:0 I0703 01:40:35.850608 15560 webserver.cc:186] Document root disabled I0703 01:40:35.851779 15560 webserver.cc:313] Webserver started. Bound to: http://127.5.240.194:35685/ I0703 01:40:35.852298 15560 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb I0703 01:40:35.854111 15560 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:40:35.859666 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 15560 I0703 01:40:35.860397 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.5.240.195:0 --local_ip_for_outbound_sockets=127.5.240.195 --webserver_interface=127.5.240.195 --webserver_port=0 --tserver_master_addrs=127.5.240.254:39327 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --enable_leader_failure_detection=false with env {} I0703 01:40:35.867221 15681 heartbeater.cc:346] Connected to a master server at 127.5.240.254:39327 I0703 01:40:35.867384 15681 heartbeater.cc:426] Registering TS with master... I0703 01:40:35.868288 15681 heartbeater.cc:475] Master 127.5.240.254:39327 requested a full tablet report, sending... I0703 01:40:35.869422 15395 ts_manager.cc:98] Registered new tserver with Master: 2fda9a8952024790ba8814dc6c8269cf (127.5.240.194:55439) I0703 01:40:35.870731 15395 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.194:54353 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:40:36.018762 15684 flags.cc:406] Enabled unsafe flag: --enable_leader_failure_detection=false W0703 01:40:36.019767 15684 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:40:36.019884 15684 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:40:36.022018 15684 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:40:36.022114 15684 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.195 I0703 01:40:36.029021 15684 tablet_server_main.cc:78] Tablet server non-default flags: --enable_leader_failure_detection=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.195:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.5.240.195 --webserver_port=0 --tserver_master_addrs=127.5.240.254:39327 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.195 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:40:36.035540 15684 tablet_server_main.cc:85] Initializing tablet server... I0703 01:40:36.037041 15684 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 171708us I0703 01:40:36.037596 15684 fs_manager.cc:261] Metadata directory not provided I0703 01:40:36.037653 15684 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:40:36.037878 15684 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:40:36.037945 15684 server_base.cc:433] Attempting to create new FS layout instead I0703 01:40:36.042019 15684 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "38ebdb15b8a94873a770fef1e4aa8161" format_stamp: "Formatted at 2018-07-03 08:40:36 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:36.042541 15684 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "38ebdb15b8a94873a770fef1e4aa8161" format_stamp: "Formatted at 2018-07-03 08:40:36 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:36.049867 15684 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s I0703 01:40:36.051077 15684 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:40:36.051481 15684 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:40:36.055152 15684 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0703 01:40:36.055204 15684 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal uuid: "38ebdb15b8a94873a770fef1e4aa8161" format_stamp: "Formatted at 2018-07-03 08:40:36 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:36.055337 15684 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:40:36.103490 15684 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:40:36.104974 15684 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:40:36.105242 15684 tablet_server_main.cc:90] Starting tablet server... I0703 01:40:36.191479 15684 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.195:47275 I0703 01:40:36.191912 15684 webserver.cc:175] Starting webserver on 127.5.240.195:0 I0703 01:40:36.191977 15684 webserver.cc:186] Document root disabled I0703 01:40:36.193189 15684 webserver.cc:313] Webserver started. Bound to: http://127.5.240.195:56108/ I0703 01:40:36.193748 15684 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestLeaderStepDownWhenNotPresent.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb I0703 01:40:36.197774 15684 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:40:36.203897 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 15684 I0703 01:40:36.206820 15803 heartbeater.cc:346] Connected to a master server at 127.5.240.254:39327 I0703 01:40:36.206961 15803 heartbeater.cc:426] Registering TS with master... I0703 01:40:36.207747 15803 heartbeater.cc:475] Master 127.5.240.254:39327 requested a full tablet report, sending... I0703 01:40:36.208863 15395 ts_manager.cc:98] Registered new tserver with Master: 38ebdb15b8a94873a770fef1e4aa8161 (127.5.240.195:47275) I0703 01:40:36.209949 6083 external_mini_cluster.cc:490] 3 TS(s) registered with all masters I0703 01:40:36.210145 15395 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.195:42247 I0703 01:40:36.220067 15395 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.5.240.254:48948: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } W0703 01:40:36.224908 15395 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended). I0703 01:40:36.247220 15740 tablet_service.cc:759] Processing CreateTablet for tablet e655a85172234565af3dc6ecb103d406 (table=TestTable [id=ab1f2a44b1094b119cd036e314276319]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:40:36.248620 15740 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet e655a85172234565af3dc6ecb103d406. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:40:36.254098 15618 tablet_service.cc:759] Processing CreateTablet for tablet e655a85172234565af3dc6ecb103d406 (table=TestTable [id=ab1f2a44b1094b119cd036e314276319]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:40:36.254710 15618 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet e655a85172234565af3dc6ecb103d406. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:40:36.255704 15495 tablet_service.cc:759] Processing CreateTablet for tablet e655a85172234565af3dc6ecb103d406 (table=TestTable [id=ab1f2a44b1094b119cd036e314276319]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:40:36.256342 15495 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet e655a85172234565af3dc6ecb103d406. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:40:36.263268 15740 ts_tablet_manager.cc:1173] T e655a85172234565af3dc6ecb103d406 P 38ebdb15b8a94873a770fef1e4aa8161: Registered tablet (data state: TABLET_DATA_READY) I0703 01:40:36.264964 15822 ts_tablet_manager.cc:1028] T e655a85172234565af3dc6ecb103d406 P 38ebdb15b8a94873a770fef1e4aa8161: Bootstrapping tablet I0703 01:40:36.266018 15822 tablet_bootstrap.cc:427] T e655a85172234565af3dc6ecb103d406 P 38ebdb15b8a94873a770fef1e4aa8161: Bootstrap starting. I0703 01:40:36.266208 15618 ts_tablet_manager.cc:1173] T e655a85172234565af3dc6ecb103d406 P 2fda9a8952024790ba8814dc6c8269cf: Registered tablet (data state: TABLET_DATA_READY) I0703 01:40:36.267114 15823 ts_tablet_manager.cc:1028] T e655a85172234565af3dc6ecb103d406 P 2fda9a8952024790ba8814dc6c8269cf: Bootstrapping tablet I0703 01:40:36.267462 15823 tablet_bootstrap.cc:427] T e655a85172234565af3dc6ecb103d406 P 2fda9a8952024790ba8814dc6c8269cf: Bootstrap starting. I0703 01:40:36.269256 15495 ts_tablet_manager.cc:1173] T e655a85172234565af3dc6ecb103d406 P 293b31d65e404f96baa9f3501ea3746e: Registered tablet (data state: TABLET_DATA_READY) I0703 01:40:36.269390 15822 tablet_bootstrap.cc:578] T e655a85172234565af3dc6ecb103d406 P 38ebdb15b8a94873a770fef1e4aa8161: No blocks or log segments found. Creating new log. I0703 01:40:36.270103 15824 ts_tablet_manager.cc:1028] T e655a85172234565af3dc6ecb103d406 P 293b31d65e404f96baa9f3501ea3746e: Bootstrapping tablet I0703 01:40:36.270334 15822 log.cc:525] T e655a85172234565af3dc6ecb103d406 P 38ebdb15b8a94873a770fef1e4aa8161: Log is configured to *not* fsync() on all Append() calls I0703 01:40:36.270476 15824 tablet_bootstrap.cc:427] T e655a85172234565af3dc6ecb103d406 P 293b31d65e404f96baa9f3501ea3746e: Bootstrap starting. I0703 01:40:36.270870 15823 tablet_bootstrap.cc:578] T e655a85172234565af3dc6ecb103d406 P 2fda9a8952024790ba8814dc6c8269cf: No blocks or log segments found. Creating new log. I0703 01:40:36.271833 15823 log.cc:525] T e655a85172234565af3dc6ecb103d406 P 2fda9a8952024790ba8814dc6c8269cf: Log is configured to *not* fsync() on all Append() calls I0703 01:40:36.272828 15822 tablet_bootstrap.cc:427] T e655a85172234565af3dc6ecb103d406 P 38ebdb15b8a94873a770fef1e4aa8161: No bootstrap required, opened a new log I0703 01:40:36.273041 15822 ts_tablet_manager.cc:1045] T e655a85172234565af3dc6ecb103d406 P 38ebdb15b8a94873a770fef1e4aa8161: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.001s I0703 01:40:36.273294 15824 tablet_bootstrap.cc:578] T e655a85172234565af3dc6ecb103d406 P 293b31d65e404f96baa9f3501ea3746e: No blocks or log segments found. Creating new log. I0703 01:40:36.274248 15824 log.cc:525] T e655a85172234565af3dc6ecb103d406 P 293b31d65e404f96baa9f3501ea3746e: Log is configured to *not* fsync() on all Append() calls I0703 01:40:36.276494 15824 tablet_bootstrap.cc:427] T e655a85172234565af3dc6ecb103d406 P 293b31d65e404f96baa9f3501ea3746e: No bootstrap required, opened a new log I0703 01:40:36.277243 15824 ts_tablet_manager.cc:1045] T e655a85172234565af3dc6ecb103d406 P 293b31d65e404f96baa9f3501ea3746e: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.001s I0703 01:40:36.281599 15822 raft_consensus.cc:325] T e655a85172234565af3dc6ecb103d406 P 38ebdb15b8a94873a770fef1e4aa8161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ebdb15b8a94873a770fef1e4aa8161" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 47275 } } peers { permanent_uuid: "293b31d65e404f96baa9f3501ea3746e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 57702 } } peers { permanent_uuid: "2fda9a8952024790ba8814dc6c8269cf" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 55439 } } I0703 01:40:36.282027 15822 raft_consensus.cc:628] T e655a85172234565af3dc6ecb103d406 P 38ebdb15b8a94873a770fef1e4aa8161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38ebdb15b8a94873a770fef1e4aa8161, State: Initialized, Role: FOLLOWER I0703 01:40:36.282325 15822 consensus_queue.cc:227] T e655a85172234565af3dc6ecb103d406 P 38ebdb15b8a94873a770fef1e4aa8161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ebdb15b8a94873a770fef1e4aa8161" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 47275 } } peers { permanent_uuid: "293b31d65e404f96baa9f3501ea3746e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 57702 } } peers { permanent_uuid: "2fda9a8952024790ba8814dc6c8269cf" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 55439 } } I0703 01:40:36.283210 15803 heartbeater.cc:467] Master 127.5.240.254:39327 was elected leader, sending a full tablet report... I0703 01:40:36.284062 15822 ts_tablet_manager.cc:1073] T e655a85172234565af3dc6ecb103d406 P 38ebdb15b8a94873a770fef1e4aa8161: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s I0703 01:40:36.285104 15824 raft_consensus.cc:325] T e655a85172234565af3dc6ecb103d406 P 293b31d65e404f96baa9f3501ea3746e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ebdb15b8a94873a770fef1e4aa8161" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 47275 } } peers { permanent_uuid: "293b31d65e404f96baa9f3501ea3746e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 57702 } } peers { permanent_uuid: "2fda9a8952024790ba8814dc6c8269cf" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 55439 } } I0703 01:40:36.285382 15824 raft_consensus.cc:628] T e655a85172234565af3dc6ecb103d406 P 293b31d65e404f96baa9f3501ea3746e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 293b31d65e404f96baa9f3501ea3746e, State: Initialized, Role: FOLLOWER I0703 01:40:36.285609 15824 consensus_queue.cc:227] T e655a85172234565af3dc6ecb103d406 P 293b31d65e404f96baa9f3501ea3746e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ebdb15b8a94873a770fef1e4aa8161" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 47275 } } peers { permanent_uuid: "293b31d65e404f96baa9f3501ea3746e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 57702 } } peers { permanent_uuid: "2fda9a8952024790ba8814dc6c8269cf" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 55439 } } I0703 01:40:36.286954 15558 heartbeater.cc:467] Master 127.5.240.254:39327 was elected leader, sending a full tablet report... I0703 01:40:36.287106 15823 tablet_bootstrap.cc:427] T e655a85172234565af3dc6ecb103d406 P 2fda9a8952024790ba8814dc6c8269cf: No bootstrap required, opened a new log I0703 01:40:36.287336 15823 ts_tablet_manager.cc:1045] T e655a85172234565af3dc6ecb103d406 P 2fda9a8952024790ba8814dc6c8269cf: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.011s I0703 01:40:36.287833 15824 ts_tablet_manager.cc:1073] T e655a85172234565af3dc6ecb103d406 P 293b31d65e404f96baa9f3501ea3746e: Time spent starting tablet: real 0.010s user 0.009s sys 0.001s I0703 01:40:36.295226 15823 raft_consensus.cc:325] T e655a85172234565af3dc6ecb103d406 P 2fda9a8952024790ba8814dc6c8269cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ebdb15b8a94873a770fef1e4aa8161" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 47275 } } peers { permanent_uuid: "293b31d65e404f96baa9f3501ea3746e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 57702 } } peers { permanent_uuid: "2fda9a8952024790ba8814dc6c8269cf" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 55439 } } I0703 01:40:36.295503 15823 raft_consensus.cc:628] T e655a85172234565af3dc6ecb103d406 P 2fda9a8952024790ba8814dc6c8269cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fda9a8952024790ba8814dc6c8269cf, State: Initialized, Role: FOLLOWER I0703 01:40:36.295748 15823 consensus_queue.cc:227] T e655a85172234565af3dc6ecb103d406 P 2fda9a8952024790ba8814dc6c8269cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ebdb15b8a94873a770fef1e4aa8161" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 47275 } } peers { permanent_uuid: "293b31d65e404f96baa9f3501ea3746e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 57702 } } peers { permanent_uuid: "2fda9a8952024790ba8814dc6c8269cf" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 55439 } } I0703 01:40:36.297009 15681 heartbeater.cc:467] Master 127.5.240.254:39327 was elected leader, sending a full tablet report... I0703 01:40:36.297353 15823 ts_tablet_manager.cc:1073] T e655a85172234565af3dc6ecb103d406 P 2fda9a8952024790ba8814dc6c8269cf: Time spent starting tablet: real 0.009s user 0.007s sys 0.002s I0703 01:40:36.302284 6083 external_mini_cluster.cc:490] 3 TS(s) registered with all masters I0703 01:40:36.303330 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 293b31d65e404f96baa9f3501ea3746e to finish bootstrapping I0703 01:40:36.308400 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 2fda9a8952024790ba8814dc6c8269cf to finish bootstrapping I0703 01:40:36.312616 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 38ebdb15b8a94873a770fef1e4aa8161 to finish bootstrapping W0703 01:40:36.354441 15682 tablet.cc:1678] T e655a85172234565af3dc6ecb103d406 P 2fda9a8952024790ba8814dc6c8269cf: Can't schedule compaction. Clean time has not been advanced past its initial value. W0703 01:40:36.445789 15804 tablet.cc:1678] T e655a85172234565af3dc6ecb103d406 P 38ebdb15b8a94873a770fef1e4aa8161: Can't schedule compaction. Clean time has not been advanced past its initial value. W0703 01:40:36.528868 15559 tablet.cc:1678] T e655a85172234565af3dc6ecb103d406 P 293b31d65e404f96baa9f3501ea3746e: Can't schedule compaction. Clean time has not been advanced past its initial value. W0703 01:40:36.542177 15832 flags.cc:406] Enabled unsafe flag: --never_fsync=true I0703 01:40:36.630699 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 15429 I0703 01:40:36.633937 15555 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:40:36.675740 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 15429 W0703 01:40:36.684430 15379 connection.cc:512] server connection from 127.5.240.193:48392 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:48392): Connection reset by peer (error 104) I0703 01:40:36.684880 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 15560 I0703 01:40:36.685745 15678 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:40:36.727803 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 15560 W0703 01:40:36.735558 15377 connection.cc:512] server connection from 127.5.240.194:54353 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:54353): Connection reset by peer (error 104) I0703 01:40:36.735774 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 15684 I0703 01:40:36.736649 15800 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:40:36.780565 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 15684 W0703 01:40:36.788700 15376 connection.cc:512] server connection from 127.5.240.195:42247 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.195:42247): Connection reset by peer (error 104) I0703 01:40:36.788805 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master pid 15369 I0703 01:40:36.789710 15425 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:40:36.851511 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master with pid 15369 [ OK ] AdminCliTest.TestLeaderStepDownWhenNotPresent (1976 ms) [ RUN ] AdminCliTest.TestDeleteTable I0703 01:40:36.872822 6083 test_util.cc:208] Using random seed: 1151662691 I0703 01:40:36.874742 6083 ts_itest-base.cc:113] Starting cluster with: I0703 01:40:36.874783 6083 ts_itest-base.cc:114] -------------- I0703 01:40:36.874828 6083 ts_itest-base.cc:115] 1 tablet servers I0703 01:40:36.874857 6083 ts_itest-base.cc:116] 1 replicas per TS I0703 01:40:36.874893 6083 ts_itest-base.cc:117] -------------- I0703 01:40:36.876041 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:50087 --webserver_interface=127.5.240.254 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:40:37.056268 15849 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:40:37.057284 15849 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:40:37.059535 15849 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0703 01:40:37.059603 15849 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:40:37.059653 15849 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0703 01:40:37.059700 15849 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0703 01:40:37.066365 15849 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:50087 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.5.240.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:40:37.071992 15849 master_main.cc:80] Initializing master server... I0703 01:40:37.073103 15849 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 172208us I0703 01:40:37.074172 15849 fs_manager.cc:261] Metadata directory not provided I0703 01:40:37.074237 15849 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:40:37.074877 15849 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:40:37.074939 15849 server_base.cc:433] Attempting to create new FS layout instead I0703 01:40:37.079905 15849 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/instance: uuid: "e79c508e5d1c40d3b5e279657a72bc6a" format_stamp: "Formatted at 2018-07-03 08:40:37 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:37.080478 15849 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "e79c508e5d1c40d3b5e279657a72bc6a" format_stamp: "Formatted at 2018-07-03 08:40:37 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:37.088699 15849 fs_manager.cc:501] Time spent creating directory manager: real 0.008s user 0.004s sys 0.003s I0703 01:40:37.089913 15849 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:40:37.090319 15849 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:40:37.094012 15849 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0703 01:40:37.094071 15849 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal uuid: "e79c508e5d1c40d3b5e279657a72bc6a" format_stamp: "Formatted at 2018-07-03 08:40:37 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:37.094182 15849 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:40:37.137907 15849 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:40:37.139328 15849 master_main.cc:83] Starting Master server... I0703 01:40:37.175544 15849 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.254:50087 I0703 01:40:37.176017 15849 webserver.cc:175] Starting webserver on 127.5.240.254:0 I0703 01:40:37.176061 15849 webserver.cc:186] Document root disabled I0703 01:40:37.177233 15849 webserver.cc:313] Webserver started. Bound to: http://127.5.240.254:58622/ I0703 01:40:37.177837 15849 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb I0703 01:40:37.181500 15908 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:40:37.186957 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master as pid 15849 I0703 01:40:37.187610 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.5.240.193:0 --local_ip_for_outbound_sockets=127.5.240.193 --webserver_interface=127.5.240.193 --webserver_port=0 --tserver_master_addrs=127.5.240.254:50087 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0703 01:40:37.192966 15908 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P e79c508e5d1c40d3b5e279657a72bc6a: Bootstrap starting. I0703 01:40:37.195017 15908 tablet_bootstrap.cc:578] T 00000000000000000000000000000000 P e79c508e5d1c40d3b5e279657a72bc6a: No blocks or log segments found. Creating new log. I0703 01:40:37.195772 15908 log.cc:525] T 00000000000000000000000000000000 P e79c508e5d1c40d3b5e279657a72bc6a: Log is configured to *not* fsync() on all Append() calls I0703 01:40:37.198072 15908 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P e79c508e5d1c40d3b5e279657a72bc6a: No bootstrap required, opened a new log I0703 01:40:37.204490 15908 raft_consensus.cc:325] T 00000000000000000000000000000000 P e79c508e5d1c40d3b5e279657a72bc6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e79c508e5d1c40d3b5e279657a72bc6a" member_type: VOTER } I0703 01:40:37.204700 15908 raft_consensus.cc:351] T 00000000000000000000000000000000 P e79c508e5d1c40d3b5e279657a72bc6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:40:37.204762 15908 raft_consensus.cc:628] T 00000000000000000000000000000000 P e79c508e5d1c40d3b5e279657a72bc6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e79c508e5d1c40d3b5e279657a72bc6a, State: Initialized, Role: FOLLOWER I0703 01:40:37.205210 15908 consensus_queue.cc:227] T 00000000000000000000000000000000 P e79c508e5d1c40d3b5e279657a72bc6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e79c508e5d1c40d3b5e279657a72bc6a" member_type: VOTER } I0703 01:40:37.205355 15908 raft_consensus.cc:365] T 00000000000000000000000000000000 P e79c508e5d1c40d3b5e279657a72bc6a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:40:37.205435 15908 raft_consensus.cc:457] T 00000000000000000000000000000000 P e79c508e5d1c40d3b5e279657a72bc6a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:40:37.205516 15908 raft_consensus.cc:2730] T 00000000000000000000000000000000 P e79c508e5d1c40d3b5e279657a72bc6a: Snoozing failure detection for 2.046s (starting election) I0703 01:40:37.205603 15908 raft_consensus.cc:2782] T 00000000000000000000000000000000 P e79c508e5d1c40d3b5e279657a72bc6a [term 0 FOLLOWER]: Advancing to term 1 I0703 01:40:37.206931 15908 raft_consensus.cc:479] T 00000000000000000000000000000000 P e79c508e5d1c40d3b5e279657a72bc6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e79c508e5d1c40d3b5e279657a72bc6a" member_type: VOTER } I0703 01:40:37.207289 15908 leader_election.cc:270] T 00000000000000000000000000000000 P e79c508e5d1c40d3b5e279657a72bc6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:40:37.208156 15911 raft_consensus.cc:2730] T 00000000000000000000000000000000 P e79c508e5d1c40d3b5e279657a72bc6a: Snoozing failure detection for 1.848s (election complete) I0703 01:40:37.208354 15911 raft_consensus.cc:2537] T 00000000000000000000000000000000 P e79c508e5d1c40d3b5e279657a72bc6a [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:40:37.210539 15911 raft_consensus.cc:593] T 00000000000000000000000000000000 P e79c508e5d1c40d3b5e279657a72bc6a [term 1 LEADER]: Becoming Leader. State: Replica: e79c508e5d1c40d3b5e279657a72bc6a, State: Running, Role: LEADER I0703 01:40:37.210924 15911 consensus_queue.cc:204] T 00000000000000000000000000000000 P e79c508e5d1c40d3b5e279657a72bc6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e79c508e5d1c40d3b5e279657a72bc6a" member_type: VOTER } I0703 01:40:37.212676 15908 sys_catalog.cc:337] T 00000000000000000000000000000000 P e79c508e5d1c40d3b5e279657a72bc6a [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "e79c508e5d1c40d3b5e279657a72bc6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e79c508e5d1c40d3b5e279657a72bc6a" member_type: VOTER } } I0703 01:40:37.213223 15908 sys_catalog.cc:340] T 00000000000000000000000000000000 P e79c508e5d1c40d3b5e279657a72bc6a [sys.catalog]: This master's current role is: LEADER I0703 01:40:37.212990 15913 sys_catalog.cc:337] T 00000000000000000000000000000000 P e79c508e5d1c40d3b5e279657a72bc6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e79c508e5d1c40d3b5e279657a72bc6a. Latest consensus state: current_term: 1 leader_uuid: "e79c508e5d1c40d3b5e279657a72bc6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e79c508e5d1c40d3b5e279657a72bc6a" member_type: VOTER } } I0703 01:40:37.213685 15913 sys_catalog.cc:340] T 00000000000000000000000000000000 P e79c508e5d1c40d3b5e279657a72bc6a [sys.catalog]: This master's current role is: LEADER I0703 01:40:37.212993 15912 sys_catalog.cc:337] T 00000000000000000000000000000000 P e79c508e5d1c40d3b5e279657a72bc6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e79c508e5d1c40d3b5e279657a72bc6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e79c508e5d1c40d3b5e279657a72bc6a" member_type: VOTER } } I0703 01:40:37.215363 15912 sys_catalog.cc:340] T 00000000000000000000000000000000 P e79c508e5d1c40d3b5e279657a72bc6a [sys.catalog]: This master's current role is: LEADER I0703 01:40:37.216827 15908 sys_catalog.cc:424] T 00000000000000000000000000000000 P e79c508e5d1c40d3b5e279657a72bc6a [sys.catalog]: configured and running, proceeding with master startup. I0703 01:40:37.217203 15915 catalog_manager.cc:1006] Loading table and tablet metadata into memory... I0703 01:40:37.218689 15849 master_main.cc:86] Master server successfully started. I0703 01:40:37.224320 15915 catalog_manager.cc:1017] Initializing Kudu internal certificate authority... I0703 01:40:37.249181 15915 catalog_manager.cc:904] Generated new certificate authority record I0703 01:40:37.251135 15915 catalog_manager.cc:1026] Loading token signing keys... I0703 01:40:37.258361 15915 catalog_manager.cc:4102] T 00000000000000000000000000000000 P e79c508e5d1c40d3b5e279657a72bc6a: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:40:37.352833 15909 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:40:37.353855 15909 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:40:37.355865 15909 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:40:37.355973 15909 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.193 I0703 01:40:37.364220 15909 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.5.240.193 --webserver_port=0 --tserver_master_addrs=127.5.240.254:50087 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.193 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:40:37.370906 15909 tablet_server_main.cc:85] Initializing tablet server... I0703 01:40:37.372073 15909 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 172208us I0703 01:40:37.373057 15909 fs_manager.cc:261] Metadata directory not provided I0703 01:40:37.373121 15909 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:40:37.373354 15909 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:40:37.373419 15909 server_base.cc:433] Attempting to create new FS layout instead I0703 01:40:37.376821 15909 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "2beb3b88657948c894640f9373b12b3e" format_stamp: "Formatted at 2018-07-03 08:40:37 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:37.377290 15909 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "2beb3b88657948c894640f9373b12b3e" format_stamp: "Formatted at 2018-07-03 08:40:37 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:37.385079 15909 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s I0703 01:40:37.386240 15909 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:40:37.386656 15909 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:40:37.391764 15909 fs_manager.cc:417] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s I0703 01:40:37.391824 15909 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal uuid: "2beb3b88657948c894640f9373b12b3e" format_stamp: "Formatted at 2018-07-03 08:40:37 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:37.391952 15909 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:40:37.420475 15909 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:40:37.422578 15909 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:40:37.423045 15909 tablet_server_main.cc:90] Starting tablet server... I0703 01:40:37.527279 15909 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.193:50959 I0703 01:40:37.527771 15909 webserver.cc:175] Starting webserver on 127.5.240.193:0 I0703 01:40:37.527808 15909 webserver.cc:186] Document root disabled I0703 01:40:37.528627 15909 webserver.cc:313] Webserver started. Bound to: http://127.5.240.193:57336/ I0703 01:40:37.529156 15909 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb I0703 01:40:37.529456 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 15909 I0703 01:40:37.537041 15909 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:40:37.550377 16038 heartbeater.cc:346] Connected to a master server at 127.5.240.254:50087 I0703 01:40:37.550550 16038 heartbeater.cc:426] Registering TS with master... I0703 01:40:37.551054 16038 heartbeater.cc:475] Master 127.5.240.254:50087 requested a full tablet report, sending... I0703 01:40:37.552707 15874 ts_manager.cc:98] Registered new tserver with Master: 2beb3b88657948c894640f9373b12b3e (127.5.240.193:50959) I0703 01:40:37.555683 15874 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.193:44848 I0703 01:40:37.556079 6083 external_mini_cluster.cc:490] 1 TS(s) registered with all masters I0703 01:40:37.570140 15874 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.5.240.254:49892: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 1 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0703 01:40:37.591329 15975 tablet_service.cc:759] Processing CreateTablet for tablet 14cee2f2dcc44294a1758f66b5f9b9cc (table=TestTable [id=19fb6a2b418e457cbba097191b87f889]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:40:37.591998 15975 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 14cee2f2dcc44294a1758f66b5f9b9cc. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:40:37.601357 15975 ts_tablet_manager.cc:1173] T 14cee2f2dcc44294a1758f66b5f9b9cc P 2beb3b88657948c894640f9373b12b3e: Registered tablet (data state: TABLET_DATA_READY) I0703 01:40:37.602160 16051 ts_tablet_manager.cc:1028] T 14cee2f2dcc44294a1758f66b5f9b9cc P 2beb3b88657948c894640f9373b12b3e: Bootstrapping tablet I0703 01:40:37.602501 16051 tablet_bootstrap.cc:427] T 14cee2f2dcc44294a1758f66b5f9b9cc P 2beb3b88657948c894640f9373b12b3e: Bootstrap starting. I0703 01:40:37.605428 16051 tablet_bootstrap.cc:578] T 14cee2f2dcc44294a1758f66b5f9b9cc P 2beb3b88657948c894640f9373b12b3e: No blocks or log segments found. Creating new log. I0703 01:40:37.606150 16051 log.cc:525] T 14cee2f2dcc44294a1758f66b5f9b9cc P 2beb3b88657948c894640f9373b12b3e: Log is configured to *not* fsync() on all Append() calls I0703 01:40:37.608067 16051 tablet_bootstrap.cc:427] T 14cee2f2dcc44294a1758f66b5f9b9cc P 2beb3b88657948c894640f9373b12b3e: No bootstrap required, opened a new log I0703 01:40:37.608238 16051 ts_tablet_manager.cc:1045] T 14cee2f2dcc44294a1758f66b5f9b9cc P 2beb3b88657948c894640f9373b12b3e: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s I0703 01:40:37.614411 16051 raft_consensus.cc:325] T 14cee2f2dcc44294a1758f66b5f9b9cc P 2beb3b88657948c894640f9373b12b3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2beb3b88657948c894640f9373b12b3e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 50959 } } I0703 01:40:37.614675 16051 raft_consensus.cc:351] T 14cee2f2dcc44294a1758f66b5f9b9cc P 2beb3b88657948c894640f9373b12b3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:40:37.614751 16051 raft_consensus.cc:628] T 14cee2f2dcc44294a1758f66b5f9b9cc P 2beb3b88657948c894640f9373b12b3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2beb3b88657948c894640f9373b12b3e, State: Initialized, Role: FOLLOWER I0703 01:40:37.615120 16051 consensus_queue.cc:227] T 14cee2f2dcc44294a1758f66b5f9b9cc P 2beb3b88657948c894640f9373b12b3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2beb3b88657948c894640f9373b12b3e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 50959 } } I0703 01:40:37.615294 16051 raft_consensus.cc:365] T 14cee2f2dcc44294a1758f66b5f9b9cc P 2beb3b88657948c894640f9373b12b3e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:40:37.615368 16051 raft_consensus.cc:457] T 14cee2f2dcc44294a1758f66b5f9b9cc P 2beb3b88657948c894640f9373b12b3e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:40:37.615447 16051 raft_consensus.cc:2730] T 14cee2f2dcc44294a1758f66b5f9b9cc P 2beb3b88657948c894640f9373b12b3e: Snoozing failure detection for 1.528s (starting election) I0703 01:40:37.615525 16051 raft_consensus.cc:2782] T 14cee2f2dcc44294a1758f66b5f9b9cc P 2beb3b88657948c894640f9373b12b3e [term 0 FOLLOWER]: Advancing to term 1 I0703 01:40:37.616776 16051 raft_consensus.cc:479] T 14cee2f2dcc44294a1758f66b5f9b9cc P 2beb3b88657948c894640f9373b12b3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2beb3b88657948c894640f9373b12b3e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 50959 } } I0703 01:40:37.617105 16051 leader_election.cc:270] T 14cee2f2dcc44294a1758f66b5f9b9cc P 2beb3b88657948c894640f9373b12b3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:40:37.618162 16053 raft_consensus.cc:2730] T 14cee2f2dcc44294a1758f66b5f9b9cc P 2beb3b88657948c894640f9373b12b3e: Snoozing failure detection for 1.783s (election complete) I0703 01:40:37.618260 16053 raft_consensus.cc:2537] T 14cee2f2dcc44294a1758f66b5f9b9cc P 2beb3b88657948c894640f9373b12b3e [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:40:37.619812 16038 heartbeater.cc:467] Master 127.5.240.254:50087 was elected leader, sending a full tablet report... I0703 01:40:37.620857 16051 ts_tablet_manager.cc:1073] T 14cee2f2dcc44294a1758f66b5f9b9cc P 2beb3b88657948c894640f9373b12b3e: Time spent starting tablet: real 0.012s user 0.009s sys 0.003s I0703 01:40:37.620857 16053 raft_consensus.cc:593] T 14cee2f2dcc44294a1758f66b5f9b9cc P 2beb3b88657948c894640f9373b12b3e [term 1 LEADER]: Becoming Leader. State: Replica: 2beb3b88657948c894640f9373b12b3e, State: Running, Role: LEADER I0703 01:40:37.621229 16053 consensus_queue.cc:204] T 14cee2f2dcc44294a1758f66b5f9b9cc P 2beb3b88657948c894640f9373b12b3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2beb3b88657948c894640f9373b12b3e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 50959 } } I0703 01:40:37.624866 15874 catalog_manager.cc:3798] T 14cee2f2dcc44294a1758f66b5f9b9cc P 2beb3b88657948c894640f9373b12b3e reported cstate change: term changed from 0 to 1, leader changed from to 2beb3b88657948c894640f9373b12b3e (127.5.240.193). New cstate: current_term: 1 leader_uuid: "2beb3b88657948c894640f9373b12b3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2beb3b88657948c894640f9373b12b3e" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 50959 } health_report { overall_health: HEALTHY } } } I0703 01:40:37.630983 6083 external_mini_cluster.cc:490] 1 TS(s) registered with all masters I0703 01:40:37.631990 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 2beb3b88657948c894640f9373b12b3e to finish bootstrapping W0703 01:40:37.786892 16040 tablet.cc:1678] T 14cee2f2dcc44294a1758f66b5f9b9cc P 2beb3b88657948c894640f9373b12b3e: Can't schedule compaction. Clean time has not been advanced past its initial value. W0703 01:40:37.853920 16062 flags.cc:406] Enabled unsafe flag: --never_fsync=true I0703 01:40:37.898901 15874 catalog_manager.cc:1731] Servicing DeleteTable request from {username='jenkins'} at 127.5.240.254:49896: table { table_name: "TestTable" } I0703 01:40:37.901278 15874 catalog_manager.cc:4037] T 00000000000000000000000000000000 P e79c508e5d1c40d3b5e279657a72bc6a: Sending DeleteTablet for 1 replicas of tablet 14cee2f2dcc44294a1758f66b5f9b9cc I0703 01:40:37.901466 15874 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 14cee2f2dcc44294a1758f66b5f9b9cc on 2beb3b88657948c894640f9373b12b3e (127.5.240.193:50959) (Table deleted at 2018-07-03 01:40:37 PDT) I0703 01:40:37.902096 15975 tablet_service.cc:800] Processing DeleteTablet for tablet 14cee2f2dcc44294a1758f66b5f9b9cc with delete_type TABLET_DATA_DELETED (Table deleted at 2018-07-03 01:40:37 PDT) from {username='jenkins'} at 127.5.240.193:45135 I0703 01:40:37.902694 16069 tablet_replica.cc:261] T 14cee2f2dcc44294a1758f66b5f9b9cc P 2beb3b88657948c894640f9373b12b3e: stopping tablet replica I0703 01:40:37.902848 16069 maintenance_manager.cc:237] P 2beb3b88657948c894640f9373b12b3e: Unregistered op CompactRowSetsOp(14cee2f2dcc44294a1758f66b5f9b9cc) I0703 01:40:37.902982 16069 maintenance_manager.cc:237] P 2beb3b88657948c894640f9373b12b3e: Unregistered op MinorDeltaCompactionOp(14cee2f2dcc44294a1758f66b5f9b9cc) I0703 01:40:37.903070 16069 maintenance_manager.cc:237] P 2beb3b88657948c894640f9373b12b3e: Unregistered op MajorDeltaCompactionOp(14cee2f2dcc44294a1758f66b5f9b9cc) I0703 01:40:37.903127 16069 maintenance_manager.cc:237] P 2beb3b88657948c894640f9373b12b3e: Unregistered op UndoDeltaBlockGCOp(14cee2f2dcc44294a1758f66b5f9b9cc) I0703 01:40:37.903210 16069 maintenance_manager.cc:237] P 2beb3b88657948c894640f9373b12b3e: Unregistered op FlushMRSOp(14cee2f2dcc44294a1758f66b5f9b9cc) I0703 01:40:37.903285 16069 maintenance_manager.cc:237] P 2beb3b88657948c894640f9373b12b3e: Unregistered op FlushDeltaMemStoresOp(14cee2f2dcc44294a1758f66b5f9b9cc) I0703 01:40:37.903327 16069 maintenance_manager.cc:237] P 2beb3b88657948c894640f9373b12b3e: Unregistered op LogGCOp(14cee2f2dcc44294a1758f66b5f9b9cc) I0703 01:40:37.903405 16069 raft_consensus.cc:2040] T 14cee2f2dcc44294a1758f66b5f9b9cc P 2beb3b88657948c894640f9373b12b3e [term 1 LEADER]: Raft consensus shutting down. I0703 01:40:37.903604 16069 raft_consensus.cc:2067] T 14cee2f2dcc44294a1758f66b5f9b9cc P 2beb3b88657948c894640f9373b12b3e [term 1 FOLLOWER]: Raft consensus is shut down! I0703 01:40:37.904835 16069 ts_tablet_manager.cc:1369] T 14cee2f2dcc44294a1758f66b5f9b9cc P 2beb3b88657948c894640f9373b12b3e: Deleting tablet data with delete state TABLET_DATA_DELETED I0703 01:40:37.908563 16069 ts_tablet_manager.cc:1382] T 14cee2f2dcc44294a1758f66b5f9b9cc P 2beb3b88657948c894640f9373b12b3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1 I0703 01:40:37.908654 16069 log.cc:986] T 14cee2f2dcc44294a1758f66b5f9b9cc P 2beb3b88657948c894640f9373b12b3e: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestDeleteTable.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal/wals/14cee2f2dcc44294a1758f66b5f9b9cc I0703 01:40:37.908968 16069 ts_tablet_manager.cc:1403] T 14cee2f2dcc44294a1758f66b5f9b9cc P 2beb3b88657948c894640f9373b12b3e: Deleting consensus metadata I0703 01:40:37.910190 15859 catalog_manager.cc:3103] TS 2beb3b88657948c894640f9373b12b3e (127.5.240.193:50959): tablet 14cee2f2dcc44294a1758f66b5f9b9cc (table TestTable [id=19fb6a2b418e457cbba097191b87f889]) successfully deleted I0703 01:40:37.953233 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 15909 I0703 01:40:37.955484 16035 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:40:37.998605 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 15909 W0703 01:40:38.006727 15856 connection.cc:512] server connection from 127.5.240.193:44848 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:44848): Connection reset by peer (error 104) I0703 01:40:38.006862 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master pid 15849 I0703 01:40:38.007656 15905 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:40:38.071915 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master with pid 15849 [ OK ] AdminCliTest.TestDeleteTable (1220 ms) [ RUN ] AdminCliTest.TestListTables I0703 01:40:38.092430 6083 test_util.cc:208] Using random seed: 1152882306 I0703 01:40:38.094296 6083 ts_itest-base.cc:113] Starting cluster with: I0703 01:40:38.094326 6083 ts_itest-base.cc:114] -------------- I0703 01:40:38.094377 6083 ts_itest-base.cc:115] 1 tablet servers I0703 01:40:38.094400 6083 ts_itest-base.cc:116] 1 replicas per TS I0703 01:40:38.094446 6083 ts_itest-base.cc:117] -------------- I0703 01:40:38.095631 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTables.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTables.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:43925 --webserver_interface=127.5.240.254 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTables.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTables.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:40:38.274025 16078 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:40:38.275046 16078 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:40:38.277271 16078 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0703 01:40:38.277315 16078 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:40:38.277362 16078 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0703 01:40:38.277410 16078 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0703 01:40:38.284682 16078 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTables.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTables.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:43925 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTables.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.5.240.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTables.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:40:38.290290 16078 master_main.cc:80] Initializing master server... I0703 01:40:38.291368 16078 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 172708us I0703 01:40:38.292407 16078 fs_manager.cc:261] Metadata directory not provided I0703 01:40:38.292472 16078 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:40:38.292733 16078 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:40:38.292832 16078 server_base.cc:433] Attempting to create new FS layout instead I0703 01:40:38.297786 16078 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTables.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/instance: uuid: "a1b30a965070410b893196243d0e86fb" format_stamp: "Formatted at 2018-07-03 08:40:38 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:38.298295 16078 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTables.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "a1b30a965070410b893196243d0e86fb" format_stamp: "Formatted at 2018-07-03 08:40:38 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:38.306336 16078 fs_manager.cc:501] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s I0703 01:40:38.307545 16078 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:40:38.307947 16078 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:40:38.311575 16078 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s I0703 01:40:38.311633 16078 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTables.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTables.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal uuid: "a1b30a965070410b893196243d0e86fb" format_stamp: "Formatted at 2018-07-03 08:40:38 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:38.311740 16078 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTables.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTables.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTables.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:40:38.333992 16078 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:40:38.335427 16078 master_main.cc:83] Starting Master server... I0703 01:40:38.372139 16078 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.254:43925 I0703 01:40:38.372622 16078 webserver.cc:175] Starting webserver on 127.5.240.254:0 I0703 01:40:38.372668 16078 webserver.cc:186] Document root disabled I0703 01:40:38.373353 16078 webserver.cc:313] Webserver started. Bound to: http://127.5.240.254:58695/ I0703 01:40:38.373957 16078 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTables.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb I0703 01:40:38.376075 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master as pid 16078 I0703 01:40:38.376713 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTables.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTables.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.5.240.193:0 --local_ip_for_outbound_sockets=127.5.240.193 --webserver_interface=127.5.240.193 --webserver_port=0 --tserver_master_addrs=127.5.240.254:43925 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTables.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTables.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0703 01:40:38.377070 16137 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:40:38.388751 16137 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P a1b30a965070410b893196243d0e86fb: Bootstrap starting. I0703 01:40:38.390720 16137 tablet_bootstrap.cc:578] T 00000000000000000000000000000000 P a1b30a965070410b893196243d0e86fb: No blocks or log segments found. Creating new log. I0703 01:40:38.391445 16137 log.cc:525] T 00000000000000000000000000000000 P a1b30a965070410b893196243d0e86fb: Log is configured to *not* fsync() on all Append() calls I0703 01:40:38.393657 16137 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P a1b30a965070410b893196243d0e86fb: No bootstrap required, opened a new log I0703 01:40:38.399842 16137 raft_consensus.cc:325] T 00000000000000000000000000000000 P a1b30a965070410b893196243d0e86fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1b30a965070410b893196243d0e86fb" member_type: VOTER } I0703 01:40:38.400034 16137 raft_consensus.cc:351] T 00000000000000000000000000000000 P a1b30a965070410b893196243d0e86fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:40:38.400099 16137 raft_consensus.cc:628] T 00000000000000000000000000000000 P a1b30a965070410b893196243d0e86fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1b30a965070410b893196243d0e86fb, State: Initialized, Role: FOLLOWER I0703 01:40:38.400557 16137 consensus_queue.cc:227] T 00000000000000000000000000000000 P a1b30a965070410b893196243d0e86fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1b30a965070410b893196243d0e86fb" member_type: VOTER } I0703 01:40:38.400724 16137 raft_consensus.cc:365] T 00000000000000000000000000000000 P a1b30a965070410b893196243d0e86fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:40:38.400806 16137 raft_consensus.cc:457] T 00000000000000000000000000000000 P a1b30a965070410b893196243d0e86fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:40:38.400894 16137 raft_consensus.cc:2730] T 00000000000000000000000000000000 P a1b30a965070410b893196243d0e86fb: Snoozing failure detection for 1.942s (starting election) I0703 01:40:38.400985 16137 raft_consensus.cc:2782] T 00000000000000000000000000000000 P a1b30a965070410b893196243d0e86fb [term 0 FOLLOWER]: Advancing to term 1 I0703 01:40:38.402312 16137 raft_consensus.cc:479] T 00000000000000000000000000000000 P a1b30a965070410b893196243d0e86fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1b30a965070410b893196243d0e86fb" member_type: VOTER } I0703 01:40:38.402700 16137 leader_election.cc:270] T 00000000000000000000000000000000 P a1b30a965070410b893196243d0e86fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:40:38.403470 16140 raft_consensus.cc:2730] T 00000000000000000000000000000000 P a1b30a965070410b893196243d0e86fb: Snoozing failure detection for 1.964s (election complete) I0703 01:40:38.403733 16140 raft_consensus.cc:2537] T 00000000000000000000000000000000 P a1b30a965070410b893196243d0e86fb [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:40:38.405818 16140 raft_consensus.cc:593] T 00000000000000000000000000000000 P a1b30a965070410b893196243d0e86fb [term 1 LEADER]: Becoming Leader. State: Replica: a1b30a965070410b893196243d0e86fb, State: Running, Role: LEADER I0703 01:40:38.406196 16140 consensus_queue.cc:204] T 00000000000000000000000000000000 P a1b30a965070410b893196243d0e86fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1b30a965070410b893196243d0e86fb" member_type: VOTER } I0703 01:40:38.408186 16137 sys_catalog.cc:337] T 00000000000000000000000000000000 P a1b30a965070410b893196243d0e86fb [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "a1b30a965070410b893196243d0e86fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1b30a965070410b893196243d0e86fb" member_type: VOTER } } I0703 01:40:38.407984 16141 sys_catalog.cc:337] T 00000000000000000000000000000000 P a1b30a965070410b893196243d0e86fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1b30a965070410b893196243d0e86fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1b30a965070410b893196243d0e86fb" member_type: VOTER } } I0703 01:40:38.408474 16137 sys_catalog.cc:340] T 00000000000000000000000000000000 P a1b30a965070410b893196243d0e86fb [sys.catalog]: This master's current role is: LEADER I0703 01:40:38.408594 16141 sys_catalog.cc:340] T 00000000000000000000000000000000 P a1b30a965070410b893196243d0e86fb [sys.catalog]: This master's current role is: LEADER I0703 01:40:38.407984 16142 sys_catalog.cc:337] T 00000000000000000000000000000000 P a1b30a965070410b893196243d0e86fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1b30a965070410b893196243d0e86fb. Latest consensus state: current_term: 1 leader_uuid: "a1b30a965070410b893196243d0e86fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1b30a965070410b893196243d0e86fb" member_type: VOTER } } I0703 01:40:38.410490 16142 sys_catalog.cc:340] T 00000000000000000000000000000000 P a1b30a965070410b893196243d0e86fb [sys.catalog]: This master's current role is: LEADER I0703 01:40:38.412425 16137 sys_catalog.cc:424] T 00000000000000000000000000000000 P a1b30a965070410b893196243d0e86fb [sys.catalog]: configured and running, proceeding with master startup. I0703 01:40:38.413795 16078 master_main.cc:86] Master server successfully started. W0703 01:40:38.417721 16145 catalog_manager.cc:1064] T 00000000000000000000000000000000 P a1b30a965070410b893196243d0e86fb: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0703 01:40:38.417824 16145 catalog_manager.cc:563] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0703 01:40:38.417922 16144 catalog_manager.cc:1006] Loading table and tablet metadata into memory... I0703 01:40:38.421221 16144 catalog_manager.cc:1017] Initializing Kudu internal certificate authority... I0703 01:40:38.453320 16144 catalog_manager.cc:904] Generated new certificate authority record I0703 01:40:38.455483 16144 catalog_manager.cc:1026] Loading token signing keys... I0703 01:40:38.465473 16144 catalog_manager.cc:4102] T 00000000000000000000000000000000 P a1b30a965070410b893196243d0e86fb: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:40:38.539350 16138 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:40:38.540356 16138 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:40:38.542385 16138 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:40:38.542488 16138 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.193 I0703 01:40:38.550262 16138 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTables.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTables.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTables.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.5.240.193 --webserver_port=0 --tserver_master_addrs=127.5.240.254:43925 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.193 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTables.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:40:38.556550 16138 tablet_server_main.cc:85] Initializing tablet server... I0703 01:40:38.557713 16138 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 172708us I0703 01:40:38.558652 16138 fs_manager.cc:261] Metadata directory not provided I0703 01:40:38.558706 16138 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:40:38.558934 16138 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:40:38.558993 16138 server_base.cc:433] Attempting to create new FS layout instead I0703 01:40:38.562347 16138 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTables.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "9406f6772f9e4d6f938368e782878db8" format_stamp: "Formatted at 2018-07-03 08:40:38 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:38.562816 16138 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTables.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "9406f6772f9e4d6f938368e782878db8" format_stamp: "Formatted at 2018-07-03 08:40:38 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:38.570636 16138 fs_manager.cc:501] Time spent creating directory manager: real 0.008s user 0.002s sys 0.004s I0703 01:40:38.571820 16138 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:40:38.572206 16138 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:40:38.576575 16138 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s I0703 01:40:38.576637 16138 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTables.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTables.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal uuid: "9406f6772f9e4d6f938368e782878db8" format_stamp: "Formatted at 2018-07-03 08:40:38 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:38.576756 16138 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTables.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTables.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTables.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:40:38.640146 16138 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:40:38.641729 16138 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:40:38.642102 16138 tablet_server_main.cc:90] Starting tablet server... I0703 01:40:38.742308 16138 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.193:50723 I0703 01:40:38.742792 16138 webserver.cc:175] Starting webserver on 127.5.240.193:0 I0703 01:40:38.742831 16138 webserver.cc:186] Document root disabled I0703 01:40:38.744163 16138 webserver.cc:313] Webserver started. Bound to: http://127.5.240.193:53347/ I0703 01:40:38.744727 16138 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTables.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb I0703 01:40:38.746795 16138 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:40:38.749563 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 16138 I0703 01:40:38.767585 16267 heartbeater.cc:346] Connected to a master server at 127.5.240.254:43925 I0703 01:40:38.767736 16267 heartbeater.cc:426] Registering TS with master... I0703 01:40:38.768486 16267 heartbeater.cc:475] Master 127.5.240.254:43925 requested a full tablet report, sending... I0703 01:40:38.770637 16103 ts_manager.cc:98] Registered new tserver with Master: 9406f6772f9e4d6f938368e782878db8 (127.5.240.193:50723) I0703 01:40:38.772250 6083 external_mini_cluster.cc:490] 1 TS(s) registered with all masters I0703 01:40:38.772419 16103 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.193:51853 I0703 01:40:38.781713 16103 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.5.240.254:37023: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 1 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0703 01:40:38.800463 16204 tablet_service.cc:759] Processing CreateTablet for tablet 43b99f8a4fde4ea2941cdf5160e85fa6 (table=TestTable [id=2d9f5aaecb944af89f677822356b4c35]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:40:38.801131 16204 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 43b99f8a4fde4ea2941cdf5160e85fa6. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:40:38.810075 16204 ts_tablet_manager.cc:1173] T 43b99f8a4fde4ea2941cdf5160e85fa6 P 9406f6772f9e4d6f938368e782878db8: Registered tablet (data state: TABLET_DATA_READY) I0703 01:40:38.811228 16280 ts_tablet_manager.cc:1028] T 43b99f8a4fde4ea2941cdf5160e85fa6 P 9406f6772f9e4d6f938368e782878db8: Bootstrapping tablet I0703 01:40:38.811967 16280 tablet_bootstrap.cc:427] T 43b99f8a4fde4ea2941cdf5160e85fa6 P 9406f6772f9e4d6f938368e782878db8: Bootstrap starting. I0703 01:40:38.814363 16280 tablet_bootstrap.cc:578] T 43b99f8a4fde4ea2941cdf5160e85fa6 P 9406f6772f9e4d6f938368e782878db8: No blocks or log segments found. Creating new log. I0703 01:40:38.815490 16280 log.cc:525] T 43b99f8a4fde4ea2941cdf5160e85fa6 P 9406f6772f9e4d6f938368e782878db8: Log is configured to *not* fsync() on all Append() calls I0703 01:40:38.817389 16280 tablet_bootstrap.cc:427] T 43b99f8a4fde4ea2941cdf5160e85fa6 P 9406f6772f9e4d6f938368e782878db8: No bootstrap required, opened a new log I0703 01:40:38.817570 16280 ts_tablet_manager.cc:1045] T 43b99f8a4fde4ea2941cdf5160e85fa6 P 9406f6772f9e4d6f938368e782878db8: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.002s I0703 01:40:38.823673 16280 raft_consensus.cc:325] T 43b99f8a4fde4ea2941cdf5160e85fa6 P 9406f6772f9e4d6f938368e782878db8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9406f6772f9e4d6f938368e782878db8" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 50723 } } I0703 01:40:38.823918 16280 raft_consensus.cc:351] T 43b99f8a4fde4ea2941cdf5160e85fa6 P 9406f6772f9e4d6f938368e782878db8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:40:38.823985 16280 raft_consensus.cc:628] T 43b99f8a4fde4ea2941cdf5160e85fa6 P 9406f6772f9e4d6f938368e782878db8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9406f6772f9e4d6f938368e782878db8, State: Initialized, Role: FOLLOWER I0703 01:40:38.824344 16280 consensus_queue.cc:227] T 43b99f8a4fde4ea2941cdf5160e85fa6 P 9406f6772f9e4d6f938368e782878db8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9406f6772f9e4d6f938368e782878db8" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 50723 } } I0703 01:40:38.824517 16280 raft_consensus.cc:365] T 43b99f8a4fde4ea2941cdf5160e85fa6 P 9406f6772f9e4d6f938368e782878db8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:40:38.824609 16280 raft_consensus.cc:457] T 43b99f8a4fde4ea2941cdf5160e85fa6 P 9406f6772f9e4d6f938368e782878db8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:40:38.824685 16280 raft_consensus.cc:2730] T 43b99f8a4fde4ea2941cdf5160e85fa6 P 9406f6772f9e4d6f938368e782878db8: Snoozing failure detection for 2.072s (starting election) I0703 01:40:38.824759 16280 raft_consensus.cc:2782] T 43b99f8a4fde4ea2941cdf5160e85fa6 P 9406f6772f9e4d6f938368e782878db8 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:40:38.826001 16280 raft_consensus.cc:479] T 43b99f8a4fde4ea2941cdf5160e85fa6 P 9406f6772f9e4d6f938368e782878db8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9406f6772f9e4d6f938368e782878db8" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 50723 } } I0703 01:40:38.826328 16280 leader_election.cc:270] T 43b99f8a4fde4ea2941cdf5160e85fa6 P 9406f6772f9e4d6f938368e782878db8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:40:38.826992 16282 raft_consensus.cc:2730] T 43b99f8a4fde4ea2941cdf5160e85fa6 P 9406f6772f9e4d6f938368e782878db8: Snoozing failure detection for 1.919s (election complete) I0703 01:40:38.827208 16282 raft_consensus.cc:2537] T 43b99f8a4fde4ea2941cdf5160e85fa6 P 9406f6772f9e4d6f938368e782878db8 [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:40:38.828841 16267 heartbeater.cc:467] Master 127.5.240.254:43925 was elected leader, sending a full tablet report... I0703 01:40:38.829944 16282 raft_consensus.cc:593] T 43b99f8a4fde4ea2941cdf5160e85fa6 P 9406f6772f9e4d6f938368e782878db8 [term 1 LEADER]: Becoming Leader. State: Replica: 9406f6772f9e4d6f938368e782878db8, State: Running, Role: LEADER I0703 01:40:38.830080 16280 ts_tablet_manager.cc:1073] T 43b99f8a4fde4ea2941cdf5160e85fa6 P 9406f6772f9e4d6f938368e782878db8: Time spent starting tablet: real 0.012s user 0.006s sys 0.005s I0703 01:40:38.830329 16282 consensus_queue.cc:204] T 43b99f8a4fde4ea2941cdf5160e85fa6 P 9406f6772f9e4d6f938368e782878db8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9406f6772f9e4d6f938368e782878db8" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 50723 } } I0703 01:40:38.833925 16103 catalog_manager.cc:3798] T 43b99f8a4fde4ea2941cdf5160e85fa6 P 9406f6772f9e4d6f938368e782878db8 reported cstate change: term changed from 0 to 1, leader changed from to 9406f6772f9e4d6f938368e782878db8 (127.5.240.193). New cstate: current_term: 1 leader_uuid: "9406f6772f9e4d6f938368e782878db8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9406f6772f9e4d6f938368e782878db8" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 50723 } health_report { overall_health: HEALTHY } } } I0703 01:40:38.842505 6083 external_mini_cluster.cc:490] 1 TS(s) registered with all masters I0703 01:40:38.843542 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 9406f6772f9e4d6f938368e782878db8 to finish bootstrapping W0703 01:40:38.997079 16268 tablet.cc:1678] T 43b99f8a4fde4ea2941cdf5160e85fa6 P 9406f6772f9e4d6f938368e782878db8: Can't schedule compaction. Clean time has not been advanced past its initial value. W0703 01:40:39.058645 16286 flags.cc:406] Enabled unsafe flag: --never_fsync=true I0703 01:40:39.146958 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 16138 I0703 01:40:39.150738 16264 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:40:39.193619 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 16138 I0703 01:40:39.201764 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master pid 16078 W0703 01:40:39.201776 16086 connection.cc:512] server connection from 127.5.240.193:51853 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:51853): Connection reset by peer (error 104) I0703 01:40:39.202631 16134 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:40:39.264060 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master with pid 16078 [ OK ] AdminCliTest.TestListTables (1192 ms) [ RUN ] AdminCliTest.TestListTablesDetail I0703 01:40:39.284548 6083 test_util.cc:208] Using random seed: 1154074417 I0703 01:40:39.286566 6083 ts_itest-base.cc:113] Starting cluster with: I0703 01:40:39.286597 6083 ts_itest-base.cc:114] -------------- I0703 01:40:39.286610 6083 ts_itest-base.cc:115] 3 tablet servers I0703 01:40:39.286641 6083 ts_itest-base.cc:116] 3 replicas per TS I0703 01:40:39.286660 6083 ts_itest-base.cc:117] -------------- I0703 01:40:39.287770 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:60310 --webserver_interface=127.5.240.254 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:40:39.466132 16301 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:40:39.467133 16301 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:40:39.469336 16301 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0703 01:40:39.469372 16301 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:40:39.469399 16301 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0703 01:40:39.469424 16301 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0703 01:40:39.476683 16301 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:60310 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.5.240.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:40:39.482250 16301 master_main.cc:80] Initializing master server... I0703 01:40:39.483325 16301 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 173208us I0703 01:40:39.484362 16301 fs_manager.cc:261] Metadata directory not provided I0703 01:40:39.484417 16301 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:40:39.484657 16301 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:40:39.484721 16301 server_base.cc:433] Attempting to create new FS layout instead I0703 01:40:39.489672 16301 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/instance: uuid: "a357f5e7a3be473cad9c8ae8f16ab94d" format_stamp: "Formatted at 2018-07-03 08:40:39 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:39.490155 16301 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "a357f5e7a3be473cad9c8ae8f16ab94d" format_stamp: "Formatted at 2018-07-03 08:40:39 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:39.498183 16301 fs_manager.cc:501] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s I0703 01:40:39.499404 16301 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:40:39.499804 16301 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:40:39.503399 16301 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s I0703 01:40:39.503451 16301 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal uuid: "a357f5e7a3be473cad9c8ae8f16ab94d" format_stamp: "Formatted at 2018-07-03 08:40:39 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:39.503551 16301 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:40:39.537796 16301 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:40:39.539609 16301 master_main.cc:83] Starting Master server... I0703 01:40:39.575875 16301 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.254:60310 I0703 01:40:39.576321 16301 webserver.cc:175] Starting webserver on 127.5.240.254:0 I0703 01:40:39.576365 16301 webserver.cc:186] Document root disabled I0703 01:40:39.577131 16301 webserver.cc:313] Webserver started. Bound to: http://127.5.240.254:58530/ I0703 01:40:39.577729 16301 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb I0703 01:40:39.578447 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master as pid 16301 I0703 01:40:39.579332 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.5.240.193:0 --local_ip_for_outbound_sockets=127.5.240.193 --webserver_interface=127.5.240.193 --webserver_port=0 --tserver_master_addrs=127.5.240.254:60310 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0703 01:40:39.582286 16360 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:40:39.594545 16360 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P a357f5e7a3be473cad9c8ae8f16ab94d: Bootstrap starting. I0703 01:40:39.596508 16360 tablet_bootstrap.cc:578] T 00000000000000000000000000000000 P a357f5e7a3be473cad9c8ae8f16ab94d: No blocks or log segments found. Creating new log. I0703 01:40:39.597231 16360 log.cc:525] T 00000000000000000000000000000000 P a357f5e7a3be473cad9c8ae8f16ab94d: Log is configured to *not* fsync() on all Append() calls I0703 01:40:39.599467 16360 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P a357f5e7a3be473cad9c8ae8f16ab94d: No bootstrap required, opened a new log I0703 01:40:39.605672 16360 raft_consensus.cc:325] T 00000000000000000000000000000000 P a357f5e7a3be473cad9c8ae8f16ab94d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a357f5e7a3be473cad9c8ae8f16ab94d" member_type: VOTER } I0703 01:40:39.605854 16360 raft_consensus.cc:351] T 00000000000000000000000000000000 P a357f5e7a3be473cad9c8ae8f16ab94d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:40:39.605902 16360 raft_consensus.cc:628] T 00000000000000000000000000000000 P a357f5e7a3be473cad9c8ae8f16ab94d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a357f5e7a3be473cad9c8ae8f16ab94d, State: Initialized, Role: FOLLOWER I0703 01:40:39.606319 16360 consensus_queue.cc:227] T 00000000000000000000000000000000 P a357f5e7a3be473cad9c8ae8f16ab94d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a357f5e7a3be473cad9c8ae8f16ab94d" member_type: VOTER } I0703 01:40:39.606449 16360 raft_consensus.cc:365] T 00000000000000000000000000000000 P a357f5e7a3be473cad9c8ae8f16ab94d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:40:39.606511 16360 raft_consensus.cc:457] T 00000000000000000000000000000000 P a357f5e7a3be473cad9c8ae8f16ab94d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:40:39.606582 16360 raft_consensus.cc:2730] T 00000000000000000000000000000000 P a357f5e7a3be473cad9c8ae8f16ab94d: Snoozing failure detection for 1.701s (starting election) I0703 01:40:39.606672 16360 raft_consensus.cc:2782] T 00000000000000000000000000000000 P a357f5e7a3be473cad9c8ae8f16ab94d [term 0 FOLLOWER]: Advancing to term 1 I0703 01:40:39.607985 16360 raft_consensus.cc:479] T 00000000000000000000000000000000 P a357f5e7a3be473cad9c8ae8f16ab94d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a357f5e7a3be473cad9c8ae8f16ab94d" member_type: VOTER } I0703 01:40:39.608330 16360 leader_election.cc:270] T 00000000000000000000000000000000 P a357f5e7a3be473cad9c8ae8f16ab94d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:40:39.609218 16363 raft_consensus.cc:2730] T 00000000000000000000000000000000 P a357f5e7a3be473cad9c8ae8f16ab94d: Snoozing failure detection for 1.620s (election complete) I0703 01:40:39.609613 16363 raft_consensus.cc:2537] T 00000000000000000000000000000000 P a357f5e7a3be473cad9c8ae8f16ab94d [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:40:39.611054 16363 raft_consensus.cc:593] T 00000000000000000000000000000000 P a357f5e7a3be473cad9c8ae8f16ab94d [term 1 LEADER]: Becoming Leader. State: Replica: a357f5e7a3be473cad9c8ae8f16ab94d, State: Running, Role: LEADER I0703 01:40:39.611490 16363 consensus_queue.cc:204] T 00000000000000000000000000000000 P a357f5e7a3be473cad9c8ae8f16ab94d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a357f5e7a3be473cad9c8ae8f16ab94d" member_type: VOTER } I0703 01:40:39.613447 16360 sys_catalog.cc:337] T 00000000000000000000000000000000 P a357f5e7a3be473cad9c8ae8f16ab94d [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "a357f5e7a3be473cad9c8ae8f16ab94d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a357f5e7a3be473cad9c8ae8f16ab94d" member_type: VOTER } } I0703 01:40:39.613251 16365 sys_catalog.cc:337] T 00000000000000000000000000000000 P a357f5e7a3be473cad9c8ae8f16ab94d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a357f5e7a3be473cad9c8ae8f16ab94d. Latest consensus state: current_term: 1 leader_uuid: "a357f5e7a3be473cad9c8ae8f16ab94d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a357f5e7a3be473cad9c8ae8f16ab94d" member_type: VOTER } } I0703 01:40:39.614109 16360 sys_catalog.cc:340] T 00000000000000000000000000000000 P a357f5e7a3be473cad9c8ae8f16ab94d [sys.catalog]: This master's current role is: LEADER I0703 01:40:39.613449 16364 sys_catalog.cc:337] T 00000000000000000000000000000000 P a357f5e7a3be473cad9c8ae8f16ab94d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a357f5e7a3be473cad9c8ae8f16ab94d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a357f5e7a3be473cad9c8ae8f16ab94d" member_type: VOTER } } I0703 01:40:39.614321 16364 sys_catalog.cc:340] T 00000000000000000000000000000000 P a357f5e7a3be473cad9c8ae8f16ab94d [sys.catalog]: This master's current role is: LEADER I0703 01:40:39.614435 16365 sys_catalog.cc:340] T 00000000000000000000000000000000 P a357f5e7a3be473cad9c8ae8f16ab94d [sys.catalog]: This master's current role is: LEADER I0703 01:40:39.617820 16360 sys_catalog.cc:424] T 00000000000000000000000000000000 P a357f5e7a3be473cad9c8ae8f16ab94d [sys.catalog]: configured and running, proceeding with master startup. I0703 01:40:39.617887 16367 catalog_manager.cc:1006] Loading table and tablet metadata into memory... I0703 01:40:39.619583 16301 master_main.cc:86] Master server successfully started. I0703 01:40:39.624876 16367 catalog_manager.cc:1017] Initializing Kudu internal certificate authority... I0703 01:40:39.670531 16367 catalog_manager.cc:904] Generated new certificate authority record I0703 01:40:39.672443 16367 catalog_manager.cc:1026] Loading token signing keys... I0703 01:40:39.688158 16367 catalog_manager.cc:4102] T 00000000000000000000000000000000 P a357f5e7a3be473cad9c8ae8f16ab94d: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:40:39.753311 16361 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:40:39.754326 16361 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:40:39.756381 16361 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:40:39.756485 16361 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.193 I0703 01:40:39.765789 16361 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.5.240.193 --webserver_port=0 --tserver_master_addrs=127.5.240.254:60310 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.193 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:40:39.772379 16361 tablet_server_main.cc:85] Initializing tablet server... I0703 01:40:39.773533 16361 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 173208us I0703 01:40:39.774489 16361 fs_manager.cc:261] Metadata directory not provided I0703 01:40:39.774552 16361 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:40:39.774806 16361 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:40:39.774869 16361 server_base.cc:433] Attempting to create new FS layout instead I0703 01:40:39.779002 16361 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "2c88231a2ae44434927855d7fd17e8e9" format_stamp: "Formatted at 2018-07-03 08:40:39 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:39.779464 16361 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "2c88231a2ae44434927855d7fd17e8e9" format_stamp: "Formatted at 2018-07-03 08:40:39 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:39.787303 16361 fs_manager.cc:501] Time spent creating directory manager: real 0.008s user 0.003s sys 0.002s I0703 01:40:39.788513 16361 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:40:39.788919 16361 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:40:39.792964 16361 fs_manager.cc:417] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s I0703 01:40:39.793025 16361 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal uuid: "2c88231a2ae44434927855d7fd17e8e9" format_stamp: "Formatted at 2018-07-03 08:40:39 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:39.793148 16361 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:40:39.829663 16361 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:40:39.831846 16361 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:40:39.832314 16361 tablet_server_main.cc:90] Starting tablet server... I0703 01:40:39.939414 16361 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.193:60206 I0703 01:40:39.939872 16361 webserver.cc:175] Starting webserver on 127.5.240.193:0 I0703 01:40:39.939913 16361 webserver.cc:186] Document root disabled I0703 01:40:39.941275 16361 webserver.cc:313] Webserver started. Bound to: http://127.5.240.193:40333/ I0703 01:40:39.941830 16361 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb I0703 01:40:39.943831 16361 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:40:39.944108 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 16361 I0703 01:40:39.944676 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.5.240.194:0 --local_ip_for_outbound_sockets=127.5.240.194 --webserver_interface=127.5.240.194 --webserver_port=0 --tserver_master_addrs=127.5.240.254:60310 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0703 01:40:39.973595 16490 heartbeater.cc:346] Connected to a master server at 127.5.240.254:60310 I0703 01:40:39.973752 16490 heartbeater.cc:426] Registering TS with master... I0703 01:40:39.974536 16490 heartbeater.cc:475] Master 127.5.240.254:60310 requested a full tablet report, sending... I0703 01:40:39.975924 16327 ts_manager.cc:98] Registered new tserver with Master: 2c88231a2ae44434927855d7fd17e8e9 (127.5.240.193:60206) I0703 01:40:39.977735 16327 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.193:44201 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:40:40.106412 16492 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:40:40.107406 16492 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:40:40.109397 16492 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:40:40.109486 16492 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.194 I0703 01:40:40.116175 16492 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.194:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.5.240.194 --webserver_port=0 --tserver_master_addrs=127.5.240.254:60310 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.194 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:40:40.122680 16492 tablet_server_main.cc:85] Initializing tablet server... I0703 01:40:40.123802 16492 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 173708us I0703 01:40:40.124729 16492 fs_manager.cc:261] Metadata directory not provided I0703 01:40:40.124784 16492 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:40:40.125000 16492 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:40:40.125056 16492 server_base.cc:433] Attempting to create new FS layout instead I0703 01:40:40.129036 16492 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "719afe22f6f64605919e0814444300c8" format_stamp: "Formatted at 2018-07-03 08:40:40 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:40.129519 16492 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "719afe22f6f64605919e0814444300c8" format_stamp: "Formatted at 2018-07-03 08:40:40 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:40.136674 16492 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.004s sys 0.002s I0703 01:40:40.137835 16492 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:40:40.138188 16492 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:40:40.141265 16492 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0703 01:40:40.141316 16492 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal uuid: "719afe22f6f64605919e0814444300c8" format_stamp: "Formatted at 2018-07-03 08:40:40 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:40.141425 16492 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:40:40.192144 16492 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:40:40.193536 16492 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:40:40.193872 16492 tablet_server_main.cc:90] Starting tablet server... I0703 01:40:40.278003 16492 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.194:45951 I0703 01:40:40.278404 16492 webserver.cc:175] Starting webserver on 127.5.240.194:0 I0703 01:40:40.278435 16492 webserver.cc:186] Document root disabled I0703 01:40:40.279598 16492 webserver.cc:313] Webserver started. Bound to: http://127.5.240.194:59235/ I0703 01:40:40.280120 16492 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb I0703 01:40:40.282416 16492 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:40:40.289492 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 16492 I0703 01:40:40.290182 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.5.240.195:0 --local_ip_for_outbound_sockets=127.5.240.195 --webserver_interface=127.5.240.195 --webserver_port=0 --tserver_master_addrs=127.5.240.254:60310 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --log_cache_size_limit_mb=10 --consensus_rpc_timeout_ms=30000 with env {} I0703 01:40:40.293857 16613 heartbeater.cc:346] Connected to a master server at 127.5.240.254:60310 I0703 01:40:40.294044 16613 heartbeater.cc:426] Registering TS with master... I0703 01:40:40.294957 16613 heartbeater.cc:475] Master 127.5.240.254:60310 requested a full tablet report, sending... I0703 01:40:40.296005 16327 ts_manager.cc:98] Registered new tserver with Master: 719afe22f6f64605919e0814444300c8 (127.5.240.194:45951) I0703 01:40:40.296854 16327 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.194:54832 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:40:40.446580 16616 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:40:40.447583 16616 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:40:40.449615 16616 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:40:40.449717 16616 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.195 I0703 01:40:40.456686 16616 tablet_server_main.cc:78] Tablet server non-default flags: --log_cache_size_limit_mb=10 --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.195:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.5.240.195 --webserver_port=0 --tserver_master_addrs=127.5.240.254:60310 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.195 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:40:40.463160 16616 tablet_server_main.cc:85] Initializing tablet server... I0703 01:40:40.464254 16616 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 173708us I0703 01:40:40.465178 16616 fs_manager.cc:261] Metadata directory not provided I0703 01:40:40.465231 16616 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:40:40.465441 16616 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:40:40.465497 16616 server_base.cc:433] Attempting to create new FS layout instead I0703 01:40:40.469461 16616 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "c446c114fd3c4c29b178ea4d2423b26b" format_stamp: "Formatted at 2018-07-03 08:40:40 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:40.469961 16616 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "c446c114fd3c4c29b178ea4d2423b26b" format_stamp: "Formatted at 2018-07-03 08:40:40 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:40.477063 16616 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.003s sys 0.003s I0703 01:40:40.478225 16616 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:40:40.478597 16616 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:40:40.482129 16616 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0703 01:40:40.482180 16616 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal uuid: "c446c114fd3c4c29b178ea4d2423b26b" format_stamp: "Formatted at 2018-07-03 08:40:40 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:40.482287 16616 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:40:40.525015 16616 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:40:40.526857 16616 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:40:40.527215 16616 tablet_server_main.cc:90] Starting tablet server... I0703 01:40:40.611214 16616 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.195:34006 I0703 01:40:40.611666 16616 webserver.cc:175] Starting webserver on 127.5.240.195:0 I0703 01:40:40.611706 16616 webserver.cc:186] Document root disabled I0703 01:40:40.612402 16616 webserver.cc:313] Webserver started. Bound to: http://127.5.240.195:55063/ I0703 01:40:40.613010 16616 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.AdminCliTest.TestListTablesDetail.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb I0703 01:40:40.614867 16616 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:40:40.623363 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 16616 I0703 01:40:40.628027 16735 heartbeater.cc:346] Connected to a master server at 127.5.240.254:60310 I0703 01:40:40.628198 16735 heartbeater.cc:426] Registering TS with master... I0703 01:40:40.629112 16735 heartbeater.cc:475] Master 127.5.240.254:60310 requested a full tablet report, sending... I0703 01:40:40.630338 16326 ts_manager.cc:98] Registered new tserver with Master: c446c114fd3c4c29b178ea4d2423b26b (127.5.240.195:34006) I0703 01:40:40.631223 16326 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.195:50286 I0703 01:40:40.632230 6083 external_mini_cluster.cc:490] 3 TS(s) registered with all masters I0703 01:40:40.641571 16326 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.5.240.254:43854: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } W0703 01:40:40.646199 16326 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended). I0703 01:40:40.667223 16672 tablet_service.cc:759] Processing CreateTablet for tablet 945875de50a24af89a08d08b4354e1f4 (table=TestTable [id=f784cc3ee45a42bb9af6e612193b0bec]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:40:40.667928 16672 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 945875de50a24af89a08d08b4354e1f4. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:40:40.667757 16550 tablet_service.cc:759] Processing CreateTablet for tablet 945875de50a24af89a08d08b4354e1f4 (table=TestTable [id=f784cc3ee45a42bb9af6e612193b0bec]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:40:40.668522 16550 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 945875de50a24af89a08d08b4354e1f4. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:40:40.668656 16427 tablet_service.cc:759] Processing CreateTablet for tablet 945875de50a24af89a08d08b4354e1f4 (table=TestTable [id=f784cc3ee45a42bb9af6e612193b0bec]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:40:40.669378 16427 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 945875de50a24af89a08d08b4354e1f4. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:40:40.677760 16672 ts_tablet_manager.cc:1173] T 945875de50a24af89a08d08b4354e1f4 P c446c114fd3c4c29b178ea4d2423b26b: Registered tablet (data state: TABLET_DATA_READY) I0703 01:40:40.678637 16754 ts_tablet_manager.cc:1028] T 945875de50a24af89a08d08b4354e1f4 P c446c114fd3c4c29b178ea4d2423b26b: Bootstrapping tablet I0703 01:40:40.679004 16754 tablet_bootstrap.cc:427] T 945875de50a24af89a08d08b4354e1f4 P c446c114fd3c4c29b178ea4d2423b26b: Bootstrap starting. I0703 01:40:40.681560 16754 tablet_bootstrap.cc:578] T 945875de50a24af89a08d08b4354e1f4 P c446c114fd3c4c29b178ea4d2423b26b: No blocks or log segments found. Creating new log. I0703 01:40:40.682276 16754 log.cc:525] T 945875de50a24af89a08d08b4354e1f4 P c446c114fd3c4c29b178ea4d2423b26b: Log is configured to *not* fsync() on all Append() calls I0703 01:40:40.683179 16427 ts_tablet_manager.cc:1173] T 945875de50a24af89a08d08b4354e1f4 P 2c88231a2ae44434927855d7fd17e8e9: Registered tablet (data state: TABLET_DATA_READY) I0703 01:40:40.684062 16756 ts_tablet_manager.cc:1028] T 945875de50a24af89a08d08b4354e1f4 P 2c88231a2ae44434927855d7fd17e8e9: Bootstrapping tablet I0703 01:40:40.684414 16756 tablet_bootstrap.cc:427] T 945875de50a24af89a08d08b4354e1f4 P 2c88231a2ae44434927855d7fd17e8e9: Bootstrap starting. I0703 01:40:40.684626 16754 tablet_bootstrap.cc:427] T 945875de50a24af89a08d08b4354e1f4 P c446c114fd3c4c29b178ea4d2423b26b: No bootstrap required, opened a new log I0703 01:40:40.684823 16550 ts_tablet_manager.cc:1173] T 945875de50a24af89a08d08b4354e1f4 P 719afe22f6f64605919e0814444300c8: Registered tablet (data state: TABLET_DATA_READY) I0703 01:40:40.684960 16754 ts_tablet_manager.cc:1045] T 945875de50a24af89a08d08b4354e1f4 P c446c114fd3c4c29b178ea4d2423b26b: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.004s I0703 01:40:40.685992 16757 ts_tablet_manager.cc:1028] T 945875de50a24af89a08d08b4354e1f4 P 719afe22f6f64605919e0814444300c8: Bootstrapping tablet I0703 01:40:40.686508 16757 tablet_bootstrap.cc:427] T 945875de50a24af89a08d08b4354e1f4 P 719afe22f6f64605919e0814444300c8: Bootstrap starting. I0703 01:40:40.687616 16756 tablet_bootstrap.cc:578] T 945875de50a24af89a08d08b4354e1f4 P 2c88231a2ae44434927855d7fd17e8e9: No blocks or log segments found. Creating new log. I0703 01:40:40.688581 16756 log.cc:525] T 945875de50a24af89a08d08b4354e1f4 P 2c88231a2ae44434927855d7fd17e8e9: Log is configured to *not* fsync() on all Append() calls I0703 01:40:40.688797 16757 tablet_bootstrap.cc:578] T 945875de50a24af89a08d08b4354e1f4 P 719afe22f6f64605919e0814444300c8: No blocks or log segments found. Creating new log. I0703 01:40:40.689863 16757 log.cc:525] T 945875de50a24af89a08d08b4354e1f4 P 719afe22f6f64605919e0814444300c8: Log is configured to *not* fsync() on all Append() calls I0703 01:40:40.691146 16756 tablet_bootstrap.cc:427] T 945875de50a24af89a08d08b4354e1f4 P 2c88231a2ae44434927855d7fd17e8e9: No bootstrap required, opened a new log I0703 01:40:40.691351 16756 ts_tablet_manager.cc:1045] T 945875de50a24af89a08d08b4354e1f4 P 2c88231a2ae44434927855d7fd17e8e9: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.002s I0703 01:40:40.693555 16754 raft_consensus.cc:325] T 945875de50a24af89a08d08b4354e1f4 P c446c114fd3c4c29b178ea4d2423b26b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c446c114fd3c4c29b178ea4d2423b26b" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 34006 } } peers { permanent_uuid: "719afe22f6f64605919e0814444300c8" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 45951 } } peers { permanent_uuid: "2c88231a2ae44434927855d7fd17e8e9" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 60206 } } I0703 01:40:40.693958 16754 raft_consensus.cc:351] T 945875de50a24af89a08d08b4354e1f4 P c446c114fd3c4c29b178ea4d2423b26b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:40:40.694028 16754 raft_consensus.cc:628] T 945875de50a24af89a08d08b4354e1f4 P c446c114fd3c4c29b178ea4d2423b26b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c446c114fd3c4c29b178ea4d2423b26b, State: Initialized, Role: FOLLOWER I0703 01:40:40.694675 16754 consensus_queue.cc:227] T 945875de50a24af89a08d08b4354e1f4 P c446c114fd3c4c29b178ea4d2423b26b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c446c114fd3c4c29b178ea4d2423b26b" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 34006 } } peers { permanent_uuid: "719afe22f6f64605919e0814444300c8" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 45951 } } peers { permanent_uuid: "2c88231a2ae44434927855d7fd17e8e9" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 60206 } } I0703 01:40:40.696161 16735 heartbeater.cc:467] Master 127.5.240.254:60310 was elected leader, sending a full tablet report... I0703 01:40:40.696743 16754 ts_tablet_manager.cc:1073] T 945875de50a24af89a08d08b4354e1f4 P c446c114fd3c4c29b178ea4d2423b26b: Time spent starting tablet: real 0.012s user 0.008s sys 0.003s I0703 01:40:40.699172 16756 raft_consensus.cc:325] T 945875de50a24af89a08d08b4354e1f4 P 2c88231a2ae44434927855d7fd17e8e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c446c114fd3c4c29b178ea4d2423b26b" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 34006 } } peers { permanent_uuid: "719afe22f6f64605919e0814444300c8" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 45951 } } peers { permanent_uuid: "2c88231a2ae44434927855d7fd17e8e9" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 60206 } } I0703 01:40:40.699450 16756 raft_consensus.cc:351] T 945875de50a24af89a08d08b4354e1f4 P 2c88231a2ae44434927855d7fd17e8e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:40:40.699517 16756 raft_consensus.cc:628] T 945875de50a24af89a08d08b4354e1f4 P 2c88231a2ae44434927855d7fd17e8e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c88231a2ae44434927855d7fd17e8e9, State: Initialized, Role: FOLLOWER I0703 01:40:40.699797 16757 tablet_bootstrap.cc:427] T 945875de50a24af89a08d08b4354e1f4 P 719afe22f6f64605919e0814444300c8: No bootstrap required, opened a new log I0703 01:40:40.700034 16757 ts_tablet_manager.cc:1045] T 945875de50a24af89a08d08b4354e1f4 P 719afe22f6f64605919e0814444300c8: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.007s I0703 01:40:40.699967 16756 consensus_queue.cc:227] T 945875de50a24af89a08d08b4354e1f4 P 2c88231a2ae44434927855d7fd17e8e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c446c114fd3c4c29b178ea4d2423b26b" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 34006 } } peers { permanent_uuid: "719afe22f6f64605919e0814444300c8" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 45951 } } peers { permanent_uuid: "2c88231a2ae44434927855d7fd17e8e9" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 60206 } } I0703 01:40:40.700860 16490 heartbeater.cc:467] Master 127.5.240.254:60310 was elected leader, sending a full tablet report... I0703 01:40:40.701979 16756 ts_tablet_manager.cc:1073] T 945875de50a24af89a08d08b4354e1f4 P 2c88231a2ae44434927855d7fd17e8e9: Time spent starting tablet: real 0.011s user 0.009s sys 0.001s I0703 01:40:40.706564 16757 raft_consensus.cc:325] T 945875de50a24af89a08d08b4354e1f4 P 719afe22f6f64605919e0814444300c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c446c114fd3c4c29b178ea4d2423b26b" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 34006 } } peers { permanent_uuid: "719afe22f6f64605919e0814444300c8" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 45951 } } peers { permanent_uuid: "2c88231a2ae44434927855d7fd17e8e9" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 60206 } } I0703 01:40:40.706836 16757 raft_consensus.cc:351] T 945875de50a24af89a08d08b4354e1f4 P 719afe22f6f64605919e0814444300c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:40:40.706887 16757 raft_consensus.cc:628] T 945875de50a24af89a08d08b4354e1f4 P 719afe22f6f64605919e0814444300c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 719afe22f6f64605919e0814444300c8, State: Initialized, Role: FOLLOWER I0703 01:40:40.707262 16757 consensus_queue.cc:227] T 945875de50a24af89a08d08b4354e1f4 P 719afe22f6f64605919e0814444300c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c446c114fd3c4c29b178ea4d2423b26b" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 34006 } } peers { permanent_uuid: "719afe22f6f64605919e0814444300c8" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 45951 } } peers { permanent_uuid: "2c88231a2ae44434927855d7fd17e8e9" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 60206 } } I0703 01:40:40.707970 16613 heartbeater.cc:467] Master 127.5.240.254:60310 was elected leader, sending a full tablet report... I0703 01:40:40.708268 16757 ts_tablet_manager.cc:1073] T 945875de50a24af89a08d08b4354e1f4 P 719afe22f6f64605919e0814444300c8: Time spent starting tablet: real 0.008s user 0.006s sys 0.002s W0703 01:40:40.782801 16614 tablet.cc:1678] T 945875de50a24af89a08d08b4354e1f4 P 719afe22f6f64605919e0814444300c8: Can't schedule compaction. Clean time has not been advanced past its initial value. W0703 01:40:40.865002 16736 tablet.cc:1678] T 945875de50a24af89a08d08b4354e1f4 P c446c114fd3c4c29b178ea4d2423b26b: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:40:40.912608 16760 raft_consensus.cc:457] T 945875de50a24af89a08d08b4354e1f4 P c446c114fd3c4c29b178ea4d2423b26b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0703 01:40:40.913085 16760 raft_consensus.cc:2730] T 945875de50a24af89a08d08b4354e1f4 P c446c114fd3c4c29b178ea4d2423b26b: Snoozing failure detection for 2.051s (starting election) I0703 01:40:40.913180 16760 raft_consensus.cc:479] T 945875de50a24af89a08d08b4354e1f4 P c446c114fd3c4c29b178ea4d2423b26b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c446c114fd3c4c29b178ea4d2423b26b" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 34006 } } peers { permanent_uuid: "719afe22f6f64605919e0814444300c8" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 45951 } } peers { permanent_uuid: "2c88231a2ae44434927855d7fd17e8e9" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 60206 } } I0703 01:40:40.914321 16760 leader_election.cc:243] T 945875de50a24af89a08d08b4354e1f4 P c446c114fd3c4c29b178ea4d2423b26b [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 719afe22f6f64605919e0814444300c8 (127.5.240.194:45951) I0703 01:40:40.914638 16760 leader_election.cc:243] T 945875de50a24af89a08d08b4354e1f4 P c446c114fd3c4c29b178ea4d2423b26b [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 2c88231a2ae44434927855d7fd17e8e9 (127.5.240.193:60206) I0703 01:40:40.921667 16570 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "945875de50a24af89a08d08b4354e1f4" candidate_uuid: "c446c114fd3c4c29b178ea4d2423b26b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "719afe22f6f64605919e0814444300c8" is_pre_election: true I0703 01:40:40.921371 16447 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "945875de50a24af89a08d08b4354e1f4" candidate_uuid: "c446c114fd3c4c29b178ea4d2423b26b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c88231a2ae44434927855d7fd17e8e9" is_pre_election: true I0703 01:40:40.921965 16570 raft_consensus.cc:2730] T 945875de50a24af89a08d08b4354e1f4 P 719afe22f6f64605919e0814444300c8: Snoozing failure detection for 2.236s (vote granted) I0703 01:40:40.922081 16570 raft_consensus.cc:2243] T 945875de50a24af89a08d08b4354e1f4 P 719afe22f6f64605919e0814444300c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c446c114fd3c4c29b178ea4d2423b26b in term 0. I0703 01:40:40.922097 16447 raft_consensus.cc:2730] T 945875de50a24af89a08d08b4354e1f4 P 2c88231a2ae44434927855d7fd17e8e9: Snoozing failure detection for 1.868s (vote granted) I0703 01:40:40.922199 16447 raft_consensus.cc:2243] T 945875de50a24af89a08d08b4354e1f4 P 2c88231a2ae44434927855d7fd17e8e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c446c114fd3c4c29b178ea4d2423b26b in term 0. I0703 01:40:40.922760 16626 leader_election.cc:379] T 945875de50a24af89a08d08b4354e1f4 P c446c114fd3c4c29b178ea4d2423b26b [CANDIDATE]: Term 1 pre-election: Vote granted by peer 719afe22f6f64605919e0814444300c8 (127.5.240.194:45951) I0703 01:40:40.922868 16626 leader_election.cc:270] T 945875de50a24af89a08d08b4354e1f4 P c446c114fd3c4c29b178ea4d2423b26b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0703 01:40:40.922982 16623 leader_election.cc:379] T 945875de50a24af89a08d08b4354e1f4 P c446c114fd3c4c29b178ea4d2423b26b [CANDIDATE]: Term 1 pre-election: Vote granted by peer 2c88231a2ae44434927855d7fd17e8e9 (127.5.240.193:60206) I0703 01:40:40.923168 16760 raft_consensus.cc:2730] T 945875de50a24af89a08d08b4354e1f4 P c446c114fd3c4c29b178ea4d2423b26b: Snoozing failure detection for 1.665s (election complete) I0703 01:40:40.923244 16760 raft_consensus.cc:2537] T 945875de50a24af89a08d08b4354e1f4 P c446c114fd3c4c29b178ea4d2423b26b [term 0 FOLLOWER]: Leader pre-election won for term 1 I0703 01:40:40.923296 16760 raft_consensus.cc:457] T 945875de50a24af89a08d08b4354e1f4 P c446c114fd3c4c29b178ea4d2423b26b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0703 01:40:40.923339 16760 raft_consensus.cc:2730] T 945875de50a24af89a08d08b4354e1f4 P c446c114fd3c4c29b178ea4d2423b26b: Snoozing failure detection for 1.708s (starting election) I0703 01:40:40.923382 16760 raft_consensus.cc:2782] T 945875de50a24af89a08d08b4354e1f4 P c446c114fd3c4c29b178ea4d2423b26b [term 0 FOLLOWER]: Advancing to term 1 I0703 01:40:40.924774 16760 raft_consensus.cc:479] T 945875de50a24af89a08d08b4354e1f4 P c446c114fd3c4c29b178ea4d2423b26b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c446c114fd3c4c29b178ea4d2423b26b" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 34006 } } peers { permanent_uuid: "719afe22f6f64605919e0814444300c8" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 45951 } } peers { permanent_uuid: "2c88231a2ae44434927855d7fd17e8e9" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 60206 } } I0703 01:40:40.925076 16760 leader_election.cc:243] T 945875de50a24af89a08d08b4354e1f4 P c446c114fd3c4c29b178ea4d2423b26b [CANDIDATE]: Term 1 election: Requesting vote from peer 719afe22f6f64605919e0814444300c8 (127.5.240.194:45951) I0703 01:40:40.925220 16760 leader_election.cc:243] T 945875de50a24af89a08d08b4354e1f4 P c446c114fd3c4c29b178ea4d2423b26b [CANDIDATE]: Term 1 election: Requesting vote from peer 2c88231a2ae44434927855d7fd17e8e9 (127.5.240.193:60206) I0703 01:40:40.925762 16570 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "945875de50a24af89a08d08b4354e1f4" candidate_uuid: "c446c114fd3c4c29b178ea4d2423b26b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "719afe22f6f64605919e0814444300c8" I0703 01:40:40.925859 16447 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "945875de50a24af89a08d08b4354e1f4" candidate_uuid: "c446c114fd3c4c29b178ea4d2423b26b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c88231a2ae44434927855d7fd17e8e9" I0703 01:40:40.925887 16570 raft_consensus.cc:2782] T 945875de50a24af89a08d08b4354e1f4 P 719afe22f6f64605919e0814444300c8 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:40:40.925966 16570 raft_consensus.cc:2730] T 945875de50a24af89a08d08b4354e1f4 P 719afe22f6f64605919e0814444300c8: Snoozing failure detection for 1.869s (vote granted) I0703 01:40:40.925964 16447 raft_consensus.cc:2782] T 945875de50a24af89a08d08b4354e1f4 P 2c88231a2ae44434927855d7fd17e8e9 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:40:40.926030 16447 raft_consensus.cc:2730] T 945875de50a24af89a08d08b4354e1f4 P 2c88231a2ae44434927855d7fd17e8e9: Snoozing failure detection for 2.194s (vote granted) I0703 01:40:40.927311 16570 raft_consensus.cc:2243] T 945875de50a24af89a08d08b4354e1f4 P 719afe22f6f64605919e0814444300c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c446c114fd3c4c29b178ea4d2423b26b in term 1. I0703 01:40:40.927711 16626 leader_election.cc:379] T 945875de50a24af89a08d08b4354e1f4 P c446c114fd3c4c29b178ea4d2423b26b [CANDIDATE]: Term 1 election: Vote granted by peer 719afe22f6f64605919e0814444300c8 (127.5.240.194:45951) I0703 01:40:40.927747 16447 raft_consensus.cc:2243] T 945875de50a24af89a08d08b4354e1f4 P 2c88231a2ae44434927855d7fd17e8e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c446c114fd3c4c29b178ea4d2423b26b in term 1. I0703 01:40:40.927768 16626 leader_election.cc:270] T 945875de50a24af89a08d08b4354e1f4 P c446c114fd3c4c29b178ea4d2423b26b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:40:40.927953 16760 raft_consensus.cc:2730] T 945875de50a24af89a08d08b4354e1f4 P c446c114fd3c4c29b178ea4d2423b26b: Snoozing failure detection for 2.100s (election complete) I0703 01:40:40.928084 16760 raft_consensus.cc:2537] T 945875de50a24af89a08d08b4354e1f4 P c446c114fd3c4c29b178ea4d2423b26b [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:40:40.928203 16623 leader_election.cc:379] T 945875de50a24af89a08d08b4354e1f4 P c446c114fd3c4c29b178ea4d2423b26b [CANDIDATE]: Term 1 election: Vote granted by peer 2c88231a2ae44434927855d7fd17e8e9 (127.5.240.193:60206) I0703 01:40:40.929141 16760 raft_consensus.cc:593] T 945875de50a24af89a08d08b4354e1f4 P c446c114fd3c4c29b178ea4d2423b26b [term 1 LEADER]: Becoming Leader. State: Replica: c446c114fd3c4c29b178ea4d2423b26b, State: Running, Role: LEADER I0703 01:40:40.929574 16760 consensus_queue.cc:204] T 945875de50a24af89a08d08b4354e1f4 P c446c114fd3c4c29b178ea4d2423b26b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c446c114fd3c4c29b178ea4d2423b26b" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 34006 } } peers { permanent_uuid: "719afe22f6f64605919e0814444300c8" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 45951 } } peers { permanent_uuid: "2c88231a2ae44434927855d7fd17e8e9" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 60206 } } I0703 01:40:40.933709 16327 catalog_manager.cc:3798] T 945875de50a24af89a08d08b4354e1f4 P c446c114fd3c4c29b178ea4d2423b26b reported cstate change: term changed from 0 to 1, leader changed from to c446c114fd3c4c29b178ea4d2423b26b (127.5.240.195). New cstate: current_term: 1 leader_uuid: "c446c114fd3c4c29b178ea4d2423b26b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c446c114fd3c4c29b178ea4d2423b26b" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 34006 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "719afe22f6f64605919e0814444300c8" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 45951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c88231a2ae44434927855d7fd17e8e9" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 60206 } health_report { overall_health: UNKNOWN } } } W0703 01:40:40.944564 16491 tablet.cc:1678] T 945875de50a24af89a08d08b4354e1f4 P 2c88231a2ae44434927855d7fd17e8e9: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:40:40.950598 6083 external_mini_cluster.cc:490] 3 TS(s) registered with all masters I0703 01:40:40.951763 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 2c88231a2ae44434927855d7fd17e8e9 to finish bootstrapping I0703 01:40:40.957267 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 719afe22f6f64605919e0814444300c8 to finish bootstrapping I0703 01:40:40.962095 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver c446c114fd3c4c29b178ea4d2423b26b to finish bootstrapping I0703 01:40:40.967048 16327 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.5.240.254:43854: name: "TestAnotherTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } W0703 01:40:40.967578 16327 catalog_manager.cc:1488] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestAnotherTable in case of a replica failure: 4 tablet servers are needed, 3 are alive. Consider bringing up additional tablet server(s) or running both the masters and all tablet servers with --raft_prepare_replacement_before_eviction=false flag (not recommended). I0703 01:40:40.972137 16427 tablet_service.cc:759] Processing CreateTablet for tablet 9809ab0e16b24dbcae587c5bfd1425d0 (table=TestAnotherTable [id=c43d2c61ec174dac98543232351639b4]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:40:40.972352 16672 tablet_service.cc:759] Processing CreateTablet for tablet 9809ab0e16b24dbcae587c5bfd1425d0 (table=TestAnotherTable [id=c43d2c61ec174dac98543232351639b4]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:40:40.972415 16427 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 9809ab0e16b24dbcae587c5bfd1425d0. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:40:40.972645 16672 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 9809ab0e16b24dbcae587c5bfd1425d0. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:40:40.975256 16427 ts_tablet_manager.cc:1173] T 9809ab0e16b24dbcae587c5bfd1425d0 P 2c88231a2ae44434927855d7fd17e8e9: Registered tablet (data state: TABLET_DATA_READY) I0703 01:40:40.975461 16756 ts_tablet_manager.cc:1028] T 9809ab0e16b24dbcae587c5bfd1425d0 P 2c88231a2ae44434927855d7fd17e8e9: Bootstrapping tablet I0703 01:40:40.975662 16756 tablet_bootstrap.cc:427] T 9809ab0e16b24dbcae587c5bfd1425d0 P 2c88231a2ae44434927855d7fd17e8e9: Bootstrap starting. I0703 01:40:40.975788 16550 tablet_service.cc:759] Processing CreateTablet for tablet 9809ab0e16b24dbcae587c5bfd1425d0 (table=TestAnotherTable [id=c43d2c61ec174dac98543232351639b4]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:40:40.976028 16550 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 9809ab0e16b24dbcae587c5bfd1425d0. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:40:40.976707 16756 tablet_bootstrap.cc:578] T 9809ab0e16b24dbcae587c5bfd1425d0 P 2c88231a2ae44434927855d7fd17e8e9: No blocks or log segments found. Creating new log. I0703 01:40:40.977605 16672 ts_tablet_manager.cc:1173] T 9809ab0e16b24dbcae587c5bfd1425d0 P c446c114fd3c4c29b178ea4d2423b26b: Registered tablet (data state: TABLET_DATA_READY) I0703 01:40:40.977771 16754 ts_tablet_manager.cc:1028] T 9809ab0e16b24dbcae587c5bfd1425d0 P c446c114fd3c4c29b178ea4d2423b26b: Bootstrapping tablet I0703 01:40:40.977934 16754 tablet_bootstrap.cc:427] T 9809ab0e16b24dbcae587c5bfd1425d0 P c446c114fd3c4c29b178ea4d2423b26b: Bootstrap starting. I0703 01:40:40.979089 16754 tablet_bootstrap.cc:578] T 9809ab0e16b24dbcae587c5bfd1425d0 P c446c114fd3c4c29b178ea4d2423b26b: No blocks or log segments found. Creating new log. I0703 01:40:40.979112 16550 ts_tablet_manager.cc:1173] T 9809ab0e16b24dbcae587c5bfd1425d0 P 719afe22f6f64605919e0814444300c8: Registered tablet (data state: TABLET_DATA_READY) I0703 01:40:40.979270 16757 ts_tablet_manager.cc:1028] T 9809ab0e16b24dbcae587c5bfd1425d0 P 719afe22f6f64605919e0814444300c8: Bootstrapping tablet I0703 01:40:40.979395 16757 tablet_bootstrap.cc:427] T 9809ab0e16b24dbcae587c5bfd1425d0 P 719afe22f6f64605919e0814444300c8: Bootstrap starting. I0703 01:40:40.979996 16756 tablet_bootstrap.cc:427] T 9809ab0e16b24dbcae587c5bfd1425d0 P 2c88231a2ae44434927855d7fd17e8e9: No bootstrap required, opened a new log I0703 01:40:40.980094 16756 ts_tablet_manager.cc:1045] T 9809ab0e16b24dbcae587c5bfd1425d0 P 2c88231a2ae44434927855d7fd17e8e9: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s I0703 01:40:40.980691 16757 tablet_bootstrap.cc:578] T 9809ab0e16b24dbcae587c5bfd1425d0 P 719afe22f6f64605919e0814444300c8: No blocks or log segments found. Creating new log. I0703 01:40:40.980814 16756 raft_consensus.cc:325] T 9809ab0e16b24dbcae587c5bfd1425d0 P 2c88231a2ae44434927855d7fd17e8e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c88231a2ae44434927855d7fd17e8e9" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 60206 } } peers { permanent_uuid: "719afe22f6f64605919e0814444300c8" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 45951 } } peers { permanent_uuid: "c446c114fd3c4c29b178ea4d2423b26b" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 34006 } } I0703 01:40:40.980955 16756 raft_consensus.cc:351] T 9809ab0e16b24dbcae587c5bfd1425d0 P 2c88231a2ae44434927855d7fd17e8e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:40:40.981006 16756 raft_consensus.cc:628] T 9809ab0e16b24dbcae587c5bfd1425d0 P 2c88231a2ae44434927855d7fd17e8e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c88231a2ae44434927855d7fd17e8e9, State: Initialized, Role: FOLLOWER I0703 01:40:40.981202 16756 consensus_queue.cc:227] T 9809ab0e16b24dbcae587c5bfd1425d0 P 2c88231a2ae44434927855d7fd17e8e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c88231a2ae44434927855d7fd17e8e9" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 60206 } } peers { permanent_uuid: "719afe22f6f64605919e0814444300c8" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 45951 } } peers { permanent_uuid: "c446c114fd3c4c29b178ea4d2423b26b" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 34006 } } I0703 01:40:40.981389 16754 tablet_bootstrap.cc:427] T 9809ab0e16b24dbcae587c5bfd1425d0 P c446c114fd3c4c29b178ea4d2423b26b: No bootstrap required, opened a new log I0703 01:40:40.981482 16754 ts_tablet_manager.cc:1045] T 9809ab0e16b24dbcae587c5bfd1425d0 P c446c114fd3c4c29b178ea4d2423b26b: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s I0703 01:40:40.982589 16757 tablet_bootstrap.cc:427] T 9809ab0e16b24dbcae587c5bfd1425d0 P 719afe22f6f64605919e0814444300c8: No bootstrap required, opened a new log I0703 01:40:40.982641 16756 ts_tablet_manager.cc:1073] T 9809ab0e16b24dbcae587c5bfd1425d0 P 2c88231a2ae44434927855d7fd17e8e9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s I0703 01:40:40.982704 16757 ts_tablet_manager.cc:1045] T 9809ab0e16b24dbcae587c5bfd1425d0 P 719afe22f6f64605919e0814444300c8: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.002s I0703 01:40:40.982905 16754 raft_consensus.cc:325] T 9809ab0e16b24dbcae587c5bfd1425d0 P c446c114fd3c4c29b178ea4d2423b26b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c88231a2ae44434927855d7fd17e8e9" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 60206 } } peers { permanent_uuid: "719afe22f6f64605919e0814444300c8" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 45951 } } peers { permanent_uuid: "c446c114fd3c4c29b178ea4d2423b26b" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 34006 } } I0703 01:40:40.983083 16754 raft_consensus.cc:351] T 9809ab0e16b24dbcae587c5bfd1425d0 P c446c114fd3c4c29b178ea4d2423b26b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:40:40.983144 16754 raft_consensus.cc:628] T 9809ab0e16b24dbcae587c5bfd1425d0 P c446c114fd3c4c29b178ea4d2423b26b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c446c114fd3c4c29b178ea4d2423b26b, State: Initialized, Role: FOLLOWER I0703 01:40:40.983284 16757 raft_consensus.cc:325] T 9809ab0e16b24dbcae587c5bfd1425d0 P 719afe22f6f64605919e0814444300c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c88231a2ae44434927855d7fd17e8e9" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 60206 } } peers { permanent_uuid: "719afe22f6f64605919e0814444300c8" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 45951 } } peers { permanent_uuid: "c446c114fd3c4c29b178ea4d2423b26b" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 34006 } } I0703 01:40:40.983418 16757 raft_consensus.cc:351] T 9809ab0e16b24dbcae587c5bfd1425d0 P 719afe22f6f64605919e0814444300c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:40:40.983299 16754 consensus_queue.cc:227] T 9809ab0e16b24dbcae587c5bfd1425d0 P c446c114fd3c4c29b178ea4d2423b26b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c88231a2ae44434927855d7fd17e8e9" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 60206 } } peers { permanent_uuid: "719afe22f6f64605919e0814444300c8" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 45951 } } peers { permanent_uuid: "c446c114fd3c4c29b178ea4d2423b26b" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 34006 } } I0703 01:40:40.983470 16757 raft_consensus.cc:628] T 9809ab0e16b24dbcae587c5bfd1425d0 P 719afe22f6f64605919e0814444300c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 719afe22f6f64605919e0814444300c8, State: Initialized, Role: FOLLOWER I0703 01:40:40.983925 16754 ts_tablet_manager.cc:1073] T 9809ab0e16b24dbcae587c5bfd1425d0 P c446c114fd3c4c29b178ea4d2423b26b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s I0703 01:40:40.984084 16757 consensus_queue.cc:227] T 9809ab0e16b24dbcae587c5bfd1425d0 P 719afe22f6f64605919e0814444300c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c88231a2ae44434927855d7fd17e8e9" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 60206 } } peers { permanent_uuid: "719afe22f6f64605919e0814444300c8" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 45951 } } peers { permanent_uuid: "c446c114fd3c4c29b178ea4d2423b26b" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 34006 } } I0703 01:40:40.984725 16757 ts_tablet_manager.cc:1073] T 9809ab0e16b24dbcae587c5bfd1425d0 P 719afe22f6f64605919e0814444300c8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s I0703 01:40:41.106843 16762 raft_consensus.cc:457] T 9809ab0e16b24dbcae587c5bfd1425d0 P 719afe22f6f64605919e0814444300c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0703 01:40:41.106952 16762 raft_consensus.cc:2730] T 9809ab0e16b24dbcae587c5bfd1425d0 P 719afe22f6f64605919e0814444300c8: Snoozing failure detection for 2.158s (starting election) I0703 01:40:41.107065 16762 raft_consensus.cc:479] T 9809ab0e16b24dbcae587c5bfd1425d0 P 719afe22f6f64605919e0814444300c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c88231a2ae44434927855d7fd17e8e9" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 60206 } } peers { permanent_uuid: "719afe22f6f64605919e0814444300c8" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 45951 } } peers { permanent_uuid: "c446c114fd3c4c29b178ea4d2423b26b" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 34006 } } I0703 01:40:41.107853 16762 leader_election.cc:243] T 9809ab0e16b24dbcae587c5bfd1425d0 P 719afe22f6f64605919e0814444300c8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 2c88231a2ae44434927855d7fd17e8e9 (127.5.240.193:60206) I0703 01:40:41.108176 16762 leader_election.cc:243] T 9809ab0e16b24dbcae587c5bfd1425d0 P 719afe22f6f64605919e0814444300c8 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer c446c114fd3c4c29b178ea4d2423b26b (127.5.240.195:34006) I0703 01:40:41.116448 16447 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "9809ab0e16b24dbcae587c5bfd1425d0" candidate_uuid: "719afe22f6f64605919e0814444300c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c88231a2ae44434927855d7fd17e8e9" is_pre_election: true I0703 01:40:41.116612 16447 raft_consensus.cc:2730] T 9809ab0e16b24dbcae587c5bfd1425d0 P 2c88231a2ae44434927855d7fd17e8e9: Snoozing failure detection for 1.661s (vote granted) I0703 01:40:41.116677 16447 raft_consensus.cc:2243] T 9809ab0e16b24dbcae587c5bfd1425d0 P 2c88231a2ae44434927855d7fd17e8e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 719afe22f6f64605919e0814444300c8 in term 0. I0703 01:40:41.116858 16692 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "9809ab0e16b24dbcae587c5bfd1425d0" candidate_uuid: "719afe22f6f64605919e0814444300c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c446c114fd3c4c29b178ea4d2423b26b" is_pre_election: true I0703 01:40:41.117115 16692 raft_consensus.cc:2730] T 9809ab0e16b24dbcae587c5bfd1425d0 P c446c114fd3c4c29b178ea4d2423b26b: Snoozing failure detection for 1.519s (vote granted) I0703 01:40:41.117146 16501 leader_election.cc:379] T 9809ab0e16b24dbcae587c5bfd1425d0 P 719afe22f6f64605919e0814444300c8 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 2c88231a2ae44434927855d7fd17e8e9 (127.5.240.193:60206) I0703 01:40:41.117224 16501 leader_election.cc:270] T 9809ab0e16b24dbcae587c5bfd1425d0 P 719afe22f6f64605919e0814444300c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0703 01:40:41.117204 16692 raft_consensus.cc:2243] T 9809ab0e16b24dbcae587c5bfd1425d0 P c446c114fd3c4c29b178ea4d2423b26b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 719afe22f6f64605919e0814444300c8 in term 0. I0703 01:40:41.117671 16762 raft_consensus.cc:2730] T 9809ab0e16b24dbcae587c5bfd1425d0 P 719afe22f6f64605919e0814444300c8: Snoozing failure detection for 1.869s (election complete) I0703 01:40:41.117813 16762 raft_consensus.cc:2537] T 9809ab0e16b24dbcae587c5bfd1425d0 P 719afe22f6f64605919e0814444300c8 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0703 01:40:41.117815 16501 leader_election.cc:379] T 9809ab0e16b24dbcae587c5bfd1425d0 P 719afe22f6f64605919e0814444300c8 [CANDIDATE]: Term 1 pre-election: Vote granted by peer c446c114fd3c4c29b178ea4d2423b26b (127.5.240.195:34006) I0703 01:40:41.117892 16762 raft_consensus.cc:457] T 9809ab0e16b24dbcae587c5bfd1425d0 P 719afe22f6f64605919e0814444300c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0703 01:40:41.117980 16762 raft_consensus.cc:2730] T 9809ab0e16b24dbcae587c5bfd1425d0 P 719afe22f6f64605919e0814444300c8: Snoozing failure detection for 1.685s (starting election) I0703 01:40:41.118062 16762 raft_consensus.cc:2782] T 9809ab0e16b24dbcae587c5bfd1425d0 P 719afe22f6f64605919e0814444300c8 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:40:41.120182 16762 raft_consensus.cc:479] T 9809ab0e16b24dbcae587c5bfd1425d0 P 719afe22f6f64605919e0814444300c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c88231a2ae44434927855d7fd17e8e9" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 60206 } } peers { permanent_uuid: "719afe22f6f64605919e0814444300c8" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 45951 } } peers { permanent_uuid: "c446c114fd3c4c29b178ea4d2423b26b" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 34006 } } I0703 01:40:41.120506 16762 leader_election.cc:243] T 9809ab0e16b24dbcae587c5bfd1425d0 P 719afe22f6f64605919e0814444300c8 [CANDIDATE]: Term 1 election: Requesting vote from peer 2c88231a2ae44434927855d7fd17e8e9 (127.5.240.193:60206) I0703 01:40:41.120714 16762 leader_election.cc:243] T 9809ab0e16b24dbcae587c5bfd1425d0 P 719afe22f6f64605919e0814444300c8 [CANDIDATE]: Term 1 election: Requesting vote from peer c446c114fd3c4c29b178ea4d2423b26b (127.5.240.195:34006) I0703 01:40:41.121201 16447 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "9809ab0e16b24dbcae587c5bfd1425d0" candidate_uuid: "719afe22f6f64605919e0814444300c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c88231a2ae44434927855d7fd17e8e9" I0703 01:40:41.121330 16447 raft_consensus.cc:2782] T 9809ab0e16b24dbcae587c5bfd1425d0 P 2c88231a2ae44434927855d7fd17e8e9 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:40:41.121402 16447 raft_consensus.cc:2730] T 9809ab0e16b24dbcae587c5bfd1425d0 P 2c88231a2ae44434927855d7fd17e8e9: Snoozing failure detection for 1.746s (vote granted) I0703 01:40:41.121439 16692 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "9809ab0e16b24dbcae587c5bfd1425d0" candidate_uuid: "719afe22f6f64605919e0814444300c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c446c114fd3c4c29b178ea4d2423b26b" I0703 01:40:41.121539 16692 raft_consensus.cc:2782] T 9809ab0e16b24dbcae587c5bfd1425d0 P c446c114fd3c4c29b178ea4d2423b26b [term 0 FOLLOWER]: Advancing to term 1 I0703 01:40:41.121608 16692 raft_consensus.cc:2730] T 9809ab0e16b24dbcae587c5bfd1425d0 P c446c114fd3c4c29b178ea4d2423b26b: Snoozing failure detection for 2.199s (vote granted) I0703 01:40:41.122675 16447 raft_consensus.cc:2243] T 9809ab0e16b24dbcae587c5bfd1425d0 P 2c88231a2ae44434927855d7fd17e8e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 719afe22f6f64605919e0814444300c8 in term 1. I0703 01:40:41.122891 16692 raft_consensus.cc:2243] T 9809ab0e16b24dbcae587c5bfd1425d0 P c446c114fd3c4c29b178ea4d2423b26b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 719afe22f6f64605919e0814444300c8 in term 1. I0703 01:40:41.123281 16501 leader_election.cc:379] T 9809ab0e16b24dbcae587c5bfd1425d0 P 719afe22f6f64605919e0814444300c8 [CANDIDATE]: Term 1 election: Vote granted by peer 2c88231a2ae44434927855d7fd17e8e9 (127.5.240.193:60206) I0703 01:40:41.123344 16501 leader_election.cc:270] T 9809ab0e16b24dbcae587c5bfd1425d0 P 719afe22f6f64605919e0814444300c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:40:41.123499 16762 raft_consensus.cc:2730] T 9809ab0e16b24dbcae587c5bfd1425d0 P 719afe22f6f64605919e0814444300c8: Snoozing failure detection for 2.115s (election complete) I0703 01:40:41.123631 16501 leader_election.cc:379] T 9809ab0e16b24dbcae587c5bfd1425d0 P 719afe22f6f64605919e0814444300c8 [CANDIDATE]: Term 1 election: Vote granted by peer c446c114fd3c4c29b178ea4d2423b26b (127.5.240.195:34006) I0703 01:40:41.123634 16762 raft_consensus.cc:2537] T 9809ab0e16b24dbcae587c5bfd1425d0 P 719afe22f6f64605919e0814444300c8 [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:40:41.123993 16762 raft_consensus.cc:593] T 9809ab0e16b24dbcae587c5bfd1425d0 P 719afe22f6f64605919e0814444300c8 [term 1 LEADER]: Becoming Leader. State: Replica: 719afe22f6f64605919e0814444300c8, State: Running, Role: LEADER I0703 01:40:41.124364 16762 consensus_queue.cc:204] T 9809ab0e16b24dbcae587c5bfd1425d0 P 719afe22f6f64605919e0814444300c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c88231a2ae44434927855d7fd17e8e9" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 60206 } } peers { permanent_uuid: "719afe22f6f64605919e0814444300c8" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 45951 } } peers { permanent_uuid: "c446c114fd3c4c29b178ea4d2423b26b" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 34006 } } I0703 01:40:41.128396 16326 catalog_manager.cc:3798] T 9809ab0e16b24dbcae587c5bfd1425d0 P 719afe22f6f64605919e0814444300c8 reported cstate change: term changed from 0 to 1, leader changed from to 719afe22f6f64605919e0814444300c8 (127.5.240.194). New cstate: current_term: 1 leader_uuid: "719afe22f6f64605919e0814444300c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c88231a2ae44434927855d7fd17e8e9" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 60206 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "719afe22f6f64605919e0814444300c8" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 45951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c446c114fd3c4c29b178ea4d2423b26b" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 34006 } health_report { overall_health: UNKNOWN } } } I0703 01:40:41.346137 16447 raft_consensus.cc:1081] T 945875de50a24af89a08d08b4354e1f4 P 2c88231a2ae44434927855d7fd17e8e9 [term 1 FOLLOWER]: Refusing update from remote peer c446c114fd3c4c29b178ea4d2423b26b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0703 01:40:41.347205 16760 consensus_queue.cc:962] T 945875de50a24af89a08d08b4354e1f4 P c446c114fd3c4c29b178ea4d2423b26b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c88231a2ae44434927855d7fd17e8e9" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 60206 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0703 01:40:41.354596 16570 raft_consensus.cc:1081] T 945875de50a24af89a08d08b4354e1f4 P 719afe22f6f64605919e0814444300c8 [term 1 FOLLOWER]: Refusing update from remote peer c446c114fd3c4c29b178ea4d2423b26b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0703 01:40:41.356703 16760 consensus_queue.cc:962] T 945875de50a24af89a08d08b4354e1f4 P c446c114fd3c4c29b178ea4d2423b26b [LEADER]: Connected to new peer: Peer: permanent_uuid: "719afe22f6f64605919e0814444300c8" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 45951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s W0703 01:40:41.376574 16774 flags.cc:406] Enabled unsafe flag: --never_fsync=true I0703 01:40:41.472266 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 16361 I0703 01:40:41.475062 16487 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:40:41.523602 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 16361 W0703 01:40:41.533341 16310 connection.cc:512] server connection from 127.5.240.193:44201 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:44201): Connection reset by peer (error 104) W0703 01:40:41.533555 16623 connection.cc:512] client connection to 127.5.240.193:60206 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:60206): Connection reset by peer (error 104) W0703 01:40:41.533589 16501 connection.cc:512] client connection to 127.5.240.193:60206 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:60206): Connection reset by peer (error 104) I0703 01:40:41.534431 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 16492 I0703 01:40:41.535140 16610 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:40:41.579890 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 16492 I0703 01:40:41.580727 16692 raft_consensus.cc:1081] T 9809ab0e16b24dbcae587c5bfd1425d0 P c446c114fd3c4c29b178ea4d2423b26b [term 1 FOLLOWER]: Refusing update from remote peer 719afe22f6f64605919e0814444300c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) W0703 01:40:41.588922 16626 connection.cc:512] client connection to 127.5.240.194:45951 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:45951): Connection reset by peer (error 104) I0703 01:40:41.589066 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 16616 W0703 01:40:41.589040 16311 connection.cc:512] server connection from 127.5.240.194:54832 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:54832): Connection reset by peer (error 104) W0703 01:40:41.589295 16625 connection.cc:512] server connection from 127.5.240.194:44901 recv error: Network error: failed to read from TLS socket (remote: 0.0.0.0:0): Connection reset by peer (error 104) W0703 01:40:41.589556 16625 connection.cc:185] Error closing socket: Network error: TlsSocket::Close: Broken pipe I0703 01:40:41.590011 16732 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:40:41.635615 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 16616 I0703 01:40:41.644070 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master pid 16301 W0703 01:40:41.644021 16308 connection.cc:512] server connection from 127.5.240.195:50286 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.195:50286): Connection reset by peer (error 104) I0703 01:40:41.644824 16357 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:40:41.706707 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master with pid 16301 [ OK ] AdminCliTest.TestListTablesDetail (2444 ms) [----------] 14 tests from AdminCliTest (84101 ms total) [----------] 6 tests from EnableKudu1097AndDownTS/MoveTabletParamTest [ RUN ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/0 I0703 01:40:41.729144 6083 test_util.cc:208] Using random seed: 1156519019 I0703 01:40:41.730935 6083 ts_itest-base.cc:113] Starting cluster with: I0703 01:40:41.730958 6083 ts_itest-base.cc:114] -------------- I0703 01:40:41.730975 6083 ts_itest-base.cc:115] 5 tablet servers I0703 01:40:41.730986 6083 ts_itest-base.cc:116] 3 replicas per TS I0703 01:40:41.731003 6083 ts_itest-base.cc:117] -------------- I0703 01:40:41.732102 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:36679 --webserver_interface=127.5.240.254 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:40:41.912675 16794 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:40:41.912881 16794 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:40:41.915069 16794 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0703 01:40:41.915118 16794 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0703 01:40:41.915148 16794 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:40:41.915170 16794 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0703 01:40:41.915199 16794 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0703 01:40:41.922508 16794 master_main.cc:73] Master server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:36679 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.5.240.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:40:41.928125 16794 master_main.cc:80] Initializing master server... I0703 01:40:41.928817 16794 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 174208us I0703 01:40:41.929847 16794 fs_manager.cc:261] Metadata directory not provided I0703 01:40:41.929906 16794 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:40:41.930501 16794 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:40:41.930578 16794 server_base.cc:433] Attempting to create new FS layout instead I0703 01:40:41.934751 16794 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/instance: uuid: "86eddb576377425fadf1b1bd8fbab03b" format_stamp: "Formatted at 2018-07-03 08:40:41 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:41.935242 16794 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "86eddb576377425fadf1b1bd8fbab03b" format_stamp: "Formatted at 2018-07-03 08:40:41 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:41.943897 16794 fs_manager.cc:501] Time spent creating directory manager: real 0.008s user 0.004s sys 0.003s I0703 01:40:41.945086 16794 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:40:41.945857 16794 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:40:41.949033 16794 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0703 01:40:41.949079 16794 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal uuid: "86eddb576377425fadf1b1bd8fbab03b" format_stamp: "Formatted at 2018-07-03 08:40:41 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:41.949180 16794 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:40:41.984917 16794 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:40:41.985889 16794 master_main.cc:83] Starting Master server... I0703 01:40:42.022096 16794 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.254:36679 I0703 01:40:42.022560 16794 webserver.cc:175] Starting webserver on 127.5.240.254:0 I0703 01:40:42.022615 16794 webserver.cc:186] Document root disabled I0703 01:40:42.023800 16794 webserver.cc:313] Webserver started. Bound to: http://127.5.240.254:45779/ I0703 01:40:42.024369 16794 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb I0703 01:40:42.028012 16853 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:40:42.033298 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master as pid 16794 I0703 01:40:42.034324 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.5.240.193:0 --local_ip_for_outbound_sockets=127.5.240.193 --webserver_interface=127.5.240.193 --webserver_port=0 --tserver_master_addrs=127.5.240.254:36679 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0703 01:40:42.040984 16853 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 86eddb576377425fadf1b1bd8fbab03b: Bootstrap starting. I0703 01:40:42.042999 16853 tablet_bootstrap.cc:578] T 00000000000000000000000000000000 P 86eddb576377425fadf1b1bd8fbab03b: No blocks or log segments found. Creating new log. I0703 01:40:42.043753 16853 log.cc:525] T 00000000000000000000000000000000 P 86eddb576377425fadf1b1bd8fbab03b: Log is configured to *not* fsync() on all Append() calls I0703 01:40:42.045975 16853 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 86eddb576377425fadf1b1bd8fbab03b: No bootstrap required, opened a new log I0703 01:40:42.052129 16853 raft_consensus.cc:325] T 00000000000000000000000000000000 P 86eddb576377425fadf1b1bd8fbab03b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86eddb576377425fadf1b1bd8fbab03b" member_type: VOTER } I0703 01:40:42.052321 16853 raft_consensus.cc:351] T 00000000000000000000000000000000 P 86eddb576377425fadf1b1bd8fbab03b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:40:42.052382 16853 raft_consensus.cc:628] T 00000000000000000000000000000000 P 86eddb576377425fadf1b1bd8fbab03b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86eddb576377425fadf1b1bd8fbab03b, State: Initialized, Role: FOLLOWER I0703 01:40:42.052865 16853 consensus_queue.cc:227] T 00000000000000000000000000000000 P 86eddb576377425fadf1b1bd8fbab03b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86eddb576377425fadf1b1bd8fbab03b" member_type: VOTER } I0703 01:40:42.053010 16853 raft_consensus.cc:365] T 00000000000000000000000000000000 P 86eddb576377425fadf1b1bd8fbab03b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:40:42.053086 16853 raft_consensus.cc:457] T 00000000000000000000000000000000 P 86eddb576377425fadf1b1bd8fbab03b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:40:42.053165 16853 raft_consensus.cc:2730] T 00000000000000000000000000000000 P 86eddb576377425fadf1b1bd8fbab03b: Snoozing failure detection for 1.930s (starting election) I0703 01:40:42.053247 16853 raft_consensus.cc:2782] T 00000000000000000000000000000000 P 86eddb576377425fadf1b1bd8fbab03b [term 0 FOLLOWER]: Advancing to term 1 I0703 01:40:42.054558 16853 raft_consensus.cc:479] T 00000000000000000000000000000000 P 86eddb576377425fadf1b1bd8fbab03b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86eddb576377425fadf1b1bd8fbab03b" member_type: VOTER } I0703 01:40:42.054936 16853 leader_election.cc:270] T 00000000000000000000000000000000 P 86eddb576377425fadf1b1bd8fbab03b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:40:42.056335 16856 raft_consensus.cc:2730] T 00000000000000000000000000000000 P 86eddb576377425fadf1b1bd8fbab03b: Snoozing failure detection for 1.737s (election complete) I0703 01:40:42.056756 16856 raft_consensus.cc:2537] T 00000000000000000000000000000000 P 86eddb576377425fadf1b1bd8fbab03b [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:40:42.058238 16856 raft_consensus.cc:593] T 00000000000000000000000000000000 P 86eddb576377425fadf1b1bd8fbab03b [term 1 LEADER]: Becoming Leader. State: Replica: 86eddb576377425fadf1b1bd8fbab03b, State: Running, Role: LEADER I0703 01:40:42.058655 16856 consensus_queue.cc:204] T 00000000000000000000000000000000 P 86eddb576377425fadf1b1bd8fbab03b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86eddb576377425fadf1b1bd8fbab03b" member_type: VOTER } I0703 01:40:42.061552 16853 sys_catalog.cc:337] T 00000000000000000000000000000000 P 86eddb576377425fadf1b1bd8fbab03b [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "86eddb576377425fadf1b1bd8fbab03b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86eddb576377425fadf1b1bd8fbab03b" member_type: VOTER } } I0703 01:40:42.061292 16857 sys_catalog.cc:337] T 00000000000000000000000000000000 P 86eddb576377425fadf1b1bd8fbab03b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86eddb576377425fadf1b1bd8fbab03b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86eddb576377425fadf1b1bd8fbab03b" member_type: VOTER } } I0703 01:40:42.061550 16858 sys_catalog.cc:337] T 00000000000000000000000000000000 P 86eddb576377425fadf1b1bd8fbab03b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86eddb576377425fadf1b1bd8fbab03b. Latest consensus state: current_term: 1 leader_uuid: "86eddb576377425fadf1b1bd8fbab03b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86eddb576377425fadf1b1bd8fbab03b" member_type: VOTER } } I0703 01:40:42.062494 16858 sys_catalog.cc:340] T 00000000000000000000000000000000 P 86eddb576377425fadf1b1bd8fbab03b [sys.catalog]: This master's current role is: LEADER I0703 01:40:42.062492 16853 sys_catalog.cc:340] T 00000000000000000000000000000000 P 86eddb576377425fadf1b1bd8fbab03b [sys.catalog]: This master's current role is: LEADER I0703 01:40:42.062494 16857 sys_catalog.cc:340] T 00000000000000000000000000000000 P 86eddb576377425fadf1b1bd8fbab03b [sys.catalog]: This master's current role is: LEADER I0703 01:40:42.064707 16853 sys_catalog.cc:424] T 00000000000000000000000000000000 P 86eddb576377425fadf1b1bd8fbab03b [sys.catalog]: configured and running, proceeding with master startup. I0703 01:40:42.064951 16860 catalog_manager.cc:1006] Loading table and tablet metadata into memory... I0703 01:40:42.066370 16794 master_main.cc:86] Master server successfully started. I0703 01:40:42.072252 16860 catalog_manager.cc:1017] Initializing Kudu internal certificate authority... I0703 01:40:42.098219 16860 catalog_manager.cc:904] Generated new certificate authority record I0703 01:40:42.100181 16860 catalog_manager.cc:1026] Loading token signing keys... I0703 01:40:42.109901 16860 catalog_manager.cc:4102] T 00000000000000000000000000000000 P 86eddb576377425fadf1b1bd8fbab03b: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:40:42.204766 16854 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:40:42.205008 16854 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:40:42.206982 16854 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0703 01:40:42.207046 16854 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:40:42.207156 16854 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.193 I0703 01:40:42.214378 16854 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.5.240.193 --webserver_port=0 --tserver_master_addrs=127.5.240.254:36679 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.193 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:40:42.220988 16854 tablet_server_main.cc:85] Initializing tablet server... I0703 01:40:42.222602 16854 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 174708us I0703 01:40:42.223178 16854 fs_manager.cc:261] Metadata directory not provided I0703 01:40:42.223254 16854 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:40:42.223501 16854 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:40:42.223589 16854 server_base.cc:433] Attempting to create new FS layout instead I0703 01:40:42.227748 16854 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "8ace61239bda46ba874d4b094fff117d" format_stamp: "Formatted at 2018-07-03 08:40:42 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:42.228226 16854 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "8ace61239bda46ba874d4b094fff117d" format_stamp: "Formatted at 2018-07-03 08:40:42 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:42.236642 16854 fs_manager.cc:501] Time spent creating directory manager: real 0.008s user 0.006s sys 0.002s I0703 01:40:42.237833 16854 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:40:42.238226 16854 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:40:42.242311 16854 fs_manager.cc:417] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s I0703 01:40:42.242362 16854 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal uuid: "8ace61239bda46ba874d4b094fff117d" format_stamp: "Formatted at 2018-07-03 08:40:42 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:42.242496 16854 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:40:42.265027 16854 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:40:42.267522 16854 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:40:42.267961 16854 tablet_server_main.cc:90] Starting tablet server... I0703 01:40:42.367643 16854 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.193:53360 I0703 01:40:42.368106 16854 webserver.cc:175] Starting webserver on 127.5.240.193:0 I0703 01:40:42.368151 16854 webserver.cc:186] Document root disabled I0703 01:40:42.369498 16854 webserver.cc:313] Webserver started. Bound to: http://127.5.240.193:54804/ I0703 01:40:42.370045 16854 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb I0703 01:40:42.372053 16854 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:40:42.380252 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 16854 I0703 01:40:42.381121 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.5.240.194:0 --local_ip_for_outbound_sockets=127.5.240.194 --webserver_interface=127.5.240.194 --webserver_port=0 --tserver_master_addrs=127.5.240.254:36679 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0703 01:40:42.388098 16983 heartbeater.cc:346] Connected to a master server at 127.5.240.254:36679 I0703 01:40:42.388243 16983 heartbeater.cc:426] Registering TS with master... I0703 01:40:42.388988 16983 heartbeater.cc:475] Master 127.5.240.254:36679 requested a full tablet report, sending... I0703 01:40:42.390614 16820 ts_manager.cc:98] Registered new tserver with Master: 8ace61239bda46ba874d4b094fff117d (127.5.240.193:53360) I0703 01:40:42.392765 16820 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.193:43529 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:40:42.550701 16987 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:40:42.550907 16987 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:40:42.552894 16987 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0703 01:40:42.552942 16987 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:40:42.553030 16987 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.194 I0703 01:40:42.560040 16987 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.194:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.5.240.194 --webserver_port=0 --tserver_master_addrs=127.5.240.254:36679 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.194 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:40:42.566704 16987 tablet_server_main.cc:85] Initializing tablet server... I0703 01:40:42.568193 16987 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 174708us I0703 01:40:42.568769 16987 fs_manager.cc:261] Metadata directory not provided I0703 01:40:42.568828 16987 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:40:42.569043 16987 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:40:42.569104 16987 server_base.cc:433] Attempting to create new FS layout instead I0703 01:40:42.573174 16987 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "3d35a3bef6e6470d99c05c79af53ef82" format_stamp: "Formatted at 2018-07-03 08:40:42 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:42.573695 16987 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "3d35a3bef6e6470d99c05c79af53ef82" format_stamp: "Formatted at 2018-07-03 08:40:42 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:42.580919 16987 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.004s sys 0.002s I0703 01:40:42.582072 16987 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:40:42.582430 16987 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:40:42.585991 16987 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0703 01:40:42.586035 16987 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal uuid: "3d35a3bef6e6470d99c05c79af53ef82" format_stamp: "Formatted at 2018-07-03 08:40:42 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:42.586148 16987 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:40:42.608791 16987 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:40:42.610986 16987 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:40:42.611308 16987 tablet_server_main.cc:90] Starting tablet server... I0703 01:40:42.694506 16987 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.194:41984 I0703 01:40:42.694948 16987 webserver.cc:175] Starting webserver on 127.5.240.194:0 I0703 01:40:42.694993 16987 webserver.cc:186] Document root disabled I0703 01:40:42.696175 16987 webserver.cc:313] Webserver started. Bound to: http://127.5.240.194:59667/ I0703 01:40:42.696724 16987 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb I0703 01:40:42.698566 16987 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:40:42.706373 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 16987 I0703 01:40:42.707000 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.5.240.195:0 --local_ip_for_outbound_sockets=127.5.240.195 --webserver_interface=127.5.240.195 --webserver_port=0 --tserver_master_addrs=127.5.240.254:36679 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0703 01:40:42.709909 17106 heartbeater.cc:346] Connected to a master server at 127.5.240.254:36679 I0703 01:40:42.710062 17106 heartbeater.cc:426] Registering TS with master... I0703 01:40:42.710887 17106 heartbeater.cc:475] Master 127.5.240.254:36679 requested a full tablet report, sending... I0703 01:40:42.711932 16820 ts_manager.cc:98] Registered new tserver with Master: 3d35a3bef6e6470d99c05c79af53ef82 (127.5.240.194:41984) I0703 01:40:42.712862 16820 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.194:57498 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:40:42.864887 17109 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:40:42.865092 17109 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:40:42.867079 17109 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0703 01:40:42.867125 17109 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:40:42.867208 17109 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.195 I0703 01:40:42.873725 17109 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.195:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.5.240.195 --webserver_port=0 --tserver_master_addrs=127.5.240.254:36679 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.195 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:40:42.880220 17109 tablet_server_main.cc:85] Initializing tablet server... I0703 01:40:42.881742 17109 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 174708us I0703 01:40:42.882278 17109 fs_manager.cc:261] Metadata directory not provided I0703 01:40:42.882336 17109 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:40:42.882951 17109 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:40:42.883015 17109 server_base.cc:433] Attempting to create new FS layout instead I0703 01:40:42.887045 17109 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "47fc1a635c3a44d58fb94e27f2551847" format_stamp: "Formatted at 2018-07-03 08:40:42 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:42.887536 17109 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "47fc1a635c3a44d58fb94e27f2551847" format_stamp: "Formatted at 2018-07-03 08:40:42 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:42.895311 17109 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.004s sys 0.002s I0703 01:40:42.896522 17109 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:40:42.896914 17109 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:40:42.900004 17109 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0703 01:40:42.900049 17109 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal uuid: "47fc1a635c3a44d58fb94e27f2551847" format_stamp: "Formatted at 2018-07-03 08:40:42 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:42.900164 17109 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:40:42.935390 17109 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:40:42.937646 17109 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:40:42.937937 17109 tablet_server_main.cc:90] Starting tablet server... I0703 01:40:43.022819 17109 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.195:46477 I0703 01:40:43.023229 17109 webserver.cc:175] Starting webserver on 127.5.240.195:0 I0703 01:40:43.023274 17109 webserver.cc:186] Document root disabled I0703 01:40:43.024430 17109 webserver.cc:313] Webserver started. Bound to: http://127.5.240.195:55522/ I0703 01:40:43.024984 17109 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb I0703 01:40:43.027326 17109 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:40:43.030750 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 17109 I0703 01:40:43.031700 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.5.240.196:0 --local_ip_for_outbound_sockets=127.5.240.196 --webserver_interface=127.5.240.196 --webserver_port=0 --tserver_master_addrs=127.5.240.254:36679 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0703 01:40:43.040573 17228 heartbeater.cc:346] Connected to a master server at 127.5.240.254:36679 I0703 01:40:43.040733 17228 heartbeater.cc:426] Registering TS with master... I0703 01:40:43.041260 17228 heartbeater.cc:475] Master 127.5.240.254:36679 requested a full tablet report, sending... I0703 01:40:43.042349 16820 ts_manager.cc:98] Registered new tserver with Master: 47fc1a635c3a44d58fb94e27f2551847 (127.5.240.195:46477) I0703 01:40:43.043669 16820 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.195:54155 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:40:43.194304 17231 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:40:43.194541 17231 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:40:43.196604 17231 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0703 01:40:43.196666 17231 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:40:43.196768 17231 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.196 I0703 01:40:43.203248 17231 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.196:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb --webserver_interface=127.5.240.196 --webserver_port=0 --tserver_master_addrs=127.5.240.254:36679 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.196 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:40:43.209888 17231 tablet_server_main.cc:85] Initializing tablet server... I0703 01:40:43.211388 17231 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 175208us I0703 01:40:43.211957 17231 fs_manager.cc:261] Metadata directory not provided I0703 01:40:43.212024 17231 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:40:43.212256 17231 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:40:43.212326 17231 server_base.cc:433] Attempting to create new FS layout instead I0703 01:40:43.216379 17231 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/instance: uuid: "e950a6c6229e4f80ba63b793d64d1998" format_stamp: "Formatted at 2018-07-03 08:40:43 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:43.216907 17231 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal/instance: uuid: "e950a6c6229e4f80ba63b793d64d1998" format_stamp: "Formatted at 2018-07-03 08:40:43 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:43.224143 17231 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.002s sys 0.004s I0703 01:40:43.225311 17231 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:40:43.225723 17231 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:40:43.229249 17231 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0703 01:40:43.229297 17231 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal uuid: "e950a6c6229e4f80ba63b793d64d1998" format_stamp: "Formatted at 2018-07-03 08:40:43 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:43.229432 17231 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:40:43.257477 17231 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:40:43.259752 17231 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:40:43.260069 17231 tablet_server_main.cc:90] Starting tablet server... I0703 01:40:43.344535 17231 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.196:32786 I0703 01:40:43.344947 17231 webserver.cc:175] Starting webserver on 127.5.240.196:0 I0703 01:40:43.344993 17231 webserver.cc:186] Document root disabled I0703 01:40:43.346211 17231 webserver.cc:313] Webserver started. Bound to: http://127.5.240.196:55511/ I0703 01:40:43.346798 17231 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb I0703 01:40:43.348131 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 17231 I0703 01:40:43.349037 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data --block_manager=log --rpc_bind_addresses=127.5.240.197:0 --local_ip_for_outbound_sockets=127.5.240.197 --webserver_interface=127.5.240.197 --webserver_port=0 --tserver_master_addrs=127.5.240.254:36679 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0703 01:40:43.355460 17231 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:40:43.362754 17350 heartbeater.cc:346] Connected to a master server at 127.5.240.254:36679 I0703 01:40:43.362900 17350 heartbeater.cc:426] Registering TS with master... I0703 01:40:43.363653 17350 heartbeater.cc:475] Master 127.5.240.254:36679 requested a full tablet report, sending... I0703 01:40:43.364697 16820 ts_manager.cc:98] Registered new tserver with Master: e950a6c6229e4f80ba63b793d64d1998 (127.5.240.196:32786) I0703 01:40:43.365962 16820 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.196:54782 I0703 01:40:43.394997 16983 heartbeater.cc:467] Master 127.5.240.254:36679 was elected leader, sending a full tablet report... WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:40:43.509533 17353 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:40:43.509788 17353 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:40:43.511801 17353 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0703 01:40:43.511862 17353 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:40:43.511963 17353 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.197 I0703 01:40:43.518406 17353 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.197:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb --webserver_interface=127.5.240.197 --webserver_port=0 --tserver_master_addrs=127.5.240.254:36679 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.197 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:40:43.524909 17353 tablet_server_main.cc:85] Initializing tablet server... I0703 01:40:43.526422 17353 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 175208us I0703 01:40:43.526991 17353 fs_manager.cc:261] Metadata directory not provided I0703 01:40:43.527060 17353 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:40:43.527292 17353 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:40:43.527360 17353 server_base.cc:433] Attempting to create new FS layout instead I0703 01:40:43.531481 17353 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/instance: uuid: "14cbc64bc75444b8bbd9fbdcefdb86be" format_stamp: "Formatted at 2018-07-03 08:40:43 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:43.532023 17353 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal/instance: uuid: "14cbc64bc75444b8bbd9fbdcefdb86be" format_stamp: "Formatted at 2018-07-03 08:40:43 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:43.539283 17353 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.003s sys 0.002s I0703 01:40:43.540452 17353 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:40:43.540855 17353 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:40:43.544446 17353 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s I0703 01:40:43.544497 17353 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal uuid: "14cbc64bc75444b8bbd9fbdcefdb86be" format_stamp: "Formatted at 2018-07-03 08:40:43 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:43.544642 17353 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:40:43.584861 17353 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:40:43.587123 17353 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:40:43.587466 17353 tablet_server_main.cc:90] Starting tablet server... I0703 01:40:43.671092 17353 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.197:40856 I0703 01:40:43.671509 17353 webserver.cc:175] Starting webserver on 127.5.240.197:0 I0703 01:40:43.671555 17353 webserver.cc:186] Document root disabled I0703 01:40:43.672719 17353 webserver.cc:313] Webserver started. Bound to: http://127.5.240.197:59718/ I0703 01:40:43.673249 17353 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb I0703 01:40:43.674890 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 17353 I0703 01:40:43.676102 17353 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:40:43.689996 17472 heartbeater.cc:346] Connected to a master server at 127.5.240.254:36679 I0703 01:40:43.690130 17472 heartbeater.cc:426] Registering TS with master... I0703 01:40:43.690872 17472 heartbeater.cc:475] Master 127.5.240.254:36679 requested a full tablet report, sending... I0703 01:40:43.691860 16820 ts_manager.cc:98] Registered new tserver with Master: 14cbc64bc75444b8bbd9fbdcefdb86be (127.5.240.197:40856) I0703 01:40:43.692912 16820 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.197:59874 I0703 01:40:43.693224 6083 external_mini_cluster.cc:490] 5 TS(s) registered with all masters I0703 01:40:43.702960 16820 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.5.240.254:57975: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0703 01:40:43.715467 17106 heartbeater.cc:467] Master 127.5.240.254:36679 was elected leader, sending a full tablet report... I0703 01:40:43.728622 17409 tablet_service.cc:759] Processing CreateTablet for tablet 94cc179826c046169908bf910ed7c7bb (table=TestTable [id=144618e6d82940cd9252cc126e332762]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:40:43.729992 17287 tablet_service.cc:759] Processing CreateTablet for tablet 94cc179826c046169908bf910ed7c7bb (table=TestTable [id=144618e6d82940cd9252cc126e332762]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:40:43.730629 17287 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 94cc179826c046169908bf910ed7c7bb. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:40:43.730895 17409 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 94cc179826c046169908bf910ed7c7bb. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:40:43.732672 16920 tablet_service.cc:759] Processing CreateTablet for tablet 94cc179826c046169908bf910ed7c7bb (table=TestTable [id=144618e6d82940cd9252cc126e332762]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:40:43.733460 16920 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 94cc179826c046169908bf910ed7c7bb. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:40:43.740700 17287 ts_tablet_manager.cc:1173] T 94cc179826c046169908bf910ed7c7bb P e950a6c6229e4f80ba63b793d64d1998: Registered tablet (data state: TABLET_DATA_READY) I0703 01:40:43.741453 17492 ts_tablet_manager.cc:1028] T 94cc179826c046169908bf910ed7c7bb P e950a6c6229e4f80ba63b793d64d1998: Bootstrapping tablet I0703 01:40:43.742023 17492 tablet_bootstrap.cc:427] T 94cc179826c046169908bf910ed7c7bb P e950a6c6229e4f80ba63b793d64d1998: Bootstrap starting. I0703 01:40:43.744526 17492 tablet_bootstrap.cc:578] T 94cc179826c046169908bf910ed7c7bb P e950a6c6229e4f80ba63b793d64d1998: No blocks or log segments found. Creating new log. I0703 01:40:43.745111 17409 ts_tablet_manager.cc:1173] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be: Registered tablet (data state: TABLET_DATA_READY) I0703 01:40:43.745281 17492 log.cc:525] T 94cc179826c046169908bf910ed7c7bb P e950a6c6229e4f80ba63b793d64d1998: Log is configured to *not* fsync() on all Append() calls I0703 01:40:43.745885 17493 ts_tablet_manager.cc:1028] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be: Bootstrapping tablet I0703 01:40:43.746330 17493 tablet_bootstrap.cc:427] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be: Bootstrap starting. I0703 01:40:43.746601 16920 ts_tablet_manager.cc:1173] T 94cc179826c046169908bf910ed7c7bb P 8ace61239bda46ba874d4b094fff117d: Registered tablet (data state: TABLET_DATA_READY) I0703 01:40:43.749146 17493 tablet_bootstrap.cc:578] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be: No blocks or log segments found. Creating new log. I0703 01:40:43.749974 17493 log.cc:525] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be: Log is configured to *not* fsync() on all Append() calls I0703 01:40:43.751364 17495 ts_tablet_manager.cc:1028] T 94cc179826c046169908bf910ed7c7bb P 8ace61239bda46ba874d4b094fff117d: Bootstrapping tablet I0703 01:40:43.751767 17495 tablet_bootstrap.cc:427] T 94cc179826c046169908bf910ed7c7bb P 8ace61239bda46ba874d4b094fff117d: Bootstrap starting. I0703 01:40:43.751899 17492 tablet_bootstrap.cc:427] T 94cc179826c046169908bf910ed7c7bb P e950a6c6229e4f80ba63b793d64d1998: No bootstrap required, opened a new log I0703 01:40:43.752156 17492 ts_tablet_manager.cc:1045] T 94cc179826c046169908bf910ed7c7bb P e950a6c6229e4f80ba63b793d64d1998: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s I0703 01:40:43.752344 17493 tablet_bootstrap.cc:427] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be: No bootstrap required, opened a new log I0703 01:40:43.752516 17493 ts_tablet_manager.cc:1045] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s I0703 01:40:43.754441 17495 tablet_bootstrap.cc:578] T 94cc179826c046169908bf910ed7c7bb P 8ace61239bda46ba874d4b094fff117d: No blocks or log segments found. Creating new log. I0703 01:40:43.755393 17495 log.cc:525] T 94cc179826c046169908bf910ed7c7bb P 8ace61239bda46ba874d4b094fff117d: Log is configured to *not* fsync() on all Append() calls I0703 01:40:43.758705 17493 raft_consensus.cc:325] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14cbc64bc75444b8bbd9fbdcefdb86be" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 40856 } } peers { permanent_uuid: "8ace61239bda46ba874d4b094fff117d" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 53360 } } peers { permanent_uuid: "e950a6c6229e4f80ba63b793d64d1998" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 32786 } } I0703 01:40:43.759012 17493 raft_consensus.cc:351] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:40:43.759096 17493 raft_consensus.cc:628] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14cbc64bc75444b8bbd9fbdcefdb86be, State: Initialized, Role: FOLLOWER I0703 01:40:43.759503 17493 consensus_queue.cc:227] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14cbc64bc75444b8bbd9fbdcefdb86be" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 40856 } } peers { permanent_uuid: "8ace61239bda46ba874d4b094fff117d" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 53360 } } peers { permanent_uuid: "e950a6c6229e4f80ba63b793d64d1998" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 32786 } } I0703 01:40:43.760764 17492 raft_consensus.cc:325] T 94cc179826c046169908bf910ed7c7bb P e950a6c6229e4f80ba63b793d64d1998 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14cbc64bc75444b8bbd9fbdcefdb86be" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 40856 } } peers { permanent_uuid: "8ace61239bda46ba874d4b094fff117d" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 53360 } } peers { permanent_uuid: "e950a6c6229e4f80ba63b793d64d1998" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 32786 } } I0703 01:40:43.761159 17492 raft_consensus.cc:351] T 94cc179826c046169908bf910ed7c7bb P e950a6c6229e4f80ba63b793d64d1998 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:40:43.761226 17472 heartbeater.cc:467] Master 127.5.240.254:36679 was elected leader, sending a full tablet report... I0703 01:40:43.761251 17492 raft_consensus.cc:628] T 94cc179826c046169908bf910ed7c7bb P e950a6c6229e4f80ba63b793d64d1998 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e950a6c6229e4f80ba63b793d64d1998, State: Initialized, Role: FOLLOWER I0703 01:40:43.761642 17492 consensus_queue.cc:227] T 94cc179826c046169908bf910ed7c7bb P e950a6c6229e4f80ba63b793d64d1998 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14cbc64bc75444b8bbd9fbdcefdb86be" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 40856 } } peers { permanent_uuid: "8ace61239bda46ba874d4b094fff117d" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 53360 } } peers { permanent_uuid: "e950a6c6229e4f80ba63b793d64d1998" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 32786 } } I0703 01:40:43.762130 17493 ts_tablet_manager.cc:1073] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be: Time spent starting tablet: real 0.010s user 0.007s sys 0.002s I0703 01:40:43.762615 17495 tablet_bootstrap.cc:427] T 94cc179826c046169908bf910ed7c7bb P 8ace61239bda46ba874d4b094fff117d: No bootstrap required, opened a new log I0703 01:40:43.762835 17495 ts_tablet_manager.cc:1045] T 94cc179826c046169908bf910ed7c7bb P 8ace61239bda46ba874d4b094fff117d: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s I0703 01:40:43.763464 17350 heartbeater.cc:467] Master 127.5.240.254:36679 was elected leader, sending a full tablet report... I0703 01:40:43.765424 17492 ts_tablet_manager.cc:1073] T 94cc179826c046169908bf910ed7c7bb P e950a6c6229e4f80ba63b793d64d1998: Time spent starting tablet: real 0.013s user 0.005s sys 0.006s I0703 01:40:43.769534 17495 raft_consensus.cc:325] T 94cc179826c046169908bf910ed7c7bb P 8ace61239bda46ba874d4b094fff117d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14cbc64bc75444b8bbd9fbdcefdb86be" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 40856 } } peers { permanent_uuid: "8ace61239bda46ba874d4b094fff117d" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 53360 } } peers { permanent_uuid: "e950a6c6229e4f80ba63b793d64d1998" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 32786 } } I0703 01:40:43.769816 17495 raft_consensus.cc:351] T 94cc179826c046169908bf910ed7c7bb P 8ace61239bda46ba874d4b094fff117d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:40:43.769865 17495 raft_consensus.cc:628] T 94cc179826c046169908bf910ed7c7bb P 8ace61239bda46ba874d4b094fff117d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ace61239bda46ba874d4b094fff117d, State: Initialized, Role: FOLLOWER I0703 01:40:43.770196 17495 consensus_queue.cc:227] T 94cc179826c046169908bf910ed7c7bb P 8ace61239bda46ba874d4b094fff117d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14cbc64bc75444b8bbd9fbdcefdb86be" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 40856 } } peers { permanent_uuid: "8ace61239bda46ba874d4b094fff117d" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 53360 } } peers { permanent_uuid: "e950a6c6229e4f80ba63b793d64d1998" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 32786 } } I0703 01:40:43.771242 17495 ts_tablet_manager.cc:1073] T 94cc179826c046169908bf910ed7c7bb P 8ace61239bda46ba874d4b094fff117d: Time spent starting tablet: real 0.008s user 0.006s sys 0.002s W0703 01:40:43.855711 17351 tablet.cc:1678] T 94cc179826c046169908bf910ed7c7bb P e950a6c6229e4f80ba63b793d64d1998: Can't schedule compaction. Clean time has not been advanced past its initial value. W0703 01:40:43.873127 16984 tablet.cc:1678] T 94cc179826c046169908bf910ed7c7bb P 8ace61239bda46ba874d4b094fff117d: Can't schedule compaction. Clean time has not been advanced past its initial value. W0703 01:40:43.926010 17473 tablet.cc:1678] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:40:43.965415 17498 raft_consensus.cc:457] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0703 01:40:43.965911 17498 raft_consensus.cc:2730] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be: Snoozing failure detection for 1.854s (starting election) I0703 01:40:43.966042 17498 raft_consensus.cc:479] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14cbc64bc75444b8bbd9fbdcefdb86be" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 40856 } } peers { permanent_uuid: "8ace61239bda46ba874d4b094fff117d" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 53360 } } peers { permanent_uuid: "e950a6c6229e4f80ba63b793d64d1998" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 32786 } } I0703 01:40:43.967191 17498 leader_election.cc:243] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 8ace61239bda46ba874d4b094fff117d (127.5.240.193:53360) I0703 01:40:43.967511 17498 leader_election.cc:243] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer e950a6c6229e4f80ba63b793d64d1998 (127.5.240.196:32786) I0703 01:40:43.974225 16940 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "94cc179826c046169908bf910ed7c7bb" candidate_uuid: "14cbc64bc75444b8bbd9fbdcefdb86be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ace61239bda46ba874d4b094fff117d" is_pre_election: true I0703 01:40:43.974946 16940 raft_consensus.cc:2730] T 94cc179826c046169908bf910ed7c7bb P 8ace61239bda46ba874d4b094fff117d: Snoozing failure detection for 1.991s (vote granted) I0703 01:40:43.975059 16940 raft_consensus.cc:2243] T 94cc179826c046169908bf910ed7c7bb P 8ace61239bda46ba874d4b094fff117d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14cbc64bc75444b8bbd9fbdcefdb86be in term 0. I0703 01:40:43.975329 17307 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "94cc179826c046169908bf910ed7c7bb" candidate_uuid: "14cbc64bc75444b8bbd9fbdcefdb86be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e950a6c6229e4f80ba63b793d64d1998" is_pre_election: true I0703 01:40:43.975632 17307 raft_consensus.cc:2730] T 94cc179826c046169908bf910ed7c7bb P e950a6c6229e4f80ba63b793d64d1998: Snoozing failure detection for 1.603s (vote granted) I0703 01:40:43.975711 17307 raft_consensus.cc:2243] T 94cc179826c046169908bf910ed7c7bb P e950a6c6229e4f80ba63b793d64d1998 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14cbc64bc75444b8bbd9fbdcefdb86be in term 0. I0703 01:40:43.975821 17361 leader_election.cc:379] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be [CANDIDATE]: Term 1 pre-election: Vote granted by peer 8ace61239bda46ba874d4b094fff117d (127.5.240.193:53360) I0703 01:40:43.975881 17361 leader_election.cc:270] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0703 01:40:43.976203 17498 raft_consensus.cc:2730] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be: Snoozing failure detection for 1.547s (election complete) I0703 01:40:43.976318 17498 raft_consensus.cc:2537] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be [term 0 FOLLOWER]: Leader pre-election won for term 1 I0703 01:40:43.976366 17360 leader_election.cc:379] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be [CANDIDATE]: Term 1 pre-election: Vote granted by peer e950a6c6229e4f80ba63b793d64d1998 (127.5.240.196:32786) I0703 01:40:43.976429 17498 raft_consensus.cc:457] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0703 01:40:43.976527 17498 raft_consensus.cc:2730] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be: Snoozing failure detection for 1.528s (starting election) I0703 01:40:43.976639 17498 raft_consensus.cc:2782] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be [term 0 FOLLOWER]: Advancing to term 1 I0703 01:40:43.978051 17498 raft_consensus.cc:479] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14cbc64bc75444b8bbd9fbdcefdb86be" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 40856 } } peers { permanent_uuid: "8ace61239bda46ba874d4b094fff117d" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 53360 } } peers { permanent_uuid: "e950a6c6229e4f80ba63b793d64d1998" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 32786 } } I0703 01:40:43.978392 17498 leader_election.cc:243] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be [CANDIDATE]: Term 1 election: Requesting vote from peer 8ace61239bda46ba874d4b094fff117d (127.5.240.193:53360) I0703 01:40:43.978585 17498 leader_election.cc:243] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be [CANDIDATE]: Term 1 election: Requesting vote from peer e950a6c6229e4f80ba63b793d64d1998 (127.5.240.196:32786) I0703 01:40:43.979048 16940 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "94cc179826c046169908bf910ed7c7bb" candidate_uuid: "14cbc64bc75444b8bbd9fbdcefdb86be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ace61239bda46ba874d4b094fff117d" I0703 01:40:43.979158 16940 raft_consensus.cc:2782] T 94cc179826c046169908bf910ed7c7bb P 8ace61239bda46ba874d4b094fff117d [term 0 FOLLOWER]: Advancing to term 1 I0703 01:40:43.979233 16940 raft_consensus.cc:2730] T 94cc179826c046169908bf910ed7c7bb P 8ace61239bda46ba874d4b094fff117d: Snoozing failure detection for 1.520s (vote granted) I0703 01:40:43.979192 17307 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "94cc179826c046169908bf910ed7c7bb" candidate_uuid: "14cbc64bc75444b8bbd9fbdcefdb86be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e950a6c6229e4f80ba63b793d64d1998" I0703 01:40:43.979315 17307 raft_consensus.cc:2782] T 94cc179826c046169908bf910ed7c7bb P e950a6c6229e4f80ba63b793d64d1998 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:40:43.979398 17307 raft_consensus.cc:2730] T 94cc179826c046169908bf910ed7c7bb P e950a6c6229e4f80ba63b793d64d1998: Snoozing failure detection for 1.700s (vote granted) I0703 01:40:43.980648 16940 raft_consensus.cc:2243] T 94cc179826c046169908bf910ed7c7bb P 8ace61239bda46ba874d4b094fff117d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14cbc64bc75444b8bbd9fbdcefdb86be in term 1. I0703 01:40:43.980746 17307 raft_consensus.cc:2243] T 94cc179826c046169908bf910ed7c7bb P e950a6c6229e4f80ba63b793d64d1998 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14cbc64bc75444b8bbd9fbdcefdb86be in term 1. I0703 01:40:43.981075 17361 leader_election.cc:379] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be [CANDIDATE]: Term 1 election: Vote granted by peer 8ace61239bda46ba874d4b094fff117d (127.5.240.193:53360) I0703 01:40:43.981163 17361 leader_election.cc:270] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:40:43.981245 17360 leader_election.cc:379] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be [CANDIDATE]: Term 1 election: Vote granted by peer e950a6c6229e4f80ba63b793d64d1998 (127.5.240.196:32786) I0703 01:40:43.981389 17498 raft_consensus.cc:2730] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be: Snoozing failure detection for 1.782s (election complete) I0703 01:40:43.981457 17498 raft_consensus.cc:2537] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:40:43.982302 17498 raft_consensus.cc:593] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be [term 1 LEADER]: Becoming Leader. State: Replica: 14cbc64bc75444b8bbd9fbdcefdb86be, State: Running, Role: LEADER I0703 01:40:43.982767 17498 consensus_queue.cc:204] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14cbc64bc75444b8bbd9fbdcefdb86be" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 40856 } } peers { permanent_uuid: "8ace61239bda46ba874d4b094fff117d" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 53360 } } peers { permanent_uuid: "e950a6c6229e4f80ba63b793d64d1998" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 32786 } } I0703 01:40:43.986448 16820 catalog_manager.cc:3798] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be reported cstate change: term changed from 0 to 1, leader changed from to 14cbc64bc75444b8bbd9fbdcefdb86be (127.5.240.197). New cstate: current_term: 1 leader_uuid: "14cbc64bc75444b8bbd9fbdcefdb86be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14cbc64bc75444b8bbd9fbdcefdb86be" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 40856 } } peers { permanent_uuid: "8ace61239bda46ba874d4b094fff117d" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 53360 } } peers { permanent_uuid: "e950a6c6229e4f80ba63b793d64d1998" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 32786 } } } I0703 01:40:44.006409 6083 external_mini_cluster.cc:490] 5 TS(s) registered with all masters I0703 01:40:44.007488 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 8ace61239bda46ba874d4b094fff117d to finish bootstrapping I0703 01:40:44.012707 6083 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 3d35a3bef6e6470d99c05c79af53ef82 to finish bootstrapping I0703 01:40:44.018653 6083 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 47fc1a635c3a44d58fb94e27f2551847 to finish bootstrapping I0703 01:40:44.024590 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver e950a6c6229e4f80ba63b793d64d1998 to finish bootstrapping I0703 01:40:44.029328 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 14cbc64bc75444b8bbd9fbdcefdb86be to finish bootstrapping I0703 01:40:44.033601 6083 test_util.cc:208] Using random seed: 1158823473 I0703 01:40:44.042563 6083 test_workload.cc:306] TestWorkload: Skipping table creation because table TestTable already exists I0703 01:40:44.045516 17228 heartbeater.cc:467] Master 127.5.240.254:36679 was elected leader, sending a full tablet report... I0703 01:40:44.061712 17307 raft_consensus.cc:1081] T 94cc179826c046169908bf910ed7c7bb P e950a6c6229e4f80ba63b793d64d1998 [term 1 FOLLOWER]: Refusing update from remote peer 14cbc64bc75444b8bbd9fbdcefdb86be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0703 01:40:44.062168 16940 raft_consensus.cc:1081] T 94cc179826c046169908bf910ed7c7bb P 8ace61239bda46ba874d4b094fff117d [term 1 FOLLOWER]: Refusing update from remote peer 14cbc64bc75444b8bbd9fbdcefdb86be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0703 01:40:44.062826 17498 consensus_queue.cc:962] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be [LEADER]: Connected to new peer: Peer: permanent_uuid: "e950a6c6229e4f80ba63b793d64d1998" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 32786 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0703 01:40:44.063096 17502 consensus_queue.cc:962] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ace61239bda46ba874d4b094fff117d" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 53360 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s W0703 01:40:44.336124 17512 flags.cc:406] Enabled unsafe flag: --never_fsync=true I0703 01:40:44.572775 17429 consensus_peers.cc:455] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be -> Peer e950a6c6229e4f80ba63b793d64d1998 (127.5.240.196:32786): Closing peer: e950a6c6229e4f80ba63b793d64d1998 I0703 01:40:44.572854 17429 consensus_peers.cc:455] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be -> Peer 8ace61239bda46ba874d4b094fff117d (127.5.240.193:53360): Closing peer: 8ace61239bda46ba874d4b094fff117d I0703 01:40:44.573025 17429 consensus_queue.cc:204] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 92, Committed index: 92, Last appended: 1.93, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "14cbc64bc75444b8bbd9fbdcefdb86be" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 40856 } } peers { permanent_uuid: "8ace61239bda46ba874d4b094fff117d" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 53360 } } peers { permanent_uuid: "e950a6c6229e4f80ba63b793d64d1998" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 32786 } } peers { permanent_uuid: "3d35a3bef6e6470d99c05c79af53ef82" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 41984 } } I0703 01:40:44.575033 17307 raft_consensus.cc:1081] T 94cc179826c046169908bf910ed7c7bb P e950a6c6229e4f80ba63b793d64d1998 [term 1 FOLLOWER]: Refusing update from remote peer 14cbc64bc75444b8bbd9fbdcefdb86be: Log matching property violated. Preceding OpId in replica: term: 1 index: 93. Preceding OpId from leader: term: 1 index: 94. (index mismatch) I0703 01:40:44.575819 17524 consensus_queue.cc:962] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be [LEADER]: Connected to new peer: Peer: permanent_uuid: "e950a6c6229e4f80ba63b793d64d1998" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 32786 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 94, Last known committed idx: 92, Time since last communication: 0.000s I0703 01:40:44.578862 16939 raft_consensus.cc:1081] T 94cc179826c046169908bf910ed7c7bb P 8ace61239bda46ba874d4b094fff117d [term 1 FOLLOWER]: Refusing update from remote peer 14cbc64bc75444b8bbd9fbdcefdb86be: Log matching property violated. Preceding OpId in replica: term: 1 index: 93. Preceding OpId from leader: term: 1 index: 94. (index mismatch) I0703 01:40:44.579524 17524 consensus_queue.cc:962] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ace61239bda46ba874d4b094fff117d" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 53360 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 94, Last known committed idx: 92, Time since last communication: 0.000s I0703 01:40:44.581761 17498 raft_consensus.cc:2684] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be [term 1 LEADER]: Committing config change with OpId 1.94: config changed from index -1 to 94, VOTER 3d35a3bef6e6470d99c05c79af53ef82 (127.5.240.194) added. New config: { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "14cbc64bc75444b8bbd9fbdcefdb86be" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 40856 } } peers { permanent_uuid: "8ace61239bda46ba874d4b094fff117d" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 53360 } } peers { permanent_uuid: "e950a6c6229e4f80ba63b793d64d1998" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 32786 } } peers { permanent_uuid: "3d35a3bef6e6470d99c05c79af53ef82" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 41984 } } } I0703 01:40:44.584451 16939 raft_consensus.cc:2684] T 94cc179826c046169908bf910ed7c7bb P 8ace61239bda46ba874d4b094fff117d [term 1 FOLLOWER]: Committing config change with OpId 1.94: config changed from index -1 to 94, VOTER 3d35a3bef6e6470d99c05c79af53ef82 (127.5.240.194) added. New config: { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "14cbc64bc75444b8bbd9fbdcefdb86be" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 40856 } } peers { permanent_uuid: "8ace61239bda46ba874d4b094fff117d" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 53360 } } peers { permanent_uuid: "e950a6c6229e4f80ba63b793d64d1998" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 32786 } } peers { permanent_uuid: "3d35a3bef6e6470d99c05c79af53ef82" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 41984 } } } W0703 01:40:44.585489 17361 consensus_peers.cc:433] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be -> Peer 3d35a3bef6e6470d99c05c79af53ef82 (127.5.240.194:41984): Couldn't send request to peer 3d35a3bef6e6470d99c05c79af53ef82 for tablet 94cc179826c046169908bf910ed7c7bb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 94cc179826c046169908bf910ed7c7bb. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:40:44.588531 17307 raft_consensus.cc:2684] T 94cc179826c046169908bf910ed7c7bb P e950a6c6229e4f80ba63b793d64d1998 [term 1 FOLLOWER]: Committing config change with OpId 1.94: config changed from index -1 to 94, VOTER 3d35a3bef6e6470d99c05c79af53ef82 (127.5.240.194) added. New config: { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "14cbc64bc75444b8bbd9fbdcefdb86be" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 40856 } } peers { permanent_uuid: "8ace61239bda46ba874d4b094fff117d" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 53360 } } peers { permanent_uuid: "e950a6c6229e4f80ba63b793d64d1998" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 32786 } } peers { permanent_uuid: "3d35a3bef6e6470d99c05c79af53ef82" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 41984 } } } I0703 01:40:44.589323 16820 catalog_manager.cc:3798] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be reported cstate change: config changed from index -1 to 94, VOTER 3d35a3bef6e6470d99c05c79af53ef82 (127.5.240.194) added. New cstate: current_term: 1 leader_uuid: "14cbc64bc75444b8bbd9fbdcefdb86be" committed_config { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "14cbc64bc75444b8bbd9fbdcefdb86be" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 40856 } } peers { permanent_uuid: "8ace61239bda46ba874d4b094fff117d" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 53360 } } peers { permanent_uuid: "e950a6c6229e4f80ba63b793d64d1998" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 32786 } } peers { permanent_uuid: "3d35a3bef6e6470d99c05c79af53ef82" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 41984 } } } I0703 01:40:44.996263 17620 ts_tablet_manager.cc:669] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82: Initiating tablet copy from peer 14cbc64bc75444b8bbd9fbdcefdb86be (127.5.240.197:40856) I0703 01:40:44.996532 17620 tablet_copy_client.cc:239] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.197:40856 I0703 01:40:45.007304 17449 tablet_copy_service.cc:135] P 14cbc64bc75444b8bbd9fbdcefdb86be: Received BeginTabletCopySession request for tablet 94cc179826c046169908bf910ed7c7bb from peer 3d35a3bef6e6470d99c05c79af53ef82 ({username='jenkins'} at 127.5.240.194:32885) I0703 01:40:45.007643 17449 tablet_copy_service.cc:156] P 14cbc64bc75444b8bbd9fbdcefdb86be: Beginning new tablet copy session on tablet 94cc179826c046169908bf910ed7c7bb from peer 3d35a3bef6e6470d99c05c79af53ef82 at {username='jenkins'} at 127.5.240.194:32885: session id = 3d35a3bef6e6470d99c05c79af53ef82-94cc179826c046169908bf910ed7c7bb I0703 01:40:45.012145 17449 tablet_copy_source_session.cc:202] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:40:45.016403 17620 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 94cc179826c046169908bf910ed7c7bb. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:40:45.030288 17620 ts_tablet_manager.cc:1173] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:40:45.030679 17620 tablet_copy_client.cc:566] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82: tablet copy: Starting download of 0 data blocks... I0703 01:40:45.030970 17620 tablet_copy_client.cc:529] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82: tablet copy: Starting download of 1 WAL segments... I0703 01:40:45.037098 17620 tablet_copy_client.cc:405] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:40:45.039288 17620 ts_tablet_manager.cc:1028] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82: Bootstrapping tablet I0703 01:40:45.039507 17620 tablet_bootstrap.cc:427] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82: Bootstrap starting. I0703 01:40:45.063068 17620 log.cc:525] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82: Log is configured to *not* fsync() on all Append() calls I0703 01:40:45.195238 17620 tablet_bootstrap.cc:427] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82: Bootstrap replayed 1/1 log segments. Stats: ops{read=167 overwritten=0 applied=166 ignored=0} inserts{seen=164 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates I0703 01:40:45.195730 17620 tablet_bootstrap.cc:427] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82: Bootstrap complete. I0703 01:40:45.196055 17620 ts_tablet_manager.cc:1045] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82: Time spent bootstrapping tablet: real 0.157s user 0.148s sys 0.008s I0703 01:40:45.200904 17620 raft_consensus.cc:325] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending transactions. Active config: opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "14cbc64bc75444b8bbd9fbdcefdb86be" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 40856 } } peers { permanent_uuid: "8ace61239bda46ba874d4b094fff117d" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 53360 } } peers { permanent_uuid: "e950a6c6229e4f80ba63b793d64d1998" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 32786 } } peers { permanent_uuid: "3d35a3bef6e6470d99c05c79af53ef82" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 41984 } } I0703 01:40:45.203755 17620 raft_consensus.cc:628] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d35a3bef6e6470d99c05c79af53ef82, State: Initialized, Role: FOLLOWER I0703 01:40:45.204356 17620 consensus_queue.cc:227] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 166, Last appended: 1.167, Last appended by leader: 167, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "14cbc64bc75444b8bbd9fbdcefdb86be" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 40856 } } peers { permanent_uuid: "8ace61239bda46ba874d4b094fff117d" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 53360 } } peers { permanent_uuid: "e950a6c6229e4f80ba63b793d64d1998" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 32786 } } peers { permanent_uuid: "3d35a3bef6e6470d99c05c79af53ef82" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 41984 } } I0703 01:40:45.207891 17620 ts_tablet_manager.cc:1073] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82: Time spent starting tablet: real 0.011s user 0.008s sys 0.003s I0703 01:40:45.209117 17449 tablet_copy_service.cc:325] P 14cbc64bc75444b8bbd9fbdcefdb86be: Request end of tablet copy session 3d35a3bef6e6470d99c05c79af53ef82-94cc179826c046169908bf910ed7c7bb received from {username='jenkins'} at 127.5.240.194:32885 I0703 01:40:45.209200 17449 tablet_copy_service.cc:413] P 14cbc64bc75444b8bbd9fbdcefdb86be: ending tablet copy session 3d35a3bef6e6470d99c05c79af53ef82-94cc179826c046169908bf910ed7c7bb on tablet 94cc179826c046169908bf910ed7c7bb with peer 3d35a3bef6e6470d99c05c79af53ef82 I0703 01:40:45.579365 17063 raft_consensus.cc:1026] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.93->[1.94-1.262] Dedup: 1.167->[1.168-1.262] I0703 01:40:46.266680 17429 tablet_service.cc:1140] Received LeaderStepDown RPC: tablet_id: "94cc179826c046169908bf910ed7c7bb" dest_uuid: "14cbc64bc75444b8bbd9fbdcefdb86be" from {username='jenkins'} at 127.5.240.197:41271 I0703 01:40:46.267371 17429 raft_consensus.cc:552] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be [term 1 LEADER]: Received request to step down I0703 01:40:46.267457 17429 raft_consensus.cc:628] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 14cbc64bc75444b8bbd9fbdcefdb86be, State: Running, Role: LEADER I0703 01:40:46.267710 17429 consensus_queue.cc:227] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 375, Committed index: 375, Last appended: 1.375, Last appended by leader: 375, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "14cbc64bc75444b8bbd9fbdcefdb86be" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 40856 } } peers { permanent_uuid: "8ace61239bda46ba874d4b094fff117d" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 53360 } } peers { permanent_uuid: "e950a6c6229e4f80ba63b793d64d1998" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 32786 } } peers { permanent_uuid: "3d35a3bef6e6470d99c05c79af53ef82" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 41984 } } I0703 01:40:46.268002 17429 consensus_peers.cc:455] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be -> Peer e950a6c6229e4f80ba63b793d64d1998 (127.5.240.196:32786): Closing peer: e950a6c6229e4f80ba63b793d64d1998 I0703 01:40:46.268061 17429 consensus_peers.cc:455] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be -> Peer 3d35a3bef6e6470d99c05c79af53ef82 (127.5.240.194:41984): Closing peer: 3d35a3bef6e6470d99c05c79af53ef82 I0703 01:40:46.268110 17429 consensus_peers.cc:455] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be -> Peer 8ace61239bda46ba874d4b094fff117d (127.5.240.193:53360): Closing peer: 8ace61239bda46ba874d4b094fff117d I0703 01:40:46.268168 17429 raft_consensus.cc:2730] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be: Snoozing failure detection for 3.000s (explicit stepdown request) I0703 01:40:47.764616 17813 raft_consensus.cc:457] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 14cbc64bc75444b8bbd9fbdcefdb86be) I0703 01:40:47.764739 17813 raft_consensus.cc:2730] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82: Snoozing failure detection for 1.883s (starting election) I0703 01:40:47.764847 17813 raft_consensus.cc:479] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "14cbc64bc75444b8bbd9fbdcefdb86be" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 40856 } } peers { permanent_uuid: "8ace61239bda46ba874d4b094fff117d" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 53360 } } peers { permanent_uuid: "e950a6c6229e4f80ba63b793d64d1998" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 32786 } } peers { permanent_uuid: "3d35a3bef6e6470d99c05c79af53ef82" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 41984 } } I0703 01:40:47.765666 17813 leader_election.cc:243] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 14cbc64bc75444b8bbd9fbdcefdb86be (127.5.240.197:40856) I0703 01:40:47.765760 17814 raft_consensus.cc:457] T 94cc179826c046169908bf910ed7c7bb P 8ace61239bda46ba874d4b094fff117d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 14cbc64bc75444b8bbd9fbdcefdb86be) I0703 01:40:47.765854 17814 raft_consensus.cc:2730] T 94cc179826c046169908bf910ed7c7bb P 8ace61239bda46ba874d4b094fff117d: Snoozing failure detection for 2.022s (starting election) I0703 01:40:47.765949 17813 leader_election.cc:243] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 8ace61239bda46ba874d4b094fff117d (127.5.240.193:53360) I0703 01:40:47.766098 17813 leader_election.cc:243] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer e950a6c6229e4f80ba63b793d64d1998 (127.5.240.196:32786) I0703 01:40:47.765962 17814 raft_consensus.cc:479] T 94cc179826c046169908bf910ed7c7bb P 8ace61239bda46ba874d4b094fff117d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "14cbc64bc75444b8bbd9fbdcefdb86be" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 40856 } } peers { permanent_uuid: "8ace61239bda46ba874d4b094fff117d" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 53360 } } peers { permanent_uuid: "e950a6c6229e4f80ba63b793d64d1998" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 32786 } } peers { permanent_uuid: "3d35a3bef6e6470d99c05c79af53ef82" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 41984 } } I0703 01:40:47.766974 17814 leader_election.cc:243] T 94cc179826c046169908bf910ed7c7bb P 8ace61239bda46ba874d4b094fff117d [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 14cbc64bc75444b8bbd9fbdcefdb86be (127.5.240.197:40856) I0703 01:40:47.766995 17429 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "94cc179826c046169908bf910ed7c7bb" candidate_uuid: "3d35a3bef6e6470d99c05c79af53ef82" candidate_term: 2 candidate_status { last_received { term: 1 index: 375 } } ignore_live_leader: false dest_uuid: "14cbc64bc75444b8bbd9fbdcefdb86be" is_pre_election: true I0703 01:40:47.767220 17429 raft_consensus.cc:2730] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be: Snoozing failure detection for 1.683s (vote granted) I0703 01:40:47.767304 17429 raft_consensus.cc:2243] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d35a3bef6e6470d99c05c79af53ef82 in term 1. I0703 01:40:47.767431 17814 leader_election.cc:243] T 94cc179826c046169908bf910ed7c7bb P 8ace61239bda46ba874d4b094fff117d [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer e950a6c6229e4f80ba63b793d64d1998 (127.5.240.196:32786) I0703 01:40:47.768059 16994 leader_election.cc:379] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 [CANDIDATE]: Term 2 pre-election: Vote granted by peer 14cbc64bc75444b8bbd9fbdcefdb86be (127.5.240.197:40856) I0703 01:40:47.770427 17814 leader_election.cc:243] T 94cc179826c046169908bf910ed7c7bb P 8ace61239bda46ba874d4b094fff117d [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 3d35a3bef6e6470d99c05c79af53ef82 (127.5.240.194:41984) I0703 01:40:47.771128 17819 raft_consensus.cc:457] T 94cc179826c046169908bf910ed7c7bb P e950a6c6229e4f80ba63b793d64d1998 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 14cbc64bc75444b8bbd9fbdcefdb86be) I0703 01:40:47.771266 17819 raft_consensus.cc:2730] T 94cc179826c046169908bf910ed7c7bb P e950a6c6229e4f80ba63b793d64d1998: Snoozing failure detection for 2.188s (starting election) I0703 01:40:47.771390 17819 raft_consensus.cc:479] T 94cc179826c046169908bf910ed7c7bb P e950a6c6229e4f80ba63b793d64d1998 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "14cbc64bc75444b8bbd9fbdcefdb86be" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 40856 } } peers { permanent_uuid: "8ace61239bda46ba874d4b094fff117d" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 53360 } } peers { permanent_uuid: "e950a6c6229e4f80ba63b793d64d1998" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 32786 } } peers { permanent_uuid: "3d35a3bef6e6470d99c05c79af53ef82" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 41984 } } I0703 01:40:47.772248 17819 leader_election.cc:243] T 94cc179826c046169908bf910ed7c7bb P e950a6c6229e4f80ba63b793d64d1998 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 14cbc64bc75444b8bbd9fbdcefdb86be (127.5.240.197:40856) I0703 01:40:47.772769 17819 leader_election.cc:243] T 94cc179826c046169908bf910ed7c7bb P e950a6c6229e4f80ba63b793d64d1998 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 8ace61239bda46ba874d4b094fff117d (127.5.240.193:53360) I0703 01:40:47.772958 17819 leader_election.cc:243] T 94cc179826c046169908bf910ed7c7bb P e950a6c6229e4f80ba63b793d64d1998 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 3d35a3bef6e6470d99c05c79af53ef82 (127.5.240.194:41984) I0703 01:40:47.779295 16940 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "94cc179826c046169908bf910ed7c7bb" candidate_uuid: "3d35a3bef6e6470d99c05c79af53ef82" candidate_term: 2 candidate_status { last_received { term: 1 index: 375 } } ignore_live_leader: false dest_uuid: "8ace61239bda46ba874d4b094fff117d" is_pre_election: true I0703 01:40:47.779479 16940 raft_consensus.cc:2730] T 94cc179826c046169908bf910ed7c7bb P 8ace61239bda46ba874d4b094fff117d: Snoozing failure detection for 2.111s (vote granted) I0703 01:40:47.779600 16940 raft_consensus.cc:2243] T 94cc179826c046169908bf910ed7c7bb P 8ace61239bda46ba874d4b094fff117d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d35a3bef6e6470d99c05c79af53ef82 in term 1. I0703 01:40:47.780081 16995 leader_election.cc:379] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 [CANDIDATE]: Term 2 pre-election: Vote granted by peer 8ace61239bda46ba874d4b094fff117d (127.5.240.193:53360) I0703 01:40:47.780149 16995 leader_election.cc:270] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. I0703 01:40:47.780546 17813 raft_consensus.cc:2730] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82: Snoozing failure detection for 2.142s (election complete) I0703 01:40:47.780644 17813 raft_consensus.cc:2537] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0703 01:40:47.780720 17813 raft_consensus.cc:457] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 14cbc64bc75444b8bbd9fbdcefdb86be) I0703 01:40:47.780774 17813 raft_consensus.cc:2730] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82: Snoozing failure detection for 2.155s (starting election) I0703 01:40:47.780848 17813 raft_consensus.cc:2782] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 [term 1 FOLLOWER]: Advancing to term 2 I0703 01:40:47.784160 17307 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "94cc179826c046169908bf910ed7c7bb" candidate_uuid: "3d35a3bef6e6470d99c05c79af53ef82" candidate_term: 2 candidate_status { last_received { term: 1 index: 375 } } ignore_live_leader: false dest_uuid: "e950a6c6229e4f80ba63b793d64d1998" is_pre_election: true I0703 01:40:47.784384 17307 raft_consensus.cc:2730] T 94cc179826c046169908bf910ed7c7bb P e950a6c6229e4f80ba63b793d64d1998: Snoozing failure detection for 1.517s (vote granted) I0703 01:40:47.784485 17307 raft_consensus.cc:2243] T 94cc179826c046169908bf910ed7c7bb P e950a6c6229e4f80ba63b793d64d1998 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d35a3bef6e6470d99c05c79af53ef82 in term 1. I0703 01:40:47.784986 16994 leader_election.cc:379] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 [CANDIDATE]: Term 2 pre-election: Vote granted by peer e950a6c6229e4f80ba63b793d64d1998 (127.5.240.196:32786) I0703 01:40:47.785320 17813 raft_consensus.cc:479] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "14cbc64bc75444b8bbd9fbdcefdb86be" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 40856 } } peers { permanent_uuid: "8ace61239bda46ba874d4b094fff117d" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 53360 } } peers { permanent_uuid: "e950a6c6229e4f80ba63b793d64d1998" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 32786 } } peers { permanent_uuid: "3d35a3bef6e6470d99c05c79af53ef82" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 41984 } } I0703 01:40:47.785892 17813 leader_election.cc:243] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 [CANDIDATE]: Term 2 election: Requesting vote from peer 14cbc64bc75444b8bbd9fbdcefdb86be (127.5.240.197:40856) I0703 01:40:47.786283 17813 leader_election.cc:243] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 [CANDIDATE]: Term 2 election: Requesting vote from peer 8ace61239bda46ba874d4b094fff117d (127.5.240.193:53360) I0703 01:40:47.786753 17429 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "94cc179826c046169908bf910ed7c7bb" candidate_uuid: "3d35a3bef6e6470d99c05c79af53ef82" candidate_term: 2 candidate_status { last_received { term: 1 index: 375 } } ignore_live_leader: false dest_uuid: "14cbc64bc75444b8bbd9fbdcefdb86be" I0703 01:40:47.786981 17429 raft_consensus.cc:2782] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be [term 1 FOLLOWER]: Advancing to term 2 I0703 01:40:47.787055 16940 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "94cc179826c046169908bf910ed7c7bb" candidate_uuid: "3d35a3bef6e6470d99c05c79af53ef82" candidate_term: 2 candidate_status { last_received { term: 1 index: 375 } } ignore_live_leader: false dest_uuid: "8ace61239bda46ba874d4b094fff117d" I0703 01:40:47.787159 17429 raft_consensus.cc:2730] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be: Snoozing failure detection for 2.052s (vote granted) I0703 01:40:47.787179 16940 raft_consensus.cc:2782] T 94cc179826c046169908bf910ed7c7bb P 8ace61239bda46ba874d4b094fff117d [term 1 FOLLOWER]: Advancing to term 2 I0703 01:40:47.787262 16940 raft_consensus.cc:2730] T 94cc179826c046169908bf910ed7c7bb P 8ace61239bda46ba874d4b094fff117d: Snoozing failure detection for 2.022s (vote granted) I0703 01:40:47.787154 17428 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "94cc179826c046169908bf910ed7c7bb" candidate_uuid: "e950a6c6229e4f80ba63b793d64d1998" candidate_term: 2 candidate_status { last_received { term: 1 index: 375 } } ignore_live_leader: false dest_uuid: "14cbc64bc75444b8bbd9fbdcefdb86be" is_pre_election: true I0703 01:40:47.788141 17813 leader_election.cc:243] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 [CANDIDATE]: Term 2 election: Requesting vote from peer e950a6c6229e4f80ba63b793d64d1998 (127.5.240.196:32786) I0703 01:40:47.788288 17427 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "94cc179826c046169908bf910ed7c7bb" candidate_uuid: "8ace61239bda46ba874d4b094fff117d" candidate_term: 2 candidate_status { last_received { term: 1 index: 375 } } ignore_live_leader: false dest_uuid: "14cbc64bc75444b8bbd9fbdcefdb86be" is_pre_election: true I0703 01:40:47.789017 17307 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "94cc179826c046169908bf910ed7c7bb" candidate_uuid: "3d35a3bef6e6470d99c05c79af53ef82" candidate_term: 2 candidate_status { last_received { term: 1 index: 375 } } ignore_live_leader: false dest_uuid: "e950a6c6229e4f80ba63b793d64d1998" I0703 01:40:47.789141 17307 raft_consensus.cc:2782] T 94cc179826c046169908bf910ed7c7bb P e950a6c6229e4f80ba63b793d64d1998 [term 1 FOLLOWER]: Advancing to term 2 I0703 01:40:47.789212 17307 raft_consensus.cc:2730] T 94cc179826c046169908bf910ed7c7bb P e950a6c6229e4f80ba63b793d64d1998: Snoozing failure detection for 1.918s (vote granted) I0703 01:40:47.789664 17429 raft_consensus.cc:2243] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d35a3bef6e6470d99c05c79af53ef82 in term 2. I0703 01:40:47.789714 16940 raft_consensus.cc:2243] T 94cc179826c046169908bf910ed7c7bb P 8ace61239bda46ba874d4b094fff117d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d35a3bef6e6470d99c05c79af53ef82 in term 2. I0703 01:40:47.789834 17428 raft_consensus.cc:2218] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e950a6c6229e4f80ba63b793d64d1998 for term 2 because replica is already servicing an update from a current leader or another vote. I0703 01:40:47.790191 17427 raft_consensus.cc:2218] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8ace61239bda46ba874d4b094fff117d for term 2 because replica is already servicing an update from a current leader or another vote. I0703 01:40:47.790830 16871 leader_election.cc:393] T 94cc179826c046169908bf910ed7c7bb P 8ace61239bda46ba874d4b094fff117d [CANDIDATE]: Term 2 pre-election: Vote denied by peer 14cbc64bc75444b8bbd9fbdcefdb86be (127.5.240.197:40856). Message: Service unavailable: T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8ace61239bda46ba874d4b094fff117d for term 2 because replica is already servicing an update from a current leader or another vote. I0703 01:40:47.791071 16994 leader_election.cc:379] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 [CANDIDATE]: Term 2 election: Vote granted by peer 14cbc64bc75444b8bbd9fbdcefdb86be (127.5.240.197:40856) I0703 01:40:47.791522 17307 raft_consensus.cc:2243] T 94cc179826c046169908bf910ed7c7bb P e950a6c6229e4f80ba63b793d64d1998 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d35a3bef6e6470d99c05c79af53ef82 in term 2. I0703 01:40:47.791788 17238 leader_election.cc:393] T 94cc179826c046169908bf910ed7c7bb P e950a6c6229e4f80ba63b793d64d1998 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 14cbc64bc75444b8bbd9fbdcefdb86be (127.5.240.197:40856). Message: Service unavailable: T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e950a6c6229e4f80ba63b793d64d1998 for term 2 because replica is already servicing an update from a current leader or another vote. I0703 01:40:47.792280 16994 leader_election.cc:379] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 [CANDIDATE]: Term 2 election: Vote granted by peer e950a6c6229e4f80ba63b793d64d1998 (127.5.240.196:32786) I0703 01:40:47.792382 16995 leader_election.cc:379] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 [CANDIDATE]: Term 2 election: Vote granted by peer 8ace61239bda46ba874d4b094fff117d (127.5.240.193:53360) I0703 01:40:47.792496 16995 leader_election.cc:270] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0703 01:40:47.792804 17813 raft_consensus.cc:2730] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82: Snoozing failure detection for 1.835s (election complete) I0703 01:40:47.792906 17813 raft_consensus.cc:2537] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 [term 2 FOLLOWER]: Leader election won for term 2 I0703 01:40:47.793967 17813 raft_consensus.cc:593] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 [term 2 LEADER]: Becoming Leader. State: Replica: 3d35a3bef6e6470d99c05c79af53ef82, State: Running, Role: LEADER I0703 01:40:47.794636 17813 consensus_queue.cc:204] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 374, Committed index: 374, Last appended: 1.375, Last appended by leader: 375, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "14cbc64bc75444b8bbd9fbdcefdb86be" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 40856 } } peers { permanent_uuid: "8ace61239bda46ba874d4b094fff117d" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 53360 } } peers { permanent_uuid: "e950a6c6229e4f80ba63b793d64d1998" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 32786 } } peers { permanent_uuid: "3d35a3bef6e6470d99c05c79af53ef82" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 41984 } } I0703 01:40:47.798400 16940 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "94cc179826c046169908bf910ed7c7bb" candidate_uuid: "e950a6c6229e4f80ba63b793d64d1998" candidate_term: 2 candidate_status { last_received { term: 1 index: 375 } } ignore_live_leader: false dest_uuid: "8ace61239bda46ba874d4b094fff117d" is_pre_election: true I0703 01:40:47.798619 16940 raft_consensus.cc:2173] T 94cc179826c046169908bf910ed7c7bb P 8ace61239bda46ba874d4b094fff117d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e950a6c6229e4f80ba63b793d64d1998 in current term 2: Already voted for candidate 3d35a3bef6e6470d99c05c79af53ef82 in this term. I0703 01:40:47.799291 17062 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "94cc179826c046169908bf910ed7c7bb" candidate_uuid: "8ace61239bda46ba874d4b094fff117d" candidate_term: 2 candidate_status { last_received { term: 1 index: 375 } } ignore_live_leader: false dest_uuid: "3d35a3bef6e6470d99c05c79af53ef82" is_pre_election: true I0703 01:40:47.799844 17062 raft_consensus.cc:2204] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 8ace61239bda46ba874d4b094fff117d for term 2 because replica is either leader or believes a valid leader to be alive. I0703 01:40:47.800626 16872 leader_election.cc:393] T 94cc179826c046169908bf910ed7c7bb P 8ace61239bda46ba874d4b094fff117d [CANDIDATE]: Term 2 pre-election: Vote denied by peer 3d35a3bef6e6470d99c05c79af53ef82 (127.5.240.194:41984). Message: Invalid argument: T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 8ace61239bda46ba874d4b094fff117d for term 2 because replica is either leader or believes a valid leader to be alive. I0703 01:40:47.800734 16872 leader_election.cc:270] T 94cc179826c046169908bf910ed7c7bb P 8ace61239bda46ba874d4b094fff117d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0703 01:40:47.800774 16818 catalog_manager.cc:3798] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 reported cstate change: term changed from 1 to 2, leader changed from 14cbc64bc75444b8bbd9fbdcefdb86be (127.5.240.197) to 3d35a3bef6e6470d99c05c79af53ef82 (127.5.240.194). New cstate: current_term: 2 leader_uuid: "3d35a3bef6e6470d99c05c79af53ef82" committed_config { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "14cbc64bc75444b8bbd9fbdcefdb86be" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 40856 } } peers { permanent_uuid: "8ace61239bda46ba874d4b094fff117d" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 53360 } } peers { permanent_uuid: "e950a6c6229e4f80ba63b793d64d1998" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 32786 } } peers { permanent_uuid: "3d35a3bef6e6470d99c05c79af53ef82" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 41984 } } } I0703 01:40:47.801362 17814 raft_consensus.cc:2730] T 94cc179826c046169908bf910ed7c7bb P 8ace61239bda46ba874d4b094fff117d: Snoozing failure detection for 1.960s (election complete) I0703 01:40:47.801455 17814 raft_consensus.cc:2485] T 94cc179826c046169908bf910ed7c7bb P 8ace61239bda46ba874d4b094fff117d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0703 01:40:47.801946 17239 leader_election.cc:393] T 94cc179826c046169908bf910ed7c7bb P e950a6c6229e4f80ba63b793d64d1998 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 8ace61239bda46ba874d4b094fff117d (127.5.240.193:53360). Message: Invalid argument: T 94cc179826c046169908bf910ed7c7bb P 8ace61239bda46ba874d4b094fff117d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e950a6c6229e4f80ba63b793d64d1998 in current term 2: Already voted for candidate 3d35a3bef6e6470d99c05c79af53ef82 in this term. I0703 01:40:47.802024 17239 leader_election.cc:270] T 94cc179826c046169908bf910ed7c7bb P e950a6c6229e4f80ba63b793d64d1998 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0703 01:40:47.802161 17307 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "94cc179826c046169908bf910ed7c7bb" candidate_uuid: "8ace61239bda46ba874d4b094fff117d" candidate_term: 2 candidate_status { last_received { term: 1 index: 375 } } ignore_live_leader: false dest_uuid: "e950a6c6229e4f80ba63b793d64d1998" is_pre_election: true I0703 01:40:47.802409 17307 raft_consensus.cc:2173] T 94cc179826c046169908bf910ed7c7bb P e950a6c6229e4f80ba63b793d64d1998 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8ace61239bda46ba874d4b094fff117d in current term 2: Already voted for candidate 3d35a3bef6e6470d99c05c79af53ef82 in this term. I0703 01:40:47.802759 17819 raft_consensus.cc:2730] T 94cc179826c046169908bf910ed7c7bb P e950a6c6229e4f80ba63b793d64d1998: Snoozing failure detection for 2.015s (election complete) I0703 01:40:47.802821 17819 raft_consensus.cc:2485] T 94cc179826c046169908bf910ed7c7bb P e950a6c6229e4f80ba63b793d64d1998 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0703 01:40:47.802942 16871 leader_election.cc:393] T 94cc179826c046169908bf910ed7c7bb P 8ace61239bda46ba874d4b094fff117d [CANDIDATE]: Term 2 pre-election: Vote denied by peer e950a6c6229e4f80ba63b793d64d1998 (127.5.240.196:32786). Message: Invalid argument: T 94cc179826c046169908bf910ed7c7bb P e950a6c6229e4f80ba63b793d64d1998 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8ace61239bda46ba874d4b094fff117d in current term 2: Already voted for candidate 3d35a3bef6e6470d99c05c79af53ef82 in this term. I0703 01:40:47.804517 17307 raft_consensus.cc:1081] T 94cc179826c046169908bf910ed7c7bb P e950a6c6229e4f80ba63b793d64d1998 [term 2 FOLLOWER]: Refusing update from remote peer 3d35a3bef6e6470d99c05c79af53ef82: Log matching property violated. Preceding OpId in replica: term: 1 index: 375. Preceding OpId from leader: term: 2 index: 377. (index mismatch) I0703 01:40:47.804832 16940 raft_consensus.cc:1081] T 94cc179826c046169908bf910ed7c7bb P 8ace61239bda46ba874d4b094fff117d [term 2 FOLLOWER]: Refusing update from remote peer 3d35a3bef6e6470d99c05c79af53ef82: Log matching property violated. Preceding OpId in replica: term: 1 index: 375. Preceding OpId from leader: term: 2 index: 377. (index mismatch) I0703 01:40:47.805303 17427 raft_consensus.cc:1081] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be [term 2 FOLLOWER]: Refusing update from remote peer 3d35a3bef6e6470d99c05c79af53ef82: Log matching property violated. Preceding OpId in replica: term: 1 index: 375. Preceding OpId from leader: term: 2 index: 377. (index mismatch) I0703 01:40:47.805359 17813 consensus_queue.cc:962] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e950a6c6229e4f80ba63b793d64d1998" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 32786 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 376, Last known committed idx: 375, Time since last communication: 0.000s I0703 01:40:47.805734 17827 consensus_queue.cc:962] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ace61239bda46ba874d4b094fff117d" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 53360 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 376, Last known committed idx: 374, Time since last communication: 0.000s I0703 01:40:47.806634 17813 consensus_queue.cc:962] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14cbc64bc75444b8bbd9fbdcefdb86be" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 40856 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 376, Last known committed idx: 375, Time since last communication: 0.000s I0703 01:40:47.808169 17062 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "94cc179826c046169908bf910ed7c7bb" candidate_uuid: "e950a6c6229e4f80ba63b793d64d1998" candidate_term: 2 candidate_status { last_received { term: 1 index: 375 } } ignore_live_leader: false dest_uuid: "3d35a3bef6e6470d99c05c79af53ef82" is_pre_election: true I0703 01:40:47.808322 17062 raft_consensus.cc:2204] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate e950a6c6229e4f80ba63b793d64d1998 for term 2 because replica is either leader or believes a valid leader to be alive. I0703 01:40:47.809064 17239 leader_election.cc:393] T 94cc179826c046169908bf910ed7c7bb P e950a6c6229e4f80ba63b793d64d1998 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 3d35a3bef6e6470d99c05c79af53ef82 (127.5.240.194:41984). Message: Invalid argument: T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate e950a6c6229e4f80ba63b793d64d1998 for term 2 because replica is either leader or believes a valid leader to be alive. I0703 01:40:48.174852 17062 consensus_peers.cc:455] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 -> Peer e950a6c6229e4f80ba63b793d64d1998 (127.5.240.196:32786): Closing peer: e950a6c6229e4f80ba63b793d64d1998 I0703 01:40:48.174924 17062 consensus_peers.cc:455] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 -> Peer 8ace61239bda46ba874d4b094fff117d (127.5.240.193:53360): Closing peer: 8ace61239bda46ba874d4b094fff117d I0703 01:40:48.174962 17062 consensus_peers.cc:455] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 -> Peer 14cbc64bc75444b8bbd9fbdcefdb86be (127.5.240.197:40856): Closing peer: 14cbc64bc75444b8bbd9fbdcefdb86be I0703 01:40:48.175173 17062 consensus_queue.cc:204] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 429, Committed index: 429, Last appended: 2.430, Last appended by leader: 375, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 431 OBSOLETE_local: false peers { permanent_uuid: "8ace61239bda46ba874d4b094fff117d" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 53360 } } peers { permanent_uuid: "e950a6c6229e4f80ba63b793d64d1998" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 32786 } } peers { permanent_uuid: "3d35a3bef6e6470d99c05c79af53ef82" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 41984 } } I0703 01:40:48.177325 17307 raft_consensus.cc:1081] T 94cc179826c046169908bf910ed7c7bb P e950a6c6229e4f80ba63b793d64d1998 [term 2 FOLLOWER]: Refusing update from remote peer 3d35a3bef6e6470d99c05c79af53ef82: Log matching property violated. Preceding OpId in replica: term: 2 index: 430. Preceding OpId from leader: term: 2 index: 431. (index mismatch) I0703 01:40:48.177325 16940 raft_consensus.cc:1081] T 94cc179826c046169908bf910ed7c7bb P 8ace61239bda46ba874d4b094fff117d [term 2 FOLLOWER]: Refusing update from remote peer 3d35a3bef6e6470d99c05c79af53ef82: Log matching property violated. Preceding OpId in replica: term: 2 index: 430. Preceding OpId from leader: term: 2 index: 431. (index mismatch) I0703 01:40:48.177969 17848 consensus_queue.cc:962] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e950a6c6229e4f80ba63b793d64d1998" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 32786 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 431, Last known committed idx: 430, Time since last communication: 0.000s I0703 01:40:48.178936 17848 consensus_queue.cc:962] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ace61239bda46ba874d4b094fff117d" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 53360 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 431, Last known committed idx: 429, Time since last communication: 0.000s I0703 01:40:48.180954 17827 raft_consensus.cc:2684] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 [term 2 LEADER]: Committing config change with OpId 2.431: config changed from index 94 to 431, VOTER 14cbc64bc75444b8bbd9fbdcefdb86be (127.5.240.197) evicted. New config: { opid_index: 431 OBSOLETE_local: false peers { permanent_uuid: "8ace61239bda46ba874d4b094fff117d" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 53360 } } peers { permanent_uuid: "e950a6c6229e4f80ba63b793d64d1998" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 32786 } } peers { permanent_uuid: "3d35a3bef6e6470d99c05c79af53ef82" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 41984 } } } I0703 01:40:48.181699 17307 raft_consensus.cc:2684] T 94cc179826c046169908bf910ed7c7bb P e950a6c6229e4f80ba63b793d64d1998 [term 2 FOLLOWER]: Committing config change with OpId 2.431: config changed from index 94 to 431, VOTER 14cbc64bc75444b8bbd9fbdcefdb86be (127.5.240.197) evicted. New config: { opid_index: 431 OBSOLETE_local: false peers { permanent_uuid: "8ace61239bda46ba874d4b094fff117d" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 53360 } } peers { permanent_uuid: "e950a6c6229e4f80ba63b793d64d1998" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 32786 } } peers { permanent_uuid: "3d35a3bef6e6470d99c05c79af53ef82" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 41984 } } } I0703 01:40:48.184919 16940 raft_consensus.cc:2684] T 94cc179826c046169908bf910ed7c7bb P 8ace61239bda46ba874d4b094fff117d [term 2 FOLLOWER]: Committing config change with OpId 2.431: config changed from index 94 to 431, VOTER 14cbc64bc75444b8bbd9fbdcefdb86be (127.5.240.197) evicted. New config: { opid_index: 431 OBSOLETE_local: false peers { permanent_uuid: "8ace61239bda46ba874d4b094fff117d" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 53360 } } peers { permanent_uuid: "e950a6c6229e4f80ba63b793d64d1998" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 32786 } } peers { permanent_uuid: "3d35a3bef6e6470d99c05c79af53ef82" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 41984 } } } I0703 01:40:48.185920 16819 catalog_manager.cc:3798] T 94cc179826c046169908bf910ed7c7bb P e950a6c6229e4f80ba63b793d64d1998 reported cstate change: config changed from index 94 to 431, VOTER 14cbc64bc75444b8bbd9fbdcefdb86be (127.5.240.197) evicted. New cstate: current_term: 2 leader_uuid: "3d35a3bef6e6470d99c05c79af53ef82" committed_config { opid_index: 431 OBSOLETE_local: false peers { permanent_uuid: "8ace61239bda46ba874d4b094fff117d" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 53360 } } peers { permanent_uuid: "e950a6c6229e4f80ba63b793d64d1998" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 32786 } } peers { permanent_uuid: "3d35a3bef6e6470d99c05c79af53ef82" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 41984 } } } I0703 01:40:48.189203 16819 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 94cc179826c046169908bf910ed7c7bb on 14cbc64bc75444b8bbd9fbdcefdb86be (127.5.240.197:40856) (TS 14cbc64bc75444b8bbd9fbdcefdb86be not found in new config with opid_index 431) I0703 01:40:48.191007 17409 tablet_service.cc:800] Processing DeleteTablet for tablet 94cc179826c046169908bf910ed7c7bb with delete_type TABLET_DATA_TOMBSTONED (TS 14cbc64bc75444b8bbd9fbdcefdb86be not found in new config with opid_index 431) from {username='jenkins'} at 127.5.240.197:41206 I0703 01:40:48.191745 17892 tablet_replica.cc:261] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be: stopping tablet replica I0703 01:40:48.191939 17892 maintenance_manager.cc:237] P 14cbc64bc75444b8bbd9fbdcefdb86be: Unregistered op CompactRowSetsOp(94cc179826c046169908bf910ed7c7bb) I0703 01:40:48.192070 17892 maintenance_manager.cc:237] P 14cbc64bc75444b8bbd9fbdcefdb86be: Unregistered op MinorDeltaCompactionOp(94cc179826c046169908bf910ed7c7bb) I0703 01:40:48.192159 17892 maintenance_manager.cc:237] P 14cbc64bc75444b8bbd9fbdcefdb86be: Unregistered op MajorDeltaCompactionOp(94cc179826c046169908bf910ed7c7bb) I0703 01:40:48.192214 17892 maintenance_manager.cc:237] P 14cbc64bc75444b8bbd9fbdcefdb86be: Unregistered op UndoDeltaBlockGCOp(94cc179826c046169908bf910ed7c7bb) I0703 01:40:48.192292 17892 maintenance_manager.cc:237] P 14cbc64bc75444b8bbd9fbdcefdb86be: Unregistered op FlushMRSOp(94cc179826c046169908bf910ed7c7bb) I0703 01:40:48.192342 17892 maintenance_manager.cc:237] P 14cbc64bc75444b8bbd9fbdcefdb86be: Unregistered op FlushDeltaMemStoresOp(94cc179826c046169908bf910ed7c7bb) I0703 01:40:48.192379 17892 maintenance_manager.cc:237] P 14cbc64bc75444b8bbd9fbdcefdb86be: Unregistered op LogGCOp(94cc179826c046169908bf910ed7c7bb) I0703 01:40:48.192445 17892 raft_consensus.cc:2040] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be [term 2 FOLLOWER]: Raft consensus shutting down. I0703 01:40:48.192917 17892 pending_rounds.cc:63] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be: Trying to abort 1 pending transactions. I0703 01:40:48.192963 17892 pending_rounds.cc:68] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be: Aborting transaction as it isn't in flight: id { term: 2 index: 430 } timestamp: 6269367288505745408 op_type: WRITE_OP write_request { tablet_id: "94cc179826c046169908bf910ed7c7bb" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true } } row_operations { rows: "\001\007\000y\356pH\342E\005;\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "ea3db73e7392ba38" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 6269367288478236672 } request_id { client_id: "def370e9747a4800ad71729cd18e0af9" seq_no: 426 first_incomplete_seq_no: 426 attempt_no: 0 } I0703 01:40:48.193356 17892 raft_consensus.cc:2067] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be [term 2 FOLLOWER]: Raft consensus is shut down! I0703 01:40:48.194629 17892 ts_tablet_manager.cc:1369] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:40:48.198313 17892 ts_tablet_manager.cc:1382] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.430 I0703 01:40:48.198402 17892 log.cc:986] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal/wals/94cc179826c046169908bf910ed7c7bb I0703 01:40:48.199214 16801 catalog_manager.cc:3103] TS 14cbc64bc75444b8bbd9fbdcefdb86be (127.5.240.197:40856): tablet 94cc179826c046169908bf910ed7c7bb (table TestTable [id=144618e6d82940cd9252cc126e332762]) successfully deleted W0703 01:40:49.064640 17902 flags.cc:406] Enabled unsafe flag: --never_fsync=true I0703 01:40:49.370882 17062 consensus_peers.cc:455] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 -> Peer e950a6c6229e4f80ba63b793d64d1998 (127.5.240.196:32786): Closing peer: e950a6c6229e4f80ba63b793d64d1998 I0703 01:40:49.370983 17062 consensus_peers.cc:455] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 -> Peer 8ace61239bda46ba874d4b094fff117d (127.5.240.193:53360): Closing peer: 8ace61239bda46ba874d4b094fff117d I0703 01:40:49.371212 17062 consensus_queue.cc:204] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 667, Committed index: 667, Last appended: 2.668, Last appended by leader: 375, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 669 OBSOLETE_local: false peers { permanent_uuid: "8ace61239bda46ba874d4b094fff117d" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 53360 } } peers { permanent_uuid: "e950a6c6229e4f80ba63b793d64d1998" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 32786 } } peers { permanent_uuid: "3d35a3bef6e6470d99c05c79af53ef82" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 41984 } } peers { permanent_uuid: "47fc1a635c3a44d58fb94e27f2551847" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46477 } } I0703 01:40:49.373306 16939 raft_consensus.cc:1081] T 94cc179826c046169908bf910ed7c7bb P 8ace61239bda46ba874d4b094fff117d [term 2 FOLLOWER]: Refusing update from remote peer 3d35a3bef6e6470d99c05c79af53ef82: Log matching property violated. Preceding OpId in replica: term: 2 index: 667. Preceding OpId from leader: term: 2 index: 669. (index mismatch) I0703 01:40:49.373888 17827 consensus_queue.cc:962] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ace61239bda46ba874d4b094fff117d" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 53360 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 669, Last known committed idx: 667, Time since last communication: 0.000s I0703 01:40:49.374581 17307 raft_consensus.cc:1081] T 94cc179826c046169908bf910ed7c7bb P e950a6c6229e4f80ba63b793d64d1998 [term 2 FOLLOWER]: Refusing update from remote peer 3d35a3bef6e6470d99c05c79af53ef82: Log matching property violated. Preceding OpId in replica: term: 2 index: 668. Preceding OpId from leader: term: 2 index: 669. (index mismatch) I0703 01:40:49.375471 17827 consensus_queue.cc:962] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e950a6c6229e4f80ba63b793d64d1998" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 32786 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 669, Last known committed idx: 667, Time since last communication: 0.000s I0703 01:40:49.377748 17848 raft_consensus.cc:2684] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 [term 2 LEADER]: Committing config change with OpId 2.669: config changed from index 431 to 669, VOTER 47fc1a635c3a44d58fb94e27f2551847 (127.5.240.195) added. New config: { opid_index: 669 OBSOLETE_local: false peers { permanent_uuid: "8ace61239bda46ba874d4b094fff117d" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 53360 } } peers { permanent_uuid: "e950a6c6229e4f80ba63b793d64d1998" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 32786 } } peers { permanent_uuid: "3d35a3bef6e6470d99c05c79af53ef82" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 41984 } } peers { permanent_uuid: "47fc1a635c3a44d58fb94e27f2551847" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46477 } } } I0703 01:40:49.378383 17307 raft_consensus.cc:2684] T 94cc179826c046169908bf910ed7c7bb P e950a6c6229e4f80ba63b793d64d1998 [term 2 FOLLOWER]: Committing config change with OpId 2.669: config changed from index 431 to 669, VOTER 47fc1a635c3a44d58fb94e27f2551847 (127.5.240.195) added. New config: { opid_index: 669 OBSOLETE_local: false peers { permanent_uuid: "8ace61239bda46ba874d4b094fff117d" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 53360 } } peers { permanent_uuid: "e950a6c6229e4f80ba63b793d64d1998" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 32786 } } peers { permanent_uuid: "3d35a3bef6e6470d99c05c79af53ef82" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 41984 } } peers { permanent_uuid: "47fc1a635c3a44d58fb94e27f2551847" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46477 } } } W0703 01:40:49.380959 16994 consensus_peers.cc:433] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 -> Peer 47fc1a635c3a44d58fb94e27f2551847 (127.5.240.195:46477): Couldn't send request to peer 47fc1a635c3a44d58fb94e27f2551847 for tablet 94cc179826c046169908bf910ed7c7bb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 94cc179826c046169908bf910ed7c7bb. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:40:49.381901 16939 raft_consensus.cc:2684] T 94cc179826c046169908bf910ed7c7bb P 8ace61239bda46ba874d4b094fff117d [term 2 FOLLOWER]: Committing config change with OpId 2.669: config changed from index 431 to 669, VOTER 47fc1a635c3a44d58fb94e27f2551847 (127.5.240.195) added. New config: { opid_index: 669 OBSOLETE_local: false peers { permanent_uuid: "8ace61239bda46ba874d4b094fff117d" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 53360 } } peers { permanent_uuid: "e950a6c6229e4f80ba63b793d64d1998" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 32786 } } peers { permanent_uuid: "3d35a3bef6e6470d99c05c79af53ef82" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 41984 } } peers { permanent_uuid: "47fc1a635c3a44d58fb94e27f2551847" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46477 } } } I0703 01:40:49.382418 16820 catalog_manager.cc:3798] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 reported cstate change: config changed from index 431 to 669, VOTER 47fc1a635c3a44d58fb94e27f2551847 (127.5.240.195) added. New cstate: current_term: 2 leader_uuid: "3d35a3bef6e6470d99c05c79af53ef82" committed_config { opid_index: 669 OBSOLETE_local: false peers { permanent_uuid: "8ace61239bda46ba874d4b094fff117d" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 53360 } } peers { permanent_uuid: "e950a6c6229e4f80ba63b793d64d1998" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 32786 } } peers { permanent_uuid: "3d35a3bef6e6470d99c05c79af53ef82" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 41984 } } peers { permanent_uuid: "47fc1a635c3a44d58fb94e27f2551847" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46477 } } } I0703 01:40:49.988286 17987 ts_tablet_manager.cc:669] T 94cc179826c046169908bf910ed7c7bb P 47fc1a635c3a44d58fb94e27f2551847: Initiating tablet copy from peer 3d35a3bef6e6470d99c05c79af53ef82 (127.5.240.194:41984) I0703 01:40:49.988557 17987 tablet_copy_client.cc:239] T 94cc179826c046169908bf910ed7c7bb P 47fc1a635c3a44d58fb94e27f2551847: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.194:41984 I0703 01:40:49.994995 17083 tablet_copy_service.cc:135] P 3d35a3bef6e6470d99c05c79af53ef82: Received BeginTabletCopySession request for tablet 94cc179826c046169908bf910ed7c7bb from peer 47fc1a635c3a44d58fb94e27f2551847 ({username='jenkins'} at 127.5.240.195:34501) I0703 01:40:49.995154 17083 tablet_copy_service.cc:156] P 3d35a3bef6e6470d99c05c79af53ef82: Beginning new tablet copy session on tablet 94cc179826c046169908bf910ed7c7bb from peer 47fc1a635c3a44d58fb94e27f2551847 at {username='jenkins'} at 127.5.240.195:34501: session id = 47fc1a635c3a44d58fb94e27f2551847-94cc179826c046169908bf910ed7c7bb I0703 01:40:49.997232 17083 tablet_copy_source_session.cc:202] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:40:49.999867 17987 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 94cc179826c046169908bf910ed7c7bb. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:40:50.010036 17987 ts_tablet_manager.cc:1173] T 94cc179826c046169908bf910ed7c7bb P 47fc1a635c3a44d58fb94e27f2551847: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:40:50.010298 17987 tablet_copy_client.cc:566] T 94cc179826c046169908bf910ed7c7bb P 47fc1a635c3a44d58fb94e27f2551847: tablet copy: Starting download of 0 data blocks... I0703 01:40:50.010697 17987 tablet_copy_client.cc:529] T 94cc179826c046169908bf910ed7c7bb P 47fc1a635c3a44d58fb94e27f2551847: tablet copy: Starting download of 1 WAL segments... I0703 01:40:50.019810 17987 tablet_copy_client.cc:405] T 94cc179826c046169908bf910ed7c7bb P 47fc1a635c3a44d58fb94e27f2551847: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:40:50.021847 17987 ts_tablet_manager.cc:1028] T 94cc179826c046169908bf910ed7c7bb P 47fc1a635c3a44d58fb94e27f2551847: Bootstrapping tablet I0703 01:40:50.022100 17987 tablet_bootstrap.cc:427] T 94cc179826c046169908bf910ed7c7bb P 47fc1a635c3a44d58fb94e27f2551847: Bootstrap starting. I0703 01:40:50.125078 17987 log.cc:525] T 94cc179826c046169908bf910ed7c7bb P 47fc1a635c3a44d58fb94e27f2551847: Log is configured to *not* fsync() on all Append() calls W0703 01:40:50.389780 16994 consensus_peers.cc:433] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 -> Peer 47fc1a635c3a44d58fb94e27f2551847 (127.5.240.195:46477): Couldn't send request to peer 47fc1a635c3a44d58fb94e27f2551847 for tablet 94cc179826c046169908bf910ed7c7bb. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 2 times. I0703 01:40:50.651135 17987 tablet_bootstrap.cc:427] T 94cc179826c046169908bf910ed7c7bb P 47fc1a635c3a44d58fb94e27f2551847: Bootstrap replayed 1/1 log segments. Stats: ops{read=713 overwritten=0 applied=712 ignored=0} inserts{seen=707 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates I0703 01:40:50.651652 17987 tablet_bootstrap.cc:427] T 94cc179826c046169908bf910ed7c7bb P 47fc1a635c3a44d58fb94e27f2551847: Bootstrap complete. I0703 01:40:50.652029 17987 ts_tablet_manager.cc:1045] T 94cc179826c046169908bf910ed7c7bb P 47fc1a635c3a44d58fb94e27f2551847: Time spent bootstrapping tablet: real 0.630s user 0.546s sys 0.037s I0703 01:40:50.656716 17987 raft_consensus.cc:325] T 94cc179826c046169908bf910ed7c7bb P 47fc1a635c3a44d58fb94e27f2551847 [term 2 FOLLOWER]: Replica starting. Triggering 1 pending transactions. Active config: opid_index: 669 OBSOLETE_local: false peers { permanent_uuid: "8ace61239bda46ba874d4b094fff117d" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 53360 } } peers { permanent_uuid: "e950a6c6229e4f80ba63b793d64d1998" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 32786 } } peers { permanent_uuid: "3d35a3bef6e6470d99c05c79af53ef82" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 41984 } } peers { permanent_uuid: "47fc1a635c3a44d58fb94e27f2551847" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46477 } } I0703 01:40:50.660385 17987 raft_consensus.cc:628] T 94cc179826c046169908bf910ed7c7bb P 47fc1a635c3a44d58fb94e27f2551847 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47fc1a635c3a44d58fb94e27f2551847, State: Initialized, Role: FOLLOWER I0703 01:40:50.660967 17987 consensus_queue.cc:227] T 94cc179826c046169908bf910ed7c7bb P 47fc1a635c3a44d58fb94e27f2551847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 712, Last appended: 2.713, Last appended by leader: 713, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 669 OBSOLETE_local: false peers { permanent_uuid: "8ace61239bda46ba874d4b094fff117d" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 53360 } } peers { permanent_uuid: "e950a6c6229e4f80ba63b793d64d1998" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 32786 } } peers { permanent_uuid: "3d35a3bef6e6470d99c05c79af53ef82" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 41984 } } peers { permanent_uuid: "47fc1a635c3a44d58fb94e27f2551847" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46477 } } I0703 01:40:50.667326 17987 ts_tablet_manager.cc:1073] T 94cc179826c046169908bf910ed7c7bb P 47fc1a635c3a44d58fb94e27f2551847: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s I0703 01:40:50.668195 17083 tablet_copy_service.cc:325] P 3d35a3bef6e6470d99c05c79af53ef82: Request end of tablet copy session 47fc1a635c3a44d58fb94e27f2551847-94cc179826c046169908bf910ed7c7bb received from {username='jenkins'} at 127.5.240.195:34501 I0703 01:40:50.668306 17083 tablet_copy_service.cc:413] P 3d35a3bef6e6470d99c05c79af53ef82: ending tablet copy session 47fc1a635c3a44d58fb94e27f2551847-94cc179826c046169908bf910ed7c7bb on tablet 94cc179826c046169908bf910ed7c7bb with peer 47fc1a635c3a44d58fb94e27f2551847 I0703 01:40:50.914191 17185 raft_consensus.cc:1026] T 94cc179826c046169908bf910ed7c7bb P 47fc1a635c3a44d58fb94e27f2551847 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.668->[2.669-2.820] Dedup: 2.713->[2.714-2.820] I0703 01:40:51.490098 17062 consensus_peers.cc:455] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 -> Peer 47fc1a635c3a44d58fb94e27f2551847 (127.5.240.195:46477): Closing peer: 47fc1a635c3a44d58fb94e27f2551847 I0703 01:40:51.490182 17062 consensus_peers.cc:455] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 -> Peer e950a6c6229e4f80ba63b793d64d1998 (127.5.240.196:32786): Closing peer: e950a6c6229e4f80ba63b793d64d1998 I0703 01:40:51.490243 17062 consensus_peers.cc:455] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 -> Peer 8ace61239bda46ba874d4b094fff117d (127.5.240.193:53360): Closing peer: 8ace61239bda46ba874d4b094fff117d I0703 01:40:51.490481 17062 consensus_queue.cc:204] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 906, Committed index: 906, Last appended: 2.907, Last appended by leader: 375, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 908 OBSOLETE_local: false peers { permanent_uuid: "e950a6c6229e4f80ba63b793d64d1998" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 32786 } } peers { permanent_uuid: "3d35a3bef6e6470d99c05c79af53ef82" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 41984 } } peers { permanent_uuid: "47fc1a635c3a44d58fb94e27f2551847" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46477 } } I0703 01:40:51.493028 17307 raft_consensus.cc:1081] T 94cc179826c046169908bf910ed7c7bb P e950a6c6229e4f80ba63b793d64d1998 [term 2 FOLLOWER]: Refusing update from remote peer 3d35a3bef6e6470d99c05c79af53ef82: Log matching property violated. Preceding OpId in replica: term: 2 index: 907. Preceding OpId from leader: term: 2 index: 908. (index mismatch) I0703 01:40:51.493453 17185 raft_consensus.cc:1081] T 94cc179826c046169908bf910ed7c7bb P 47fc1a635c3a44d58fb94e27f2551847 [term 2 FOLLOWER]: Refusing update from remote peer 3d35a3bef6e6470d99c05c79af53ef82: Log matching property violated. Preceding OpId in replica: term: 2 index: 907. Preceding OpId from leader: term: 2 index: 908. (index mismatch) I0703 01:40:51.494633 18024 consensus_queue.cc:962] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e950a6c6229e4f80ba63b793d64d1998" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 32786 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 908, Last known committed idx: 906, Time since last communication: 0.000s I0703 01:40:51.495259 18027 consensus_queue.cc:962] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47fc1a635c3a44d58fb94e27f2551847" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 908, Last known committed idx: 906, Time since last communication: 0.000s I0703 01:40:51.498239 18024 raft_consensus.cc:2684] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 [term 2 LEADER]: Committing config change with OpId 2.908: config changed from index 669 to 908, VOTER 8ace61239bda46ba874d4b094fff117d (127.5.240.193) evicted. New config: { opid_index: 908 OBSOLETE_local: false peers { permanent_uuid: "e950a6c6229e4f80ba63b793d64d1998" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 32786 } } peers { permanent_uuid: "3d35a3bef6e6470d99c05c79af53ef82" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 41984 } } peers { permanent_uuid: "47fc1a635c3a44d58fb94e27f2551847" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46477 } } } I0703 01:40:51.499301 17307 raft_consensus.cc:2684] T 94cc179826c046169908bf910ed7c7bb P e950a6c6229e4f80ba63b793d64d1998 [term 2 FOLLOWER]: Committing config change with OpId 2.908: config changed from index 669 to 908, VOTER 8ace61239bda46ba874d4b094fff117d (127.5.240.193) evicted. New config: { opid_index: 908 OBSOLETE_local: false peers { permanent_uuid: "e950a6c6229e4f80ba63b793d64d1998" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 32786 } } peers { permanent_uuid: "3d35a3bef6e6470d99c05c79af53ef82" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 41984 } } peers { permanent_uuid: "47fc1a635c3a44d58fb94e27f2551847" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46477 } } } I0703 01:40:51.501953 16817 catalog_manager.cc:3798] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 reported cstate change: config changed from index 669 to 908, VOTER 8ace61239bda46ba874d4b094fff117d (127.5.240.193) evicted. New cstate: current_term: 2 leader_uuid: "3d35a3bef6e6470d99c05c79af53ef82" committed_config { opid_index: 908 OBSOLETE_local: false peers { permanent_uuid: "e950a6c6229e4f80ba63b793d64d1998" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 32786 } } peers { permanent_uuid: "3d35a3bef6e6470d99c05c79af53ef82" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 41984 } } peers { permanent_uuid: "47fc1a635c3a44d58fb94e27f2551847" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46477 } } } I0703 01:40:51.502338 17185 raft_consensus.cc:2684] T 94cc179826c046169908bf910ed7c7bb P 47fc1a635c3a44d58fb94e27f2551847 [term 2 FOLLOWER]: Committing config change with OpId 2.908: config changed from index 669 to 908, VOTER 8ace61239bda46ba874d4b094fff117d (127.5.240.193) evicted. New config: { opid_index: 908 OBSOLETE_local: false peers { permanent_uuid: "e950a6c6229e4f80ba63b793d64d1998" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 32786 } } peers { permanent_uuid: "3d35a3bef6e6470d99c05c79af53ef82" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 41984 } } peers { permanent_uuid: "47fc1a635c3a44d58fb94e27f2551847" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46477 } } } I0703 01:40:51.506404 16817 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 94cc179826c046169908bf910ed7c7bb on 8ace61239bda46ba874d4b094fff117d (127.5.240.193:53360) (TS 8ace61239bda46ba874d4b094fff117d not found in new config with opid_index 908) I0703 01:40:51.507710 16920 tablet_service.cc:800] Processing DeleteTablet for tablet 94cc179826c046169908bf910ed7c7bb with delete_type TABLET_DATA_TOMBSTONED (TS 8ace61239bda46ba874d4b094fff117d not found in new config with opid_index 908) from {username='jenkins'} at 127.5.240.193:45235 I0703 01:40:51.515944 18127 tablet_replica.cc:261] T 94cc179826c046169908bf910ed7c7bb P 8ace61239bda46ba874d4b094fff117d: stopping tablet replica I0703 01:40:51.516083 18127 maintenance_manager.cc:237] P 8ace61239bda46ba874d4b094fff117d: Unregistered op CompactRowSetsOp(94cc179826c046169908bf910ed7c7bb) I0703 01:40:51.516163 18127 maintenance_manager.cc:237] P 8ace61239bda46ba874d4b094fff117d: Unregistered op MinorDeltaCompactionOp(94cc179826c046169908bf910ed7c7bb) I0703 01:40:51.516213 18127 maintenance_manager.cc:237] P 8ace61239bda46ba874d4b094fff117d: Unregistered op MajorDeltaCompactionOp(94cc179826c046169908bf910ed7c7bb) I0703 01:40:51.516291 18127 maintenance_manager.cc:237] P 8ace61239bda46ba874d4b094fff117d: Unregistered op UndoDeltaBlockGCOp(94cc179826c046169908bf910ed7c7bb) I0703 01:40:51.516914 18127 maintenance_manager.cc:237] P 8ace61239bda46ba874d4b094fff117d: Unregistered op FlushMRSOp(94cc179826c046169908bf910ed7c7bb) I0703 01:40:51.517004 18127 maintenance_manager.cc:237] P 8ace61239bda46ba874d4b094fff117d: Unregistered op FlushDeltaMemStoresOp(94cc179826c046169908bf910ed7c7bb) I0703 01:40:51.517078 18127 maintenance_manager.cc:237] P 8ace61239bda46ba874d4b094fff117d: Unregistered op LogGCOp(94cc179826c046169908bf910ed7c7bb) I0703 01:40:51.517174 18127 raft_consensus.cc:2040] T 94cc179826c046169908bf910ed7c7bb P 8ace61239bda46ba874d4b094fff117d [term 2 FOLLOWER]: Raft consensus shutting down. I0703 01:40:51.517448 18127 raft_consensus.cc:2067] T 94cc179826c046169908bf910ed7c7bb P 8ace61239bda46ba874d4b094fff117d [term 2 FOLLOWER]: Raft consensus is shut down! I0703 01:40:51.519173 18127 ts_tablet_manager.cc:1369] T 94cc179826c046169908bf910ed7c7bb P 8ace61239bda46ba874d4b094fff117d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:40:51.522872 18127 ts_tablet_manager.cc:1382] T 94cc179826c046169908bf910ed7c7bb P 8ace61239bda46ba874d4b094fff117d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.907 I0703 01:40:51.523017 18127 log.cc:986] T 94cc179826c046169908bf910ed7c7bb P 8ace61239bda46ba874d4b094fff117d: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal/wals/94cc179826c046169908bf910ed7c7bb I0703 01:40:51.524281 16802 catalog_manager.cc:3103] TS 8ace61239bda46ba874d4b094fff117d (127.5.240.193:53360): tablet 94cc179826c046169908bf910ed7c7bb (table TestTable [id=144618e6d82940cd9252cc126e332762]) successfully deleted W0703 01:40:52.343998 18137 flags.cc:406] Enabled unsafe flag: --never_fsync=true I0703 01:40:52.602749 17062 consensus_peers.cc:455] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 -> Peer 47fc1a635c3a44d58fb94e27f2551847 (127.5.240.195:46477): Closing peer: 47fc1a635c3a44d58fb94e27f2551847 I0703 01:40:52.602859 17062 consensus_peers.cc:455] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 -> Peer e950a6c6229e4f80ba63b793d64d1998 (127.5.240.196:32786): Closing peer: e950a6c6229e4f80ba63b793d64d1998 I0703 01:40:52.603081 17062 consensus_queue.cc:204] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1118, Committed index: 1118, Last appended: 2.1118, Last appended by leader: 375, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 1119 OBSOLETE_local: false peers { permanent_uuid: "e950a6c6229e4f80ba63b793d64d1998" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 32786 } } peers { permanent_uuid: "3d35a3bef6e6470d99c05c79af53ef82" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 41984 } } peers { permanent_uuid: "47fc1a635c3a44d58fb94e27f2551847" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46477 } } peers { permanent_uuid: "14cbc64bc75444b8bbd9fbdcefdb86be" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 40856 } } I0703 01:40:52.605904 17307 raft_consensus.cc:1081] T 94cc179826c046169908bf910ed7c7bb P e950a6c6229e4f80ba63b793d64d1998 [term 2 FOLLOWER]: Refusing update from remote peer 3d35a3bef6e6470d99c05c79af53ef82: Log matching property violated. Preceding OpId in replica: term: 2 index: 1118. Preceding OpId from leader: term: 2 index: 1119. (index mismatch) I0703 01:40:52.606011 17184 raft_consensus.cc:1081] T 94cc179826c046169908bf910ed7c7bb P 47fc1a635c3a44d58fb94e27f2551847 [term 2 FOLLOWER]: Refusing update from remote peer 3d35a3bef6e6470d99c05c79af53ef82: Log matching property violated. Preceding OpId in replica: term: 2 index: 1118. Preceding OpId from leader: term: 2 index: 1119. (index mismatch) W0703 01:40:52.606307 16994 consensus_peers.cc:433] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 -> Peer 14cbc64bc75444b8bbd9fbdcefdb86be (127.5.240.197:40856): Couldn't send request to peer 14cbc64bc75444b8bbd9fbdcefdb86be for tablet 94cc179826c046169908bf910ed7c7bb. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:40:52.606668 18027 consensus_queue.cc:962] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e950a6c6229e4f80ba63b793d64d1998" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 32786 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1119, Last known committed idx: 1118, Time since last communication: 0.000s I0703 01:40:52.606886 18024 consensus_queue.cc:962] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47fc1a635c3a44d58fb94e27f2551847" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1119, Last known committed idx: 1118, Time since last communication: 0.000s I0703 01:40:52.612406 18024 raft_consensus.cc:2684] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 [term 2 LEADER]: Committing config change with OpId 2.1119: config changed from index 908 to 1119, VOTER 14cbc64bc75444b8bbd9fbdcefdb86be (127.5.240.197) added. New config: { opid_index: 1119 OBSOLETE_local: false peers { permanent_uuid: "e950a6c6229e4f80ba63b793d64d1998" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 32786 } } peers { permanent_uuid: "3d35a3bef6e6470d99c05c79af53ef82" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 41984 } } peers { permanent_uuid: "47fc1a635c3a44d58fb94e27f2551847" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46477 } } peers { permanent_uuid: "14cbc64bc75444b8bbd9fbdcefdb86be" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 40856 } } } I0703 01:40:52.614907 17307 raft_consensus.cc:2684] T 94cc179826c046169908bf910ed7c7bb P e950a6c6229e4f80ba63b793d64d1998 [term 2 FOLLOWER]: Committing config change with OpId 2.1119: config changed from index 908 to 1119, VOTER 14cbc64bc75444b8bbd9fbdcefdb86be (127.5.240.197) added. New config: { opid_index: 1119 OBSOLETE_local: false peers { permanent_uuid: "e950a6c6229e4f80ba63b793d64d1998" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 32786 } } peers { permanent_uuid: "3d35a3bef6e6470d99c05c79af53ef82" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 41984 } } peers { permanent_uuid: "47fc1a635c3a44d58fb94e27f2551847" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46477 } } peers { permanent_uuid: "14cbc64bc75444b8bbd9fbdcefdb86be" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 40856 } } } I0703 01:40:52.617944 16818 catalog_manager.cc:3798] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 reported cstate change: config changed from index 908 to 1119, VOTER 14cbc64bc75444b8bbd9fbdcefdb86be (127.5.240.197) added. New cstate: current_term: 2 leader_uuid: "3d35a3bef6e6470d99c05c79af53ef82" committed_config { opid_index: 1119 OBSOLETE_local: false peers { permanent_uuid: "e950a6c6229e4f80ba63b793d64d1998" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 32786 } } peers { permanent_uuid: "3d35a3bef6e6470d99c05c79af53ef82" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 41984 } } peers { permanent_uuid: "47fc1a635c3a44d58fb94e27f2551847" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46477 } } peers { permanent_uuid: "14cbc64bc75444b8bbd9fbdcefdb86be" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 40856 } } } I0703 01:40:52.620945 17184 raft_consensus.cc:2684] T 94cc179826c046169908bf910ed7c7bb P 47fc1a635c3a44d58fb94e27f2551847 [term 2 FOLLOWER]: Committing config change with OpId 2.1119: config changed from index 908 to 1119, VOTER 14cbc64bc75444b8bbd9fbdcefdb86be (127.5.240.197) added. New config: { opid_index: 1119 OBSOLETE_local: false peers { permanent_uuid: "e950a6c6229e4f80ba63b793d64d1998" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 32786 } } peers { permanent_uuid: "3d35a3bef6e6470d99c05c79af53ef82" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 41984 } } peers { permanent_uuid: "47fc1a635c3a44d58fb94e27f2551847" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46477 } } peers { permanent_uuid: "14cbc64bc75444b8bbd9fbdcefdb86be" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 40856 } } } I0703 01:40:53.158819 18218 ts_tablet_manager.cc:669] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be: Initiating tablet copy from peer 3d35a3bef6e6470d99c05c79af53ef82 (127.5.240.194:41984) I0703 01:40:53.159124 18218 tablet_copy_client.cc:239] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.194:41984 I0703 01:40:53.160655 17083 tablet_copy_service.cc:135] P 3d35a3bef6e6470d99c05c79af53ef82: Received BeginTabletCopySession request for tablet 94cc179826c046169908bf910ed7c7bb from peer 14cbc64bc75444b8bbd9fbdcefdb86be ({username='jenkins'} at 127.5.240.197:42387) I0703 01:40:53.160809 17083 tablet_copy_service.cc:156] P 3d35a3bef6e6470d99c05c79af53ef82: Beginning new tablet copy session on tablet 94cc179826c046169908bf910ed7c7bb from peer 14cbc64bc75444b8bbd9fbdcefdb86be at {username='jenkins'} at 127.5.240.197:42387: session id = 14cbc64bc75444b8bbd9fbdcefdb86be-94cc179826c046169908bf910ed7c7bb I0703 01:40:53.162709 17083 tablet_copy_source_session.cc:202] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:40:53.163769 18218 ts_tablet_manager.cc:1369] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be: Deleting tablet data with delete state TABLET_DATA_COPYING I0703 01:40:53.166980 18218 ts_tablet_manager.cc:1382] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.430 I0703 01:40:53.167165 18218 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 94cc179826c046169908bf910ed7c7bb. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:40:53.169303 18218 ts_tablet_manager.cc:1173] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:40:53.169581 18218 tablet_copy_client.cc:566] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be: tablet copy: Starting download of 0 data blocks... I0703 01:40:53.169827 18218 tablet_copy_client.cc:529] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be: tablet copy: Starting download of 1 WAL segments... I0703 01:40:53.188536 18218 tablet_copy_client.cc:405] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:40:53.190747 18218 ts_tablet_manager.cc:1028] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be: Bootstrapping tablet I0703 01:40:53.190913 18218 tablet_bootstrap.cc:427] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be: Bootstrap starting. W0703 01:40:53.732964 16994 consensus_peers.cc:433] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 -> Peer 14cbc64bc75444b8bbd9fbdcefdb86be (127.5.240.197:40856): Couldn't send request to peer 14cbc64bc75444b8bbd9fbdcefdb86be for tablet 94cc179826c046169908bf910ed7c7bb. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 2 times. W0703 01:40:54.206076 16994 consensus_peers.cc:433] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 -> Peer 14cbc64bc75444b8bbd9fbdcefdb86be (127.5.240.197:40856): Couldn't send request to peer 14cbc64bc75444b8bbd9fbdcefdb86be for tablet 94cc179826c046169908bf910ed7c7bb. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 3 times. I0703 01:40:54.231748 18218 tablet_bootstrap.cc:427] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be: Bootstrap replayed 1/1 log segments. Stats: ops{read=1221 overwritten=0 applied=1220 ignored=0} inserts{seen=1213 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates I0703 01:40:54.232146 18218 tablet_bootstrap.cc:427] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be: Bootstrap complete. I0703 01:40:54.232379 18218 ts_tablet_manager.cc:1045] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be: Time spent bootstrapping tablet: real 1.042s user 0.919s sys 0.044s I0703 01:40:54.233009 18218 raft_consensus.cc:325] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be [term 2 FOLLOWER]: Replica starting. Triggering 1 pending transactions. Active config: opid_index: 1119 OBSOLETE_local: false peers { permanent_uuid: "e950a6c6229e4f80ba63b793d64d1998" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 32786 } } peers { permanent_uuid: "3d35a3bef6e6470d99c05c79af53ef82" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 41984 } } peers { permanent_uuid: "47fc1a635c3a44d58fb94e27f2551847" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46477 } } peers { permanent_uuid: "14cbc64bc75444b8bbd9fbdcefdb86be" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 40856 } } I0703 01:40:54.233777 18218 raft_consensus.cc:628] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14cbc64bc75444b8bbd9fbdcefdb86be, State: Initialized, Role: FOLLOWER I0703 01:40:54.234024 18218 consensus_queue.cc:227] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1220, Last appended: 2.1221, Last appended by leader: 1221, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1119 OBSOLETE_local: false peers { permanent_uuid: "e950a6c6229e4f80ba63b793d64d1998" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 32786 } } peers { permanent_uuid: "3d35a3bef6e6470d99c05c79af53ef82" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 41984 } } peers { permanent_uuid: "47fc1a635c3a44d58fb94e27f2551847" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46477 } } peers { permanent_uuid: "14cbc64bc75444b8bbd9fbdcefdb86be" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 40856 } } I0703 01:40:54.235292 18218 ts_tablet_manager.cc:1073] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s I0703 01:40:54.236456 17083 tablet_copy_service.cc:325] P 3d35a3bef6e6470d99c05c79af53ef82: Request end of tablet copy session 14cbc64bc75444b8bbd9fbdcefdb86be-94cc179826c046169908bf910ed7c7bb received from {username='jenkins'} at 127.5.240.197:42387 I0703 01:40:54.236554 17083 tablet_copy_service.cc:413] P 3d35a3bef6e6470d99c05c79af53ef82: ending tablet copy session 14cbc64bc75444b8bbd9fbdcefdb86be-94cc179826c046169908bf910ed7c7bb on tablet 94cc179826c046169908bf910ed7c7bb with peer 14cbc64bc75444b8bbd9fbdcefdb86be I0703 01:40:54.753382 17427 raft_consensus.cc:1026] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.1118->[2.1119-2.1457] Dedup: 2.1221->[2.1222-2.1457] I0703 01:40:55.403420 17062 consensus_peers.cc:455] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 -> Peer 14cbc64bc75444b8bbd9fbdcefdb86be (127.5.240.197:40856): Closing peer: 14cbc64bc75444b8bbd9fbdcefdb86be I0703 01:40:55.403534 17062 consensus_peers.cc:455] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 -> Peer 47fc1a635c3a44d58fb94e27f2551847 (127.5.240.195:46477): Closing peer: 47fc1a635c3a44d58fb94e27f2551847 I0703 01:40:55.403627 17062 consensus_peers.cc:455] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 -> Peer e950a6c6229e4f80ba63b793d64d1998 (127.5.240.196:32786): Closing peer: e950a6c6229e4f80ba63b793d64d1998 I0703 01:40:55.403826 17062 consensus_queue.cc:204] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1554, Committed index: 1554, Last appended: 2.1555, Last appended by leader: 375, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1556 OBSOLETE_local: false peers { permanent_uuid: "3d35a3bef6e6470d99c05c79af53ef82" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 41984 } } peers { permanent_uuid: "47fc1a635c3a44d58fb94e27f2551847" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46477 } } peers { permanent_uuid: "14cbc64bc75444b8bbd9fbdcefdb86be" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 40856 } } I0703 01:40:55.405678 17427 raft_consensus.cc:1081] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be [term 2 FOLLOWER]: Refusing update from remote peer 3d35a3bef6e6470d99c05c79af53ef82: Log matching property violated. Preceding OpId in replica: term: 2 index: 1555. Preceding OpId from leader: term: 2 index: 1556. (index mismatch) I0703 01:40:55.406008 17184 raft_consensus.cc:1081] T 94cc179826c046169908bf910ed7c7bb P 47fc1a635c3a44d58fb94e27f2551847 [term 2 FOLLOWER]: Refusing update from remote peer 3d35a3bef6e6470d99c05c79af53ef82: Log matching property violated. Preceding OpId in replica: term: 2 index: 1555. Preceding OpId from leader: term: 2 index: 1556. (index mismatch) I0703 01:40:55.406502 18204 consensus_queue.cc:962] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14cbc64bc75444b8bbd9fbdcefdb86be" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 40856 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1556, Last known committed idx: 1554, Time since last communication: 0.000s I0703 01:40:55.406922 17827 consensus_queue.cc:962] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47fc1a635c3a44d58fb94e27f2551847" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1556, Last known committed idx: 1554, Time since last communication: 0.000s I0703 01:40:55.409391 17827 raft_consensus.cc:2684] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 [term 2 LEADER]: Committing config change with OpId 2.1556: config changed from index 1119 to 1556, VOTER e950a6c6229e4f80ba63b793d64d1998 (127.5.240.196) evicted. New config: { opid_index: 1556 OBSOLETE_local: false peers { permanent_uuid: "3d35a3bef6e6470d99c05c79af53ef82" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 41984 } } peers { permanent_uuid: "47fc1a635c3a44d58fb94e27f2551847" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46477 } } peers { permanent_uuid: "14cbc64bc75444b8bbd9fbdcefdb86be" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 40856 } } } I0703 01:40:55.411059 17427 raft_consensus.cc:2684] T 94cc179826c046169908bf910ed7c7bb P 14cbc64bc75444b8bbd9fbdcefdb86be [term 2 FOLLOWER]: Committing config change with OpId 2.1556: config changed from index 1119 to 1556, VOTER e950a6c6229e4f80ba63b793d64d1998 (127.5.240.196) evicted. New config: { opid_index: 1556 OBSOLETE_local: false peers { permanent_uuid: "3d35a3bef6e6470d99c05c79af53ef82" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 41984 } } peers { permanent_uuid: "47fc1a635c3a44d58fb94e27f2551847" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46477 } } peers { permanent_uuid: "14cbc64bc75444b8bbd9fbdcefdb86be" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 40856 } } } I0703 01:40:55.411075 17184 raft_consensus.cc:2684] T 94cc179826c046169908bf910ed7c7bb P 47fc1a635c3a44d58fb94e27f2551847 [term 2 FOLLOWER]: Committing config change with OpId 2.1556: config changed from index 1119 to 1556, VOTER e950a6c6229e4f80ba63b793d64d1998 (127.5.240.196) evicted. New config: { opid_index: 1556 OBSOLETE_local: false peers { permanent_uuid: "3d35a3bef6e6470d99c05c79af53ef82" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 41984 } } peers { permanent_uuid: "47fc1a635c3a44d58fb94e27f2551847" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46477 } } peers { permanent_uuid: "14cbc64bc75444b8bbd9fbdcefdb86be" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 40856 } } } I0703 01:40:55.414393 16820 catalog_manager.cc:3798] T 94cc179826c046169908bf910ed7c7bb P 3d35a3bef6e6470d99c05c79af53ef82 reported cstate change: config changed from index 1119 to 1556, VOTER e950a6c6229e4f80ba63b793d64d1998 (127.5.240.196) evicted. New cstate: current_term: 2 leader_uuid: "3d35a3bef6e6470d99c05c79af53ef82" committed_config { opid_index: 1556 OBSOLETE_local: false peers { permanent_uuid: "3d35a3bef6e6470d99c05c79af53ef82" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 41984 } } peers { permanent_uuid: "47fc1a635c3a44d58fb94e27f2551847" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46477 } } peers { permanent_uuid: "14cbc64bc75444b8bbd9fbdcefdb86be" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 40856 } } } I0703 01:40:55.421310 16820 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 94cc179826c046169908bf910ed7c7bb on e950a6c6229e4f80ba63b793d64d1998 (127.5.240.196:32786) (TS e950a6c6229e4f80ba63b793d64d1998 not found in new config with opid_index 1556) I0703 01:40:55.422045 17287 tablet_service.cc:800] Processing DeleteTablet for tablet 94cc179826c046169908bf910ed7c7bb with delete_type TABLET_DATA_TOMBSTONED (TS e950a6c6229e4f80ba63b793d64d1998 not found in new config with opid_index 1556) from {username='jenkins'} at 127.5.240.196:54959 I0703 01:40:55.422688 18386 tablet_replica.cc:261] T 94cc179826c046169908bf910ed7c7bb P e950a6c6229e4f80ba63b793d64d1998: stopping tablet replica I0703 01:40:55.422896 18386 maintenance_manager.cc:237] P e950a6c6229e4f80ba63b793d64d1998: Unregistered op CompactRowSetsOp(94cc179826c046169908bf910ed7c7bb) I0703 01:40:55.422986 18386 maintenance_manager.cc:237] P e950a6c6229e4f80ba63b793d64d1998: Unregistered op MinorDeltaCompactionOp(94cc179826c046169908bf910ed7c7bb) I0703 01:40:55.423043 18386 maintenance_manager.cc:237] P e950a6c6229e4f80ba63b793d64d1998: Unregistered op MajorDeltaCompactionOp(94cc179826c046169908bf910ed7c7bb) I0703 01:40:55.423107 18386 maintenance_manager.cc:237] P e950a6c6229e4f80ba63b793d64d1998: Unregistered op UndoDeltaBlockGCOp(94cc179826c046169908bf910ed7c7bb) I0703 01:40:55.423208 18386 maintenance_manager.cc:237] P e950a6c6229e4f80ba63b793d64d1998: Unregistered op FlushMRSOp(94cc179826c046169908bf910ed7c7bb) I0703 01:40:55.423264 18386 maintenance_manager.cc:237] P e950a6c6229e4f80ba63b793d64d1998: Unregistered op FlushDeltaMemStoresOp(94cc179826c046169908bf910ed7c7bb) I0703 01:40:55.423315 18386 maintenance_manager.cc:237] P e950a6c6229e4f80ba63b793d64d1998: Unregistered op LogGCOp(94cc179826c046169908bf910ed7c7bb) I0703 01:40:55.423389 18386 raft_consensus.cc:2040] T 94cc179826c046169908bf910ed7c7bb P e950a6c6229e4f80ba63b793d64d1998 [term 2 FOLLOWER]: Raft consensus shutting down. I0703 01:40:55.423658 18386 pending_rounds.cc:63] T 94cc179826c046169908bf910ed7c7bb P e950a6c6229e4f80ba63b793d64d1998: Trying to abort 1 pending transactions. I0703 01:40:55.423709 18386 pending_rounds.cc:68] T 94cc179826c046169908bf910ed7c7bb P e950a6c6229e4f80ba63b793d64d1998: Aborting transaction as it isn't in flight: id { term: 2 index: 1555 } timestamp: 6269367318129397760 op_type: WRITE_OP write_request { tablet_id: "94cc179826c046169908bf910ed7c7bb" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true } } row_operations { rows: "\001\007\000\201P\215A\363c\273\036\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "d21d80d99deca15a" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 6269367318099234816 } request_id { client_id: "def370e9747a4800ad71729cd18e0af9" seq_no: 1547 first_incomplete_seq_no: 1547 attempt_no: 0 } I0703 01:40:55.424293 18386 raft_consensus.cc:2067] T 94cc179826c046169908bf910ed7c7bb P e950a6c6229e4f80ba63b793d64d1998 [term 2 FOLLOWER]: Raft consensus is shut down! I0703 01:40:55.426569 18386 ts_tablet_manager.cc:1369] T 94cc179826c046169908bf910ed7c7bb P e950a6c6229e4f80ba63b793d64d1998: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:40:55.431555 18386 ts_tablet_manager.cc:1382] T 94cc179826c046169908bf910ed7c7bb P e950a6c6229e4f80ba63b793d64d1998: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.1555 I0703 01:40:55.431684 18386 log.cc:986] T 94cc179826c046169908bf910ed7c7bb P e950a6c6229e4f80ba63b793d64d1998: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal/wals/94cc179826c046169908bf910ed7c7bb I0703 01:40:55.432451 16801 catalog_manager.cc:3103] TS e950a6c6229e4f80ba63b793d64d1998 (127.5.240.196:32786): tablet 94cc179826c046169908bf910ed7c7bb (table TestTable [id=144618e6d82940cd9252cc126e332762]) successfully deleted Master Summary UUID | Address | Status ----------------------------------+---------------------+--------- 86eddb576377425fadf1b1bd8fbab03b | 127.5.240.254:36679 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status ----------------------------------+---------------------+--------- 14cbc64bc75444b8bbd9fbdcefdb86be | 127.5.240.197:40856 | HEALTHY 3d35a3bef6e6470d99c05c79af53ef82 | 127.5.240.194:41984 | HEALTHY 47fc1a635c3a44d58fb94e27f2551847 | 127.5.240.195:46477 | HEALTHY 8ace61239bda46ba874d4b094fff117d | 127.5.240.193:53360 | HEALTHY e950a6c6229e4f80ba63b793d64d1998 | 127.5.240.196:32786 | HEALTHY Flag | Value | Tags | Tablet Server ------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_server_key_size | 1024 | experimental | all 5 server(s) checked local_ip_for_outbound_sockets | 127.5.240.193 | experimental | 127.5.240.193:53360 local_ip_for_outbound_sockets | 127.5.240.194 | experimental | 127.5.240.194:41984 local_ip_for_outbound_sockets | 127.5.240.195 | experimental | 127.5.240.195:46477 local_ip_for_outbound_sockets | 127.5.240.196 | experimental | 127.5.240.196:32786 local_ip_for_outbound_sockets | 127.5.240.197 | experimental | 127.5.240.197:40856 never_fsync | true | unsafe,advanced | all 5 server(s) checked raft_prepare_replacement_before_eviction | false | experimental,advanced | all 5 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 5 server(s) checked server_dump_info_format | pb | hidden | all 5 server(s) checked server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.5.240.193:53360 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.240.194:41984 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.240.195:46477 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.5.240.196:32786 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.240.197:40856 Version Summary Version | Servers ----------------+------------------------- 1.8.0-SNAPSHOT | all 6 server(s) checked Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable -----------+----+---------+---------------+---------+------------+------------------+------------- TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0 | Total Count ----------------+------------- Masters | 1 Tablet Servers | 5 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set OK I0703 01:40:56.103919 6083 log_verifier.cc:121] Checking tablet 94cc179826c046169908bf910ed7c7bb I0703 01:40:57.079810 6083 log_verifier.cc:172] Verified matching terms for 1680 ops in tablet 94cc179826c046169908bf910ed7c7bb I0703 01:40:57.085597 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 16854 I0703 01:40:57.088347 16980 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:40:57.180385 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 16854 W0703 01:40:57.194083 17361 connection.cc:512] client connection to 127.5.240.193:53360 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:53360): Connection reset by peer (error 104) W0703 01:40:57.194113 16995 connection.cc:512] client connection to 127.5.240.193:53360 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:53360): Connection reset by peer (error 104) W0703 01:40:57.194111 16801 connection.cc:512] server connection from 127.5.240.193:43529 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:43529): Connection reset by peer (error 104) W0703 01:40:57.194290 17239 connection.cc:512] client connection to 127.5.240.193:53360 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:53360): Connection reset by peer (error 104) W0703 01:40:57.194630 17361 connection.cc:512] server connection from 127.5.240.193:44524 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:44524): Connection reset by peer (error 104) W0703 01:40:57.194922 17241 connection.cc:512] server connection from 127.5.240.193:55996 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:55996): Connection reset by peer (error 104) W0703 01:40:57.194952 16996 connection.cc:512] server connection from 127.5.240.193:37306 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:37306): Connection reset by peer (error 104) I0703 01:40:57.195029 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 16987 I0703 01:40:57.195783 17103 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:40:57.358428 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 16987 W0703 01:40:57.380089 16803 connection.cc:512] server connection from 127.5.240.194:57498 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:57498): Connection reset by peer (error 104) W0703 01:40:57.380089 17361 connection.cc:512] server connection from 127.5.240.194:32885 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:32885): Connection reset by peer (error 104) W0703 01:40:57.380133 17239 connection.cc:512] client connection to 127.5.240.194:41984 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:41984): Connection reset by peer (error 104) W0703 01:40:57.380437 17361 connection.cc:512] client connection to 127.5.240.194:41984 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:41984): Connection reset by peer (error 104) W0703 01:40:57.380828 17238 connection.cc:512] server connection from 127.5.240.194:49715 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:49715): Connection reset by peer (error 104) W0703 01:40:57.380807 17117 connection.cc:512] client connection to 127.5.240.194:41984 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:41984): Connection reset by peer (error 104) W0703 01:40:57.381305 17117 connection.cc:512] server connection from 127.5.240.194:35487 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:35487): Connection reset by peer (error 104) I0703 01:40:57.381356 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 17109 I0703 01:40:57.382058 17225 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:40:57.479112 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 17109 W0703 01:40:57.493880 16804 connection.cc:512] server connection from 127.5.240.195:54155 recv error: Network error: failed to read from TLS socket (remote: 0.0.0.0:0): Connection reset by peer (error 104) I0703 01:40:57.494107 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 17231 W0703 01:40:57.494105 16804 connection.cc:185] Error closing socket: Network error: TlsSocket::Close: Broken pipe I0703 01:40:57.494793 17347 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:40:57.615959 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 17231 W0703 01:40:57.633101 16804 connection.cc:512] server connection from 127.5.240.196:54782 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.196:54782): Connection reset by peer (error 104) W0703 01:40:57.633111 17360 connection.cc:512] client connection to 127.5.240.196:32786 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.196:32786): Connection reset by peer (error 104) I0703 01:40:57.633421 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 17353 W0703 01:40:57.633471 17360 connection.cc:512] server connection from 127.5.240.196:43650 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.196:43650): Connection reset by peer (error 104) I0703 01:40:57.634163 17469 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:40:57.748716 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 17353 W0703 01:40:57.765413 16803 connection.cc:512] server connection from 127.5.240.197:59874 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.197:59874): Connection reset by peer (error 104) I0703 01:40:57.765645 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master pid 16794 I0703 01:40:57.766353 16850 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:40:57.841316 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master with pid 16794 [ OK ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/0 (16141 ms) [ RUN ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/1 I0703 01:40:57.870645 6083 test_util.cc:208] Using random seed: 1172660513 I0703 01:40:57.872393 6083 ts_itest-base.cc:113] Starting cluster with: I0703 01:40:57.872483 6083 ts_itest-base.cc:114] -------------- I0703 01:40:57.872511 6083 ts_itest-base.cc:115] 5 tablet servers I0703 01:40:57.872539 6083 ts_itest-base.cc:116] 3 replicas per TS I0703 01:40:57.872561 6083 ts_itest-base.cc:117] -------------- I0703 01:40:57.873687 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:35377 --webserver_interface=127.5.240.254 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:40:58.053804 18436 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:40:58.054008 18436 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:40:58.056162 18436 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0703 01:40:58.056210 18436 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0703 01:40:58.056242 18436 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:40:58.056270 18436 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0703 01:40:58.056303 18436 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0703 01:40:58.063575 18436 master_main.cc:73] Master server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:35377 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.5.240.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:40:58.069039 18436 master_main.cc:80] Initializing master server... I0703 01:40:58.069708 18436 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 182708us I0703 01:40:58.070724 18436 fs_manager.cc:261] Metadata directory not provided I0703 01:40:58.070794 18436 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:40:58.071408 18436 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:40:58.071473 18436 server_base.cc:433] Attempting to create new FS layout instead I0703 01:40:58.075979 18436 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/instance: uuid: "0540d71013fc405ca76a067ab5d14607" format_stamp: "Formatted at 2018-07-03 08:40:58 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:58.076465 18436 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "0540d71013fc405ca76a067ab5d14607" format_stamp: "Formatted at 2018-07-03 08:40:58 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:58.084987 18436 fs_manager.cc:501] Time spent creating directory manager: real 0.008s user 0.002s sys 0.005s I0703 01:40:58.086181 18436 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:40:58.086544 18436 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:40:58.089702 18436 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0703 01:40:58.089747 18436 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal uuid: "0540d71013fc405ca76a067ab5d14607" format_stamp: "Formatted at 2018-07-03 08:40:58 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:58.089848 18436 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:40:58.141661 18436 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:40:58.143103 18436 master_main.cc:83] Starting Master server... I0703 01:40:58.180199 18436 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.254:35377 I0703 01:40:58.180687 18436 webserver.cc:175] Starting webserver on 127.5.240.254:0 I0703 01:40:58.180728 18436 webserver.cc:186] Document root disabled I0703 01:40:58.181915 18436 webserver.cc:313] Webserver started. Bound to: http://127.5.240.254:57479/ I0703 01:40:58.182484 18436 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb I0703 01:40:58.185672 18495 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:40:58.186158 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master as pid 18436 I0703 01:40:58.186795 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.5.240.193:0 --local_ip_for_outbound_sockets=127.5.240.193 --webserver_interface=127.5.240.193 --webserver_port=0 --tserver_master_addrs=127.5.240.254:35377 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0703 01:40:58.197005 18495 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 0540d71013fc405ca76a067ab5d14607: Bootstrap starting. I0703 01:40:58.199244 18495 tablet_bootstrap.cc:578] T 00000000000000000000000000000000 P 0540d71013fc405ca76a067ab5d14607: No blocks or log segments found. Creating new log. I0703 01:40:58.199973 18495 log.cc:525] T 00000000000000000000000000000000 P 0540d71013fc405ca76a067ab5d14607: Log is configured to *not* fsync() on all Append() calls I0703 01:40:58.202208 18495 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 0540d71013fc405ca76a067ab5d14607: No bootstrap required, opened a new log I0703 01:40:58.208439 18495 raft_consensus.cc:325] T 00000000000000000000000000000000 P 0540d71013fc405ca76a067ab5d14607 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0540d71013fc405ca76a067ab5d14607" member_type: VOTER } I0703 01:40:58.208638 18495 raft_consensus.cc:351] T 00000000000000000000000000000000 P 0540d71013fc405ca76a067ab5d14607 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:40:58.208691 18495 raft_consensus.cc:628] T 00000000000000000000000000000000 P 0540d71013fc405ca76a067ab5d14607 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0540d71013fc405ca76a067ab5d14607, State: Initialized, Role: FOLLOWER I0703 01:40:58.209156 18495 consensus_queue.cc:227] T 00000000000000000000000000000000 P 0540d71013fc405ca76a067ab5d14607 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0540d71013fc405ca76a067ab5d14607" member_type: VOTER } I0703 01:40:58.209290 18495 raft_consensus.cc:365] T 00000000000000000000000000000000 P 0540d71013fc405ca76a067ab5d14607 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:40:58.209349 18495 raft_consensus.cc:457] T 00000000000000000000000000000000 P 0540d71013fc405ca76a067ab5d14607 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:40:58.209415 18495 raft_consensus.cc:2730] T 00000000000000000000000000000000 P 0540d71013fc405ca76a067ab5d14607: Snoozing failure detection for 1.968s (starting election) I0703 01:40:58.209489 18495 raft_consensus.cc:2782] T 00000000000000000000000000000000 P 0540d71013fc405ca76a067ab5d14607 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:40:58.210830 18495 raft_consensus.cc:479] T 00000000000000000000000000000000 P 0540d71013fc405ca76a067ab5d14607 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0540d71013fc405ca76a067ab5d14607" member_type: VOTER } I0703 01:40:58.211176 18495 leader_election.cc:270] T 00000000000000000000000000000000 P 0540d71013fc405ca76a067ab5d14607 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:40:58.212633 18498 raft_consensus.cc:2730] T 00000000000000000000000000000000 P 0540d71013fc405ca76a067ab5d14607: Snoozing failure detection for 1.615s (election complete) I0703 01:40:58.213193 18498 raft_consensus.cc:2537] T 00000000000000000000000000000000 P 0540d71013fc405ca76a067ab5d14607 [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:40:58.218833 18498 raft_consensus.cc:593] T 00000000000000000000000000000000 P 0540d71013fc405ca76a067ab5d14607 [term 1 LEADER]: Becoming Leader. State: Replica: 0540d71013fc405ca76a067ab5d14607, State: Running, Role: LEADER I0703 01:40:58.219107 18498 consensus_queue.cc:204] T 00000000000000000000000000000000 P 0540d71013fc405ca76a067ab5d14607 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0540d71013fc405ca76a067ab5d14607" member_type: VOTER } I0703 01:40:58.220842 18495 sys_catalog.cc:337] T 00000000000000000000000000000000 P 0540d71013fc405ca76a067ab5d14607 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "0540d71013fc405ca76a067ab5d14607" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0540d71013fc405ca76a067ab5d14607" member_type: VOTER } } I0703 01:40:58.221101 18495 sys_catalog.cc:340] T 00000000000000000000000000000000 P 0540d71013fc405ca76a067ab5d14607 [sys.catalog]: This master's current role is: LEADER I0703 01:40:58.221280 18499 sys_catalog.cc:337] T 00000000000000000000000000000000 P 0540d71013fc405ca76a067ab5d14607 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0540d71013fc405ca76a067ab5d14607" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0540d71013fc405ca76a067ab5d14607" member_type: VOTER } } I0703 01:40:58.223446 18499 sys_catalog.cc:340] T 00000000000000000000000000000000 P 0540d71013fc405ca76a067ab5d14607 [sys.catalog]: This master's current role is: LEADER I0703 01:40:58.224514 18495 sys_catalog.cc:424] T 00000000000000000000000000000000 P 0540d71013fc405ca76a067ab5d14607 [sys.catalog]: configured and running, proceeding with master startup. I0703 01:40:58.223062 18500 sys_catalog.cc:337] T 00000000000000000000000000000000 P 0540d71013fc405ca76a067ab5d14607 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0540d71013fc405ca76a067ab5d14607. Latest consensus state: current_term: 1 leader_uuid: "0540d71013fc405ca76a067ab5d14607" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0540d71013fc405ca76a067ab5d14607" member_type: VOTER } } I0703 01:40:58.225642 18500 sys_catalog.cc:340] T 00000000000000000000000000000000 P 0540d71013fc405ca76a067ab5d14607 [sys.catalog]: This master's current role is: LEADER I0703 01:40:58.226368 18502 catalog_manager.cc:1006] Loading table and tablet metadata into memory... I0703 01:40:58.227856 18436 master_main.cc:86] Master server successfully started. I0703 01:40:58.233533 18502 catalog_manager.cc:1017] Initializing Kudu internal certificate authority... I0703 01:40:58.288143 18502 catalog_manager.cc:904] Generated new certificate authority record I0703 01:40:58.290010 18502 catalog_manager.cc:1026] Loading token signing keys... I0703 01:40:58.300487 18502 catalog_manager.cc:4102] T 00000000000000000000000000000000 P 0540d71013fc405ca76a067ab5d14607: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:40:58.362447 18496 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:40:58.362704 18496 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:40:58.364701 18496 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0703 01:40:58.364764 18496 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:40:58.364867 18496 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.193 I0703 01:40:58.371364 18496 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.5.240.193 --webserver_port=0 --tserver_master_addrs=127.5.240.254:35377 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.193 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:40:58.377969 18496 tablet_server_main.cc:85] Initializing tablet server... I0703 01:40:58.379478 18496 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 182708us I0703 01:40:58.380053 18496 fs_manager.cc:261] Metadata directory not provided I0703 01:40:58.380122 18496 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:40:58.380756 18496 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:40:58.380826 18496 server_base.cc:433] Attempting to create new FS layout instead I0703 01:40:58.384959 18496 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "899db87084ef4ab39c468996b1edecc7" format_stamp: "Formatted at 2018-07-03 08:40:58 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:58.385430 18496 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "899db87084ef4ab39c468996b1edecc7" format_stamp: "Formatted at 2018-07-03 08:40:58 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:58.393704 18496 fs_manager.cc:501] Time spent creating directory manager: real 0.008s user 0.004s sys 0.003s I0703 01:40:58.394901 18496 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:40:58.395290 18496 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:40:58.398885 18496 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0703 01:40:58.398938 18496 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal uuid: "899db87084ef4ab39c468996b1edecc7" format_stamp: "Formatted at 2018-07-03 08:40:58 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:58.399068 18496 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:40:58.422843 18496 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:40:58.425323 18496 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:40:58.425740 18496 tablet_server_main.cc:90] Starting tablet server... I0703 01:40:58.526511 18496 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.193:56129 I0703 01:40:58.526989 18496 webserver.cc:175] Starting webserver on 127.5.240.193:0 I0703 01:40:58.527036 18496 webserver.cc:186] Document root disabled I0703 01:40:58.527835 18496 webserver.cc:313] Webserver started. Bound to: http://127.5.240.193:42974/ I0703 01:40:58.528367 18496 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb I0703 01:40:58.530424 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 18496 I0703 01:40:58.531301 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.5.240.194:0 --local_ip_for_outbound_sockets=127.5.240.194 --webserver_interface=127.5.240.194 --webserver_port=0 --tserver_master_addrs=127.5.240.254:35377 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0703 01:40:58.539878 18496 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:40:58.563267 18625 heartbeater.cc:346] Connected to a master server at 127.5.240.254:35377 I0703 01:40:58.563423 18625 heartbeater.cc:426] Registering TS with master... I0703 01:40:58.564182 18625 heartbeater.cc:475] Master 127.5.240.254:35377 requested a full tablet report, sending... I0703 01:40:58.566157 18462 ts_manager.cc:98] Registered new tserver with Master: 899db87084ef4ab39c468996b1edecc7 (127.5.240.193:56129) I0703 01:40:58.568727 18462 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.193:43510 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:40:58.709578 18627 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:40:58.709786 18627 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:40:58.711787 18627 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0703 01:40:58.711850 18627 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:40:58.711956 18627 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.194 I0703 01:40:58.719391 18627 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.194:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.5.240.194 --webserver_port=0 --tserver_master_addrs=127.5.240.254:35377 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.194 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:40:58.725975 18627 tablet_server_main.cc:85] Initializing tablet server... I0703 01:40:58.727464 18627 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 182708us I0703 01:40:58.728072 18627 fs_manager.cc:261] Metadata directory not provided I0703 01:40:58.728140 18627 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:40:58.728366 18627 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:40:58.728432 18627 server_base.cc:433] Attempting to create new FS layout instead I0703 01:40:58.732481 18627 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "3fc974f1f3ce4ae59d8220a6e3369113" format_stamp: "Formatted at 2018-07-03 08:40:58 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:58.733016 18627 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "3fc974f1f3ce4ae59d8220a6e3369113" format_stamp: "Formatted at 2018-07-03 08:40:58 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:58.740860 18627 fs_manager.cc:501] Time spent creating directory manager: real 0.008s user 0.003s sys 0.003s I0703 01:40:58.742040 18627 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:40:58.742408 18627 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:40:58.746229 18627 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s I0703 01:40:58.746279 18627 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal uuid: "3fc974f1f3ce4ae59d8220a6e3369113" format_stamp: "Formatted at 2018-07-03 08:40:58 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:58.746404 18627 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:40:58.811684 18627 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:40:58.813937 18627 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:40:58.814257 18627 tablet_server_main.cc:90] Starting tablet server... I0703 01:40:58.897838 18627 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.194:59009 I0703 01:40:58.898259 18627 webserver.cc:175] Starting webserver on 127.5.240.194:0 I0703 01:40:58.898303 18627 webserver.cc:186] Document root disabled I0703 01:40:58.899650 18627 webserver.cc:313] Webserver started. Bound to: http://127.5.240.194:40158/ I0703 01:40:58.900161 18627 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb I0703 01:40:58.900854 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 18627 I0703 01:40:58.901764 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.5.240.195:0 --local_ip_for_outbound_sockets=127.5.240.195 --webserver_interface=127.5.240.195 --webserver_port=0 --tserver_master_addrs=127.5.240.254:35377 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0703 01:40:58.901952 18627 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:40:58.914268 18748 heartbeater.cc:346] Connected to a master server at 127.5.240.254:35377 I0703 01:40:58.914402 18748 heartbeater.cc:426] Registering TS with master... I0703 01:40:58.915125 18748 heartbeater.cc:475] Master 127.5.240.254:35377 requested a full tablet report, sending... I0703 01:40:58.916059 18462 ts_manager.cc:98] Registered new tserver with Master: 3fc974f1f3ce4ae59d8220a6e3369113 (127.5.240.194:59009) I0703 01:40:58.916905 18462 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.194:55511 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:40:59.059262 18751 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:40:59.059486 18751 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:40:59.061477 18751 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0703 01:40:59.061525 18751 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:40:59.061653 18751 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.195 I0703 01:40:59.067792 18751 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.195:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.5.240.195 --webserver_port=0 --tserver_master_addrs=127.5.240.254:35377 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.195 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:40:59.074329 18751 tablet_server_main.cc:85] Initializing tablet server... I0703 01:40:59.075817 18751 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 183208us I0703 01:40:59.076345 18751 fs_manager.cc:261] Metadata directory not provided I0703 01:40:59.076409 18751 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:40:59.077039 18751 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:40:59.077102 18751 server_base.cc:433] Attempting to create new FS layout instead I0703 01:40:59.081121 18751 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "152925ceae284ca49f5e73752dc503c6" format_stamp: "Formatted at 2018-07-03 08:40:59 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:59.081670 18751 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "152925ceae284ca49f5e73752dc503c6" format_stamp: "Formatted at 2018-07-03 08:40:59 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:59.089373 18751 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.003s sys 0.003s I0703 01:40:59.090559 18751 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:40:59.090953 18751 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:40:59.094079 18751 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0703 01:40:59.094125 18751 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal uuid: "152925ceae284ca49f5e73752dc503c6" format_stamp: "Formatted at 2018-07-03 08:40:59 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:59.094246 18751 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:40:59.130717 18751 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:40:59.132925 18751 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:40:59.133221 18751 tablet_server_main.cc:90] Starting tablet server... I0703 01:40:59.218092 18751 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.195:52628 I0703 01:40:59.218463 18751 webserver.cc:175] Starting webserver on 127.5.240.195:0 I0703 01:40:59.218500 18751 webserver.cc:186] Document root disabled I0703 01:40:59.219673 18751 webserver.cc:313] Webserver started. Bound to: http://127.5.240.195:49681/ I0703 01:40:59.220196 18751 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb I0703 01:40:59.222451 18751 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:40:59.226020 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 18751 I0703 01:40:59.226923 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.5.240.196:0 --local_ip_for_outbound_sockets=127.5.240.196 --webserver_interface=127.5.240.196 --webserver_port=0 --tserver_master_addrs=127.5.240.254:35377 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0703 01:40:59.235214 18870 heartbeater.cc:346] Connected to a master server at 127.5.240.254:35377 I0703 01:40:59.235368 18870 heartbeater.cc:426] Registering TS with master... I0703 01:40:59.236127 18870 heartbeater.cc:475] Master 127.5.240.254:35377 requested a full tablet report, sending... I0703 01:40:59.237113 18462 ts_manager.cc:98] Registered new tserver with Master: 152925ceae284ca49f5e73752dc503c6 (127.5.240.195:52628) I0703 01:40:59.237996 18462 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.195:50969 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:40:59.386524 18873 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:40:59.386746 18873 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:40:59.388691 18873 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0703 01:40:59.388738 18873 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:40:59.388821 18873 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.196 I0703 01:40:59.395076 18873 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.196:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb --webserver_interface=127.5.240.196 --webserver_port=0 --tserver_master_addrs=127.5.240.254:35377 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.196 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:40:59.401682 18873 tablet_server_main.cc:85] Initializing tablet server... I0703 01:40:59.403193 18873 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 183208us I0703 01:40:59.403759 18873 fs_manager.cc:261] Metadata directory not provided I0703 01:40:59.403817 18873 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:40:59.404409 18873 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:40:59.404470 18873 server_base.cc:433] Attempting to create new FS layout instead I0703 01:40:59.408515 18873 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/instance: uuid: "0ed93606f04d4c3eb4d2c4a497ba385a" format_stamp: "Formatted at 2018-07-03 08:40:59 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:59.409024 18873 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal/instance: uuid: "0ed93606f04d4c3eb4d2c4a497ba385a" format_stamp: "Formatted at 2018-07-03 08:40:59 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:59.416766 18873 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s I0703 01:40:59.417934 18873 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:40:59.418292 18873 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:40:59.421419 18873 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0703 01:40:59.421464 18873 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal uuid: "0ed93606f04d4c3eb4d2c4a497ba385a" format_stamp: "Formatted at 2018-07-03 08:40:59 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:59.421595 18873 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:40:59.453599 18873 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:40:59.455869 18873 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:40:59.456176 18873 tablet_server_main.cc:90] Starting tablet server... I0703 01:40:59.541931 18873 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.196:45792 I0703 01:40:59.542313 18873 webserver.cc:175] Starting webserver on 127.5.240.196:0 I0703 01:40:59.542351 18873 webserver.cc:186] Document root disabled I0703 01:40:59.543052 18873 webserver.cc:313] Webserver started. Bound to: http://127.5.240.196:38501/ I0703 01:40:59.543581 18873 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb I0703 01:40:59.547648 18873 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:40:59.551812 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 18873 I0703 01:40:59.552415 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data --block_manager=log --rpc_bind_addresses=127.5.240.197:0 --local_ip_for_outbound_sockets=127.5.240.197 --webserver_interface=127.5.240.197 --webserver_port=0 --tserver_master_addrs=127.5.240.254:35377 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0703 01:40:59.557456 18992 heartbeater.cc:346] Connected to a master server at 127.5.240.254:35377 I0703 01:40:59.557620 18992 heartbeater.cc:426] Registering TS with master... I0703 01:40:59.558334 18992 heartbeater.cc:475] Master 127.5.240.254:35377 requested a full tablet report, sending... I0703 01:40:59.559291 18462 ts_manager.cc:98] Registered new tserver with Master: 0ed93606f04d4c3eb4d2c4a497ba385a (127.5.240.196:45792) I0703 01:40:59.560123 18462 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.196:58932 I0703 01:40:59.570914 18625 heartbeater.cc:467] Master 127.5.240.254:35377 was elected leader, sending a full tablet report... WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:40:59.711984 18995 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:40:59.712196 18995 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:40:59.714232 18995 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0703 01:40:59.714280 18995 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:40:59.714365 18995 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.197 I0703 01:40:59.720955 18995 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.197:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb --webserver_interface=127.5.240.197 --webserver_port=0 --tserver_master_addrs=127.5.240.254:35377 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.197 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:40:59.727515 18995 tablet_server_main.cc:85] Initializing tablet server... I0703 01:40:59.729048 18995 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 183208us I0703 01:40:59.729604 18995 fs_manager.cc:261] Metadata directory not provided I0703 01:40:59.729681 18995 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:40:59.729905 18995 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:40:59.729967 18995 server_base.cc:433] Attempting to create new FS layout instead I0703 01:40:59.734057 18995 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/instance: uuid: "8fe9841c2de149ad924d05540b4bc63f" format_stamp: "Formatted at 2018-07-03 08:40:59 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:59.734555 18995 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal/instance: uuid: "8fe9841c2de149ad924d05540b4bc63f" format_stamp: "Formatted at 2018-07-03 08:40:59 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:59.741792 18995 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.003s sys 0.003s I0703 01:40:59.743114 18995 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:40:59.743465 18995 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:40:59.746560 18995 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0703 01:40:59.746608 18995 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal uuid: "8fe9841c2de149ad924d05540b4bc63f" format_stamp: "Formatted at 2018-07-03 08:40:59 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:40:59.746742 18995 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:40:59.782106 18995 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:40:59.783900 18995 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:40:59.784179 18995 tablet_server_main.cc:90] Starting tablet server... I0703 01:40:59.869827 18995 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.197:59456 I0703 01:40:59.870206 18995 webserver.cc:175] Starting webserver on 127.5.240.197:0 I0703 01:40:59.870245 18995 webserver.cc:186] Document root disabled I0703 01:40:59.871435 18995 webserver.cc:313] Webserver started. Bound to: http://127.5.240.197:42009/ I0703 01:40:59.871989 18995 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb I0703 01:40:59.873860 18995 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:40:59.876327 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 18995 I0703 01:40:59.888336 19114 heartbeater.cc:346] Connected to a master server at 127.5.240.254:35377 I0703 01:40:59.888491 19114 heartbeater.cc:426] Registering TS with master... I0703 01:40:59.889252 19114 heartbeater.cc:475] Master 127.5.240.254:35377 requested a full tablet report, sending... I0703 01:40:59.890265 18462 ts_manager.cc:98] Registered new tserver with Master: 8fe9841c2de149ad924d05540b4bc63f (127.5.240.197:59456) I0703 01:40:59.891125 18462 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.197:52271 I0703 01:40:59.891932 6083 external_mini_cluster.cc:490] 5 TS(s) registered with all masters I0703 01:40:59.902290 18462 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.5.240.254:46696: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0703 01:40:59.918696 18748 heartbeater.cc:467] Master 127.5.240.254:35377 was elected leader, sending a full tablet report... I0703 01:40:59.927491 18807 tablet_service.cc:759] Processing CreateTablet for tablet 38f37abe75ac4a948158a3caa9fe4176 (table=TestTable [id=6215b2d4df2d463e9fe59c865c854005]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:40:59.928443 18807 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 38f37abe75ac4a948158a3caa9fe4176. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:40:59.933698 18685 tablet_service.cc:759] Processing CreateTablet for tablet 38f37abe75ac4a948158a3caa9fe4176 (table=TestTable [id=6215b2d4df2d463e9fe59c865c854005]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:40:59.934535 18685 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 38f37abe75ac4a948158a3caa9fe4176. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:40:59.935855 18929 tablet_service.cc:759] Processing CreateTablet for tablet 38f37abe75ac4a948158a3caa9fe4176 (table=TestTable [id=6215b2d4df2d463e9fe59c865c854005]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:40:59.936414 18929 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 38f37abe75ac4a948158a3caa9fe4176. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:40:59.944018 18807 ts_tablet_manager.cc:1173] T 38f37abe75ac4a948158a3caa9fe4176 P 152925ceae284ca49f5e73752dc503c6: Registered tablet (data state: TABLET_DATA_READY) I0703 01:40:59.944902 19134 ts_tablet_manager.cc:1028] T 38f37abe75ac4a948158a3caa9fe4176 P 152925ceae284ca49f5e73752dc503c6: Bootstrapping tablet I0703 01:40:59.945498 19134 tablet_bootstrap.cc:427] T 38f37abe75ac4a948158a3caa9fe4176 P 152925ceae284ca49f5e73752dc503c6: Bootstrap starting. I0703 01:40:59.947346 18685 ts_tablet_manager.cc:1173] T 38f37abe75ac4a948158a3caa9fe4176 P 3fc974f1f3ce4ae59d8220a6e3369113: Registered tablet (data state: TABLET_DATA_READY) I0703 01:40:59.948001 19135 ts_tablet_manager.cc:1028] T 38f37abe75ac4a948158a3caa9fe4176 P 3fc974f1f3ce4ae59d8220a6e3369113: Bootstrapping tablet I0703 01:40:59.948333 19135 tablet_bootstrap.cc:427] T 38f37abe75ac4a948158a3caa9fe4176 P 3fc974f1f3ce4ae59d8220a6e3369113: Bootstrap starting. I0703 01:40:59.948984 18929 ts_tablet_manager.cc:1173] T 38f37abe75ac4a948158a3caa9fe4176 P 0ed93606f04d4c3eb4d2c4a497ba385a: Registered tablet (data state: TABLET_DATA_READY) I0703 01:40:59.949399 19134 tablet_bootstrap.cc:578] T 38f37abe75ac4a948158a3caa9fe4176 P 152925ceae284ca49f5e73752dc503c6: No blocks or log segments found. Creating new log. I0703 01:40:59.949816 19136 ts_tablet_manager.cc:1028] T 38f37abe75ac4a948158a3caa9fe4176 P 0ed93606f04d4c3eb4d2c4a497ba385a: Bootstrapping tablet I0703 01:40:59.950219 19136 tablet_bootstrap.cc:427] T 38f37abe75ac4a948158a3caa9fe4176 P 0ed93606f04d4c3eb4d2c4a497ba385a: Bootstrap starting. I0703 01:40:59.950592 19134 log.cc:525] T 38f37abe75ac4a948158a3caa9fe4176 P 152925ceae284ca49f5e73752dc503c6: Log is configured to *not* fsync() on all Append() calls I0703 01:40:59.950891 19135 tablet_bootstrap.cc:578] T 38f37abe75ac4a948158a3caa9fe4176 P 3fc974f1f3ce4ae59d8220a6e3369113: No blocks or log segments found. Creating new log. I0703 01:40:59.951925 19135 log.cc:525] T 38f37abe75ac4a948158a3caa9fe4176 P 3fc974f1f3ce4ae59d8220a6e3369113: Log is configured to *not* fsync() on all Append() calls I0703 01:40:59.952770 19134 tablet_bootstrap.cc:427] T 38f37abe75ac4a948158a3caa9fe4176 P 152925ceae284ca49f5e73752dc503c6: No bootstrap required, opened a new log I0703 01:40:59.952941 19134 ts_tablet_manager.cc:1045] T 38f37abe75ac4a948158a3caa9fe4176 P 152925ceae284ca49f5e73752dc503c6: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s I0703 01:40:59.953095 19136 tablet_bootstrap.cc:578] T 38f37abe75ac4a948158a3caa9fe4176 P 0ed93606f04d4c3eb4d2c4a497ba385a: No blocks or log segments found. Creating new log. I0703 01:40:59.954031 19136 log.cc:525] T 38f37abe75ac4a948158a3caa9fe4176 P 0ed93606f04d4c3eb4d2c4a497ba385a: Log is configured to *not* fsync() on all Append() calls I0703 01:40:59.956547 19136 tablet_bootstrap.cc:427] T 38f37abe75ac4a948158a3caa9fe4176 P 0ed93606f04d4c3eb4d2c4a497ba385a: No bootstrap required, opened a new log I0703 01:40:59.956749 19136 ts_tablet_manager.cc:1045] T 38f37abe75ac4a948158a3caa9fe4176 P 0ed93606f04d4c3eb4d2c4a497ba385a: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s I0703 01:40:59.960019 19134 raft_consensus.cc:325] T 38f37abe75ac4a948158a3caa9fe4176 P 152925ceae284ca49f5e73752dc503c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152925ceae284ca49f5e73752dc503c6" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 52628 } } peers { permanent_uuid: "3fc974f1f3ce4ae59d8220a6e3369113" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59009 } } peers { permanent_uuid: "0ed93606f04d4c3eb4d2c4a497ba385a" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 45792 } } I0703 01:40:59.960314 19134 raft_consensus.cc:351] T 38f37abe75ac4a948158a3caa9fe4176 P 152925ceae284ca49f5e73752dc503c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:40:59.960398 19134 raft_consensus.cc:628] T 38f37abe75ac4a948158a3caa9fe4176 P 152925ceae284ca49f5e73752dc503c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 152925ceae284ca49f5e73752dc503c6, State: Initialized, Role: FOLLOWER I0703 01:40:59.960809 19134 consensus_queue.cc:227] T 38f37abe75ac4a948158a3caa9fe4176 P 152925ceae284ca49f5e73752dc503c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152925ceae284ca49f5e73752dc503c6" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 52628 } } peers { permanent_uuid: "3fc974f1f3ce4ae59d8220a6e3369113" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59009 } } peers { permanent_uuid: "0ed93606f04d4c3eb4d2c4a497ba385a" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 45792 } } I0703 01:40:59.962179 18870 heartbeater.cc:467] Master 127.5.240.254:35377 was elected leader, sending a full tablet report... I0703 01:40:59.963157 19134 ts_tablet_manager.cc:1073] T 38f37abe75ac4a948158a3caa9fe4176 P 152925ceae284ca49f5e73752dc503c6: Time spent starting tablet: real 0.010s user 0.007s sys 0.002s I0703 01:40:59.965201 19136 raft_consensus.cc:325] T 38f37abe75ac4a948158a3caa9fe4176 P 0ed93606f04d4c3eb4d2c4a497ba385a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152925ceae284ca49f5e73752dc503c6" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 52628 } } peers { permanent_uuid: "3fc974f1f3ce4ae59d8220a6e3369113" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59009 } } peers { permanent_uuid: "0ed93606f04d4c3eb4d2c4a497ba385a" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 45792 } } I0703 01:40:59.965466 19136 raft_consensus.cc:351] T 38f37abe75ac4a948158a3caa9fe4176 P 0ed93606f04d4c3eb4d2c4a497ba385a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:40:59.965557 19136 raft_consensus.cc:628] T 38f37abe75ac4a948158a3caa9fe4176 P 0ed93606f04d4c3eb4d2c4a497ba385a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed93606f04d4c3eb4d2c4a497ba385a, State: Initialized, Role: FOLLOWER I0703 01:40:59.965936 19136 consensus_queue.cc:227] T 38f37abe75ac4a948158a3caa9fe4176 P 0ed93606f04d4c3eb4d2c4a497ba385a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152925ceae284ca49f5e73752dc503c6" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 52628 } } peers { permanent_uuid: "3fc974f1f3ce4ae59d8220a6e3369113" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59009 } } peers { permanent_uuid: "0ed93606f04d4c3eb4d2c4a497ba385a" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 45792 } } I0703 01:40:59.966599 19135 tablet_bootstrap.cc:427] T 38f37abe75ac4a948158a3caa9fe4176 P 3fc974f1f3ce4ae59d8220a6e3369113: No bootstrap required, opened a new log I0703 01:40:59.966830 19135 ts_tablet_manager.cc:1045] T 38f37abe75ac4a948158a3caa9fe4176 P 3fc974f1f3ce4ae59d8220a6e3369113: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.012s I0703 01:40:59.966876 18992 heartbeater.cc:467] Master 127.5.240.254:35377 was elected leader, sending a full tablet report... I0703 01:40:59.967473 19136 ts_tablet_manager.cc:1073] T 38f37abe75ac4a948158a3caa9fe4176 P 0ed93606f04d4c3eb4d2c4a497ba385a: Time spent starting tablet: real 0.011s user 0.008s sys 0.001s W0703 01:40:59.973044 18871 tablet.cc:1678] T 38f37abe75ac4a948158a3caa9fe4176 P 152925ceae284ca49f5e73752dc503c6: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:40:59.973152 19135 raft_consensus.cc:325] T 38f37abe75ac4a948158a3caa9fe4176 P 3fc974f1f3ce4ae59d8220a6e3369113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152925ceae284ca49f5e73752dc503c6" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 52628 } } peers { permanent_uuid: "3fc974f1f3ce4ae59d8220a6e3369113" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59009 } } peers { permanent_uuid: "0ed93606f04d4c3eb4d2c4a497ba385a" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 45792 } } I0703 01:40:59.973569 19135 raft_consensus.cc:351] T 38f37abe75ac4a948158a3caa9fe4176 P 3fc974f1f3ce4ae59d8220a6e3369113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:40:59.973634 19135 raft_consensus.cc:628] T 38f37abe75ac4a948158a3caa9fe4176 P 3fc974f1f3ce4ae59d8220a6e3369113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc974f1f3ce4ae59d8220a6e3369113, State: Initialized, Role: FOLLOWER I0703 01:40:59.973963 19135 consensus_queue.cc:227] T 38f37abe75ac4a948158a3caa9fe4176 P 3fc974f1f3ce4ae59d8220a6e3369113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152925ceae284ca49f5e73752dc503c6" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 52628 } } peers { permanent_uuid: "3fc974f1f3ce4ae59d8220a6e3369113" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59009 } } peers { permanent_uuid: "0ed93606f04d4c3eb4d2c4a497ba385a" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 45792 } } I0703 01:40:59.975143 19135 ts_tablet_manager.cc:1073] T 38f37abe75ac4a948158a3caa9fe4176 P 3fc974f1f3ce4ae59d8220a6e3369113: Time spent starting tablet: real 0.008s user 0.005s sys 0.002s W0703 01:41:00.046260 18993 tablet.cc:1678] T 38f37abe75ac4a948158a3caa9fe4176 P 0ed93606f04d4c3eb4d2c4a497ba385a: Can't schedule compaction. Clean time has not been advanced past its initial value. W0703 01:41:00.152783 18749 tablet.cc:1678] T 38f37abe75ac4a948158a3caa9fe4176 P 3fc974f1f3ce4ae59d8220a6e3369113: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:41:00.169754 19140 raft_consensus.cc:457] T 38f37abe75ac4a948158a3caa9fe4176 P 152925ceae284ca49f5e73752dc503c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0703 01:41:00.169853 19140 raft_consensus.cc:2730] T 38f37abe75ac4a948158a3caa9fe4176 P 152925ceae284ca49f5e73752dc503c6: Snoozing failure detection for 1.585s (starting election) I0703 01:41:00.169955 19140 raft_consensus.cc:479] T 38f37abe75ac4a948158a3caa9fe4176 P 152925ceae284ca49f5e73752dc503c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152925ceae284ca49f5e73752dc503c6" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 52628 } } peers { permanent_uuid: "3fc974f1f3ce4ae59d8220a6e3369113" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59009 } } peers { permanent_uuid: "0ed93606f04d4c3eb4d2c4a497ba385a" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 45792 } } I0703 01:41:00.170713 19140 leader_election.cc:243] T 38f37abe75ac4a948158a3caa9fe4176 P 152925ceae284ca49f5e73752dc503c6 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 3fc974f1f3ce4ae59d8220a6e3369113 (127.5.240.194:59009) I0703 01:41:00.171058 19140 leader_election.cc:243] T 38f37abe75ac4a948158a3caa9fe4176 P 152925ceae284ca49f5e73752dc503c6 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 0ed93606f04d4c3eb4d2c4a497ba385a (127.5.240.196:45792) I0703 01:41:00.177206 19142 raft_consensus.cc:457] T 38f37abe75ac4a948158a3caa9fe4176 P 3fc974f1f3ce4ae59d8220a6e3369113 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0703 01:41:00.177762 19142 raft_consensus.cc:2730] T 38f37abe75ac4a948158a3caa9fe4176 P 3fc974f1f3ce4ae59d8220a6e3369113: Snoozing failure detection for 2.114s (starting election) I0703 01:41:00.178946 19142 raft_consensus.cc:479] T 38f37abe75ac4a948158a3caa9fe4176 P 3fc974f1f3ce4ae59d8220a6e3369113 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152925ceae284ca49f5e73752dc503c6" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 52628 } } peers { permanent_uuid: "3fc974f1f3ce4ae59d8220a6e3369113" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59009 } } peers { permanent_uuid: "0ed93606f04d4c3eb4d2c4a497ba385a" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 45792 } } I0703 01:41:00.180348 18949 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "38f37abe75ac4a948158a3caa9fe4176" candidate_uuid: "152925ceae284ca49f5e73752dc503c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed93606f04d4c3eb4d2c4a497ba385a" is_pre_election: true I0703 01:41:00.180584 18949 raft_consensus.cc:2730] T 38f37abe75ac4a948158a3caa9fe4176 P 0ed93606f04d4c3eb4d2c4a497ba385a: Snoozing failure detection for 1.634s (vote granted) I0703 01:41:00.180707 18949 raft_consensus.cc:2243] T 38f37abe75ac4a948158a3caa9fe4176 P 0ed93606f04d4c3eb4d2c4a497ba385a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 152925ceae284ca49f5e73752dc503c6 in term 0. I0703 01:41:00.180702 19142 leader_election.cc:243] T 38f37abe75ac4a948158a3caa9fe4176 P 3fc974f1f3ce4ae59d8220a6e3369113 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 152925ceae284ca49f5e73752dc503c6 (127.5.240.195:52628) I0703 01:41:00.181038 19142 leader_election.cc:243] T 38f37abe75ac4a948158a3caa9fe4176 P 3fc974f1f3ce4ae59d8220a6e3369113 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 0ed93606f04d4c3eb4d2c4a497ba385a (127.5.240.196:45792) I0703 01:41:00.181509 18761 leader_election.cc:379] T 38f37abe75ac4a948158a3caa9fe4176 P 152925ceae284ca49f5e73752dc503c6 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 0ed93606f04d4c3eb4d2c4a497ba385a (127.5.240.196:45792) I0703 01:41:00.181583 18761 leader_election.cc:270] T 38f37abe75ac4a948158a3caa9fe4176 P 152925ceae284ca49f5e73752dc503c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0703 01:41:00.181841 19140 raft_consensus.cc:2730] T 38f37abe75ac4a948158a3caa9fe4176 P 152925ceae284ca49f5e73752dc503c6: Snoozing failure detection for 1.701s (election complete) I0703 01:41:00.181910 19140 raft_consensus.cc:2537] T 38f37abe75ac4a948158a3caa9fe4176 P 152925ceae284ca49f5e73752dc503c6 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0703 01:41:00.181965 19140 raft_consensus.cc:457] T 38f37abe75ac4a948158a3caa9fe4176 P 152925ceae284ca49f5e73752dc503c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0703 01:41:00.182031 19140 raft_consensus.cc:2730] T 38f37abe75ac4a948158a3caa9fe4176 P 152925ceae284ca49f5e73752dc503c6: Snoozing failure detection for 1.896s (starting election) I0703 01:41:00.182102 19140 raft_consensus.cc:2782] T 38f37abe75ac4a948158a3caa9fe4176 P 152925ceae284ca49f5e73752dc503c6 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:41:00.180979 18705 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "38f37abe75ac4a948158a3caa9fe4176" candidate_uuid: "152925ceae284ca49f5e73752dc503c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc974f1f3ce4ae59d8220a6e3369113" is_pre_election: true I0703 01:41:00.184195 18705 raft_consensus.cc:2730] T 38f37abe75ac4a948158a3caa9fe4176 P 3fc974f1f3ce4ae59d8220a6e3369113: Snoozing failure detection for 1.988s (vote granted) I0703 01:41:00.184165 19140 raft_consensus.cc:479] T 38f37abe75ac4a948158a3caa9fe4176 P 152925ceae284ca49f5e73752dc503c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152925ceae284ca49f5e73752dc503c6" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 52628 } } peers { permanent_uuid: "3fc974f1f3ce4ae59d8220a6e3369113" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59009 } } peers { permanent_uuid: "0ed93606f04d4c3eb4d2c4a497ba385a" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 45792 } } I0703 01:41:00.184594 19140 leader_election.cc:243] T 38f37abe75ac4a948158a3caa9fe4176 P 152925ceae284ca49f5e73752dc503c6 [CANDIDATE]: Term 1 election: Requesting vote from peer 3fc974f1f3ce4ae59d8220a6e3369113 (127.5.240.194:59009) I0703 01:41:00.184748 19140 leader_election.cc:243] T 38f37abe75ac4a948158a3caa9fe4176 P 152925ceae284ca49f5e73752dc503c6 [CANDIDATE]: Term 1 election: Requesting vote from peer 0ed93606f04d4c3eb4d2c4a497ba385a (127.5.240.196:45792) I0703 01:41:00.184576 18705 raft_consensus.cc:2243] T 38f37abe75ac4a948158a3caa9fe4176 P 3fc974f1f3ce4ae59d8220a6e3369113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 152925ceae284ca49f5e73752dc503c6 in term 0. I0703 01:41:00.185683 18758 leader_election.cc:379] T 38f37abe75ac4a948158a3caa9fe4176 P 152925ceae284ca49f5e73752dc503c6 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 3fc974f1f3ce4ae59d8220a6e3369113 (127.5.240.194:59009) I0703 01:41:00.185652 18705 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "38f37abe75ac4a948158a3caa9fe4176" candidate_uuid: "152925ceae284ca49f5e73752dc503c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc974f1f3ce4ae59d8220a6e3369113" I0703 01:41:00.185840 18705 raft_consensus.cc:2782] T 38f37abe75ac4a948158a3caa9fe4176 P 3fc974f1f3ce4ae59d8220a6e3369113 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:41:00.185837 18949 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "38f37abe75ac4a948158a3caa9fe4176" candidate_uuid: "152925ceae284ca49f5e73752dc503c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed93606f04d4c3eb4d2c4a497ba385a" I0703 01:41:00.185930 18705 raft_consensus.cc:2730] T 38f37abe75ac4a948158a3caa9fe4176 P 3fc974f1f3ce4ae59d8220a6e3369113: Snoozing failure detection for 1.968s (vote granted) I0703 01:41:00.186007 18949 raft_consensus.cc:2782] T 38f37abe75ac4a948158a3caa9fe4176 P 0ed93606f04d4c3eb4d2c4a497ba385a [term 0 FOLLOWER]: Advancing to term 1 I0703 01:41:00.186117 18949 raft_consensus.cc:2730] T 38f37abe75ac4a948158a3caa9fe4176 P 0ed93606f04d4c3eb4d2c4a497ba385a: Snoozing failure detection for 1.599s (vote granted) I0703 01:41:00.187925 18949 raft_consensus.cc:2243] T 38f37abe75ac4a948158a3caa9fe4176 P 0ed93606f04d4c3eb4d2c4a497ba385a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 152925ceae284ca49f5e73752dc503c6 in term 1. I0703 01:41:00.188524 18761 leader_election.cc:379] T 38f37abe75ac4a948158a3caa9fe4176 P 152925ceae284ca49f5e73752dc503c6 [CANDIDATE]: Term 1 election: Vote granted by peer 0ed93606f04d4c3eb4d2c4a497ba385a (127.5.240.196:45792) I0703 01:41:00.188580 18761 leader_election.cc:270] T 38f37abe75ac4a948158a3caa9fe4176 P 152925ceae284ca49f5e73752dc503c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:41:00.188742 19140 raft_consensus.cc:2730] T 38f37abe75ac4a948158a3caa9fe4176 P 152925ceae284ca49f5e73752dc503c6: Snoozing failure detection for 2.071s (election complete) I0703 01:41:00.188813 19140 raft_consensus.cc:2537] T 38f37abe75ac4a948158a3caa9fe4176 P 152925ceae284ca49f5e73752dc503c6 [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:41:00.189708 18705 raft_consensus.cc:2243] T 38f37abe75ac4a948158a3caa9fe4176 P 3fc974f1f3ce4ae59d8220a6e3369113 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 152925ceae284ca49f5e73752dc503c6 in term 1. I0703 01:41:00.189780 19140 raft_consensus.cc:593] T 38f37abe75ac4a948158a3caa9fe4176 P 152925ceae284ca49f5e73752dc503c6 [term 1 LEADER]: Becoming Leader. State: Replica: 152925ceae284ca49f5e73752dc503c6, State: Running, Role: LEADER I0703 01:41:00.190294 18758 leader_election.cc:379] T 38f37abe75ac4a948158a3caa9fe4176 P 152925ceae284ca49f5e73752dc503c6 [CANDIDATE]: Term 1 election: Vote granted by peer 3fc974f1f3ce4ae59d8220a6e3369113 (127.5.240.194:59009) I0703 01:41:00.190522 19140 consensus_queue.cc:204] T 38f37abe75ac4a948158a3caa9fe4176 P 152925ceae284ca49f5e73752dc503c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152925ceae284ca49f5e73752dc503c6" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 52628 } } peers { permanent_uuid: "3fc974f1f3ce4ae59d8220a6e3369113" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59009 } } peers { permanent_uuid: "0ed93606f04d4c3eb4d2c4a497ba385a" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 45792 } } I0703 01:41:00.194983 18827 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "38f37abe75ac4a948158a3caa9fe4176" candidate_uuid: "3fc974f1f3ce4ae59d8220a6e3369113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "152925ceae284ca49f5e73752dc503c6" is_pre_election: true I0703 01:41:00.195427 18827 raft_consensus.cc:2204] T 38f37abe75ac4a948158a3caa9fe4176 P 152925ceae284ca49f5e73752dc503c6 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 3fc974f1f3ce4ae59d8220a6e3369113 for term 1 because replica is either leader or believes a valid leader to be alive. I0703 01:41:00.196198 18949 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "38f37abe75ac4a948158a3caa9fe4176" candidate_uuid: "3fc974f1f3ce4ae59d8220a6e3369113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed93606f04d4c3eb4d2c4a497ba385a" is_pre_election: true I0703 01:41:00.196408 18949 raft_consensus.cc:2173] T 38f37abe75ac4a948158a3caa9fe4176 P 0ed93606f04d4c3eb4d2c4a497ba385a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3fc974f1f3ce4ae59d8220a6e3369113 in current term 1: Already voted for candidate 152925ceae284ca49f5e73752dc503c6 in this term. I0703 01:41:00.196296 18462 catalog_manager.cc:3798] T 38f37abe75ac4a948158a3caa9fe4176 P 152925ceae284ca49f5e73752dc503c6 reported cstate change: term changed from 0 to 1, leader changed from to 152925ceae284ca49f5e73752dc503c6 (127.5.240.195). New cstate: current_term: 1 leader_uuid: "152925ceae284ca49f5e73752dc503c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "152925ceae284ca49f5e73752dc503c6" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 52628 } } peers { permanent_uuid: "3fc974f1f3ce4ae59d8220a6e3369113" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59009 } } peers { permanent_uuid: "0ed93606f04d4c3eb4d2c4a497ba385a" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 45792 } } } I0703 01:41:00.196640 18638 leader_election.cc:393] T 38f37abe75ac4a948158a3caa9fe4176 P 3fc974f1f3ce4ae59d8220a6e3369113 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 152925ceae284ca49f5e73752dc503c6 (127.5.240.195:52628). Message: Invalid argument: T 38f37abe75ac4a948158a3caa9fe4176 P 152925ceae284ca49f5e73752dc503c6 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 3fc974f1f3ce4ae59d8220a6e3369113 for term 1 because replica is either leader or believes a valid leader to be alive. I0703 01:41:00.197118 18639 leader_election.cc:393] T 38f37abe75ac4a948158a3caa9fe4176 P 3fc974f1f3ce4ae59d8220a6e3369113 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 0ed93606f04d4c3eb4d2c4a497ba385a (127.5.240.196:45792). Message: Invalid argument: T 38f37abe75ac4a948158a3caa9fe4176 P 0ed93606f04d4c3eb4d2c4a497ba385a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3fc974f1f3ce4ae59d8220a6e3369113 in current term 1: Already voted for candidate 152925ceae284ca49f5e73752dc503c6 in this term. I0703 01:41:00.197192 18639 leader_election.cc:270] T 38f37abe75ac4a948158a3caa9fe4176 P 3fc974f1f3ce4ae59d8220a6e3369113 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I0703 01:41:00.197515 19142 raft_consensus.cc:2730] T 38f37abe75ac4a948158a3caa9fe4176 P 3fc974f1f3ce4ae59d8220a6e3369113: Snoozing failure detection for 2.013s (election complete) I0703 01:41:00.197610 19142 raft_consensus.cc:2485] T 38f37abe75ac4a948158a3caa9fe4176 P 3fc974f1f3ce4ae59d8220a6e3369113 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0703 01:41:00.206715 6083 external_mini_cluster.cc:490] 5 TS(s) registered with all masters I0703 01:41:00.207851 6083 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 899db87084ef4ab39c468996b1edecc7 to finish bootstrapping I0703 01:41:00.213821 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 3fc974f1f3ce4ae59d8220a6e3369113 to finish bootstrapping I0703 01:41:00.218976 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 152925ceae284ca49f5e73752dc503c6 to finish bootstrapping I0703 01:41:00.223469 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 0ed93606f04d4c3eb4d2c4a497ba385a to finish bootstrapping I0703 01:41:00.228288 6083 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 8fe9841c2de149ad924d05540b4bc63f to finish bootstrapping I0703 01:41:00.235388 6083 test_util.cc:208] Using random seed: 1175025264 I0703 01:41:00.243618 6083 test_workload.cc:306] TestWorkload: Skipping table creation because table TestTable already exists I0703 01:41:00.244009 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 18627 I0703 01:41:00.244629 18745 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:41:00.257242 18949 raft_consensus.cc:1081] T 38f37abe75ac4a948158a3caa9fe4176 P 0ed93606f04d4c3eb4d2c4a497ba385a [term 1 FOLLOWER]: Refusing update from remote peer 152925ceae284ca49f5e73752dc503c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0703 01:41:00.258010 19147 consensus_queue.cc:962] T 38f37abe75ac4a948158a3caa9fe4176 P 152925ceae284ca49f5e73752dc503c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed93606f04d4c3eb4d2c4a497ba385a" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 45792 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0703 01:41:00.295895 18705 raft_consensus.cc:1081] T 38f37abe75ac4a948158a3caa9fe4176 P 3fc974f1f3ce4ae59d8220a6e3369113 [term 1 FOLLOWER]: Refusing update from remote peer 152925ceae284ca49f5e73752dc503c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0703 01:41:00.296787 19140 consensus_queue.cc:962] T 38f37abe75ac4a948158a3caa9fe4176 P 152925ceae284ca49f5e73752dc503c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fc974f1f3ce4ae59d8220a6e3369113" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0703 01:41:00.296983 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 18627 W0703 01:41:00.305681 18758 connection.cc:512] client connection to 127.5.240.194:59009 recv error: Network error: failed to read from TLS socket (remote: 0.0.0.0:0): Connection reset by peer (error 104) W0703 01:41:00.305850 18761 connection.cc:512] server connection from 127.5.240.194:41514 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:41514): Connection reset by peer (error 104) W0703 01:41:00.305896 18758 consensus_peers.cc:433] T 38f37abe75ac4a948158a3caa9fe4176 P 152925ceae284ca49f5e73752dc503c6 -> Peer 3fc974f1f3ce4ae59d8220a6e3369113 (127.5.240.194:59009): Couldn't send request to peer 3fc974f1f3ce4ae59d8220a6e3369113 for tablet 38f37abe75ac4a948158a3caa9fe4176. Status: Network error: failed to read from TLS socket (remote: 0.0.0.0:0): Connection reset by peer (error 104). Retrying in the next heartbeat period. Already tried 1 times. W0703 01:41:00.305909 18882 connection.cc:512] server connection from 127.5.240.194:38702 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:38702): Connection reset by peer (error 104) W0703 01:41:00.306004 18445 connection.cc:512] server connection from 127.5.240.194:55511 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:55511): Connection reset by peer (error 104) W0703 01:41:00.306190 18758 connection.cc:185] Error closing socket: Network error: TlsSocket::Close: Broken pipe W0703 01:41:00.563067 19165 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:41:00.724731 18758 consensus_peers.cc:433] T 38f37abe75ac4a948158a3caa9fe4176 P 152925ceae284ca49f5e73752dc503c6 -> Peer 3fc974f1f3ce4ae59d8220a6e3369113 (127.5.240.194:59009): Couldn't send request to peer 3fc974f1f3ce4ae59d8220a6e3369113 for tablet 38f37abe75ac4a948158a3caa9fe4176. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.194:59009: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. I0703 01:41:00.893218 19114 heartbeater.cc:467] Master 127.5.240.254:35377 was elected leader, sending a full tablet report... W0703 01:41:01.121846 18758 consensus_peers.cc:433] T 38f37abe75ac4a948158a3caa9fe4176 P 152925ceae284ca49f5e73752dc503c6 -> Peer 3fc974f1f3ce4ae59d8220a6e3369113 (127.5.240.194:59009): Couldn't send request to peer 3fc974f1f3ce4ae59d8220a6e3369113 for tablet 38f37abe75ac4a948158a3caa9fe4176. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.194:59009: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times. W0703 01:41:01.541649 18758 consensus_peers.cc:433] T 38f37abe75ac4a948158a3caa9fe4176 P 152925ceae284ca49f5e73752dc503c6 -> Peer 3fc974f1f3ce4ae59d8220a6e3369113 (127.5.240.194:59009): Couldn't send request to peer 3fc974f1f3ce4ae59d8220a6e3369113 for tablet 38f37abe75ac4a948158a3caa9fe4176. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.194:59009: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 4 times. W0703 01:41:01.986187 18758 consensus_peers.cc:433] T 38f37abe75ac4a948158a3caa9fe4176 P 152925ceae284ca49f5e73752dc503c6 -> Peer 3fc974f1f3ce4ae59d8220a6e3369113 (127.5.240.194:59009): Couldn't send request to peer 3fc974f1f3ce4ae59d8220a6e3369113 for tablet 38f37abe75ac4a948158a3caa9fe4176. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.194:59009: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 5 times. W0703 01:41:02.550609 18758 consensus_peers.cc:433] T 38f37abe75ac4a948158a3caa9fe4176 P 152925ceae284ca49f5e73752dc503c6 -> Peer 3fc974f1f3ce4ae59d8220a6e3369113 (127.5.240.194:59009): Couldn't send request to peer 3fc974f1f3ce4ae59d8220a6e3369113 for tablet 38f37abe75ac4a948158a3caa9fe4176. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.194:59009: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 6 times. W0703 01:41:03.052330 18758 consensus_peers.cc:433] T 38f37abe75ac4a948158a3caa9fe4176 P 152925ceae284ca49f5e73752dc503c6 -> Peer 3fc974f1f3ce4ae59d8220a6e3369113 (127.5.240.194:59009): Couldn't send request to peer 3fc974f1f3ce4ae59d8220a6e3369113 for tablet 38f37abe75ac4a948158a3caa9fe4176. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.194:59009: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 7 times. W0703 01:41:03.644120 18758 consensus_peers.cc:433] T 38f37abe75ac4a948158a3caa9fe4176 P 152925ceae284ca49f5e73752dc503c6 -> Peer 3fc974f1f3ce4ae59d8220a6e3369113 (127.5.240.194:59009): Couldn't send request to peer 3fc974f1f3ce4ae59d8220a6e3369113 for tablet 38f37abe75ac4a948158a3caa9fe4176. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.194:59009: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 8 times. W0703 01:41:04.137270 18758 consensus_peers.cc:433] T 38f37abe75ac4a948158a3caa9fe4176 P 152925ceae284ca49f5e73752dc503c6 -> Peer 3fc974f1f3ce4ae59d8220a6e3369113 (127.5.240.194:59009): Couldn't send request to peer 3fc974f1f3ce4ae59d8220a6e3369113 for tablet 38f37abe75ac4a948158a3caa9fe4176. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.194:59009: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 9 times. W0703 01:41:04.719964 18758 consensus_peers.cc:433] T 38f37abe75ac4a948158a3caa9fe4176 P 152925ceae284ca49f5e73752dc503c6 -> Peer 3fc974f1f3ce4ae59d8220a6e3369113 (127.5.240.194:59009): Couldn't send request to peer 3fc974f1f3ce4ae59d8220a6e3369113 for tablet 38f37abe75ac4a948158a3caa9fe4176. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.194:59009: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 10 times. W0703 01:41:05.323940 18758 consensus_peers.cc:433] T 38f37abe75ac4a948158a3caa9fe4176 P 152925ceae284ca49f5e73752dc503c6 -> Peer 3fc974f1f3ce4ae59d8220a6e3369113 (127.5.240.194:59009): Couldn't send request to peer 3fc974f1f3ce4ae59d8220a6e3369113 for tablet 38f37abe75ac4a948158a3caa9fe4176. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.194:59009: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 11 times. W0703 01:41:05.811224 18758 consensus_peers.cc:433] T 38f37abe75ac4a948158a3caa9fe4176 P 152925ceae284ca49f5e73752dc503c6 -> Peer 3fc974f1f3ce4ae59d8220a6e3369113 (127.5.240.194:59009): Couldn't send request to peer 3fc974f1f3ce4ae59d8220a6e3369113 for tablet 38f37abe75ac4a948158a3caa9fe4176. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.194:59009: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 12 times. Corruption: ksck pre-move health check failed: timed out with ksck errors remaining: last error: 1 out of 1 table(s) are not healthy I0703 01:41:06.077797 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 18496 I0703 01:41:06.079399 18622 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:41:06.118866 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 18496 W0703 01:41:06.126874 18446 connection.cc:512] server connection from 127.5.240.193:43510 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:43510): Connection reset by peer (error 104) I0703 01:41:06.127048 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 18751 I0703 01:41:06.127781 18867 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:41:06.240553 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 18751 W0703 01:41:06.255895 18446 connection.cc:512] server connection from 127.5.240.195:50969 recv error: Network error: failed to read from TLS socket (remote: 0.0.0.0:0): Connection reset by peer (error 104) W0703 01:41:06.255909 18883 connection.cc:512] server connection from 127.5.240.195:59283 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.195:59283): Connection reset by peer (error 104) W0703 01:41:06.256129 18446 connection.cc:185] Error closing socket: Network error: TlsSocket::Close: Broken pipe I0703 01:41:06.256151 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 18873 I0703 01:41:06.256767 18989 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:41:06.363138 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 18873 W0703 01:41:06.378605 18446 connection.cc:512] server connection from 127.5.240.196:58932 recv error: Network error: failed to read from TLS socket (remote: 0.0.0.0:0): Connection reset by peer (error 104) I0703 01:41:06.378717 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 18995 W0703 01:41:06.378764 18446 connection.cc:185] Error closing socket: Network error: TlsSocket::Close: Broken pipe I0703 01:41:06.379370 19111 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:41:06.418865 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 18995 W0703 01:41:06.426786 18446 connection.cc:512] server connection from 127.5.240.197:52271 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.197:52271): Connection reset by peer (error 104) I0703 01:41:06.426913 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master pid 18436 I0703 01:41:06.427672 18492 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:41:06.493783 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master with pid 18436 [ OK ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/1 (8646 ms) [ RUN ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/2 I0703 01:41:06.516693 6083 test_util.cc:208] Using random seed: 1181306566 I0703 01:41:06.518537 6083 ts_itest-base.cc:113] Starting cluster with: I0703 01:41:06.518597 6083 ts_itest-base.cc:114] -------------- I0703 01:41:06.518632 6083 ts_itest-base.cc:115] 5 tablet servers I0703 01:41:06.518654 6083 ts_itest-base.cc:116] 3 replicas per TS I0703 01:41:06.518684 6083 ts_itest-base.cc:117] -------------- I0703 01:41:06.519798 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:49860 --webserver_interface=127.5.240.254 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:41:06.703788 19317 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:41:06.704005 19317 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:41:06.706218 19317 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0703 01:41:06.706281 19317 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0703 01:41:06.706331 19317 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:41:06.706372 19317 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0703 01:41:06.706423 19317 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0703 01:41:06.714483 19317 master_main.cc:73] Master server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:49860 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.5.240.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:41:06.720688 19317 master_main.cc:80] Initializing master server... I0703 01:41:06.721412 19317 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 186708us I0703 01:41:06.722520 19317 fs_manager.cc:261] Metadata directory not provided I0703 01:41:06.722614 19317 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:41:06.722851 19317 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:41:06.722928 19317 server_base.cc:433] Attempting to create new FS layout instead I0703 01:41:06.727172 19317 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/instance: uuid: "0d2c312c9a074e7693e1d384090167f9" format_stamp: "Formatted at 2018-07-03 08:41:06 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:06.727721 19317 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "0d2c312c9a074e7693e1d384090167f9" format_stamp: "Formatted at 2018-07-03 08:41:06 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:06.736143 19317 fs_manager.cc:501] Time spent creating directory manager: real 0.008s user 0.004s sys 0.002s I0703 01:41:06.737387 19317 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:41:06.737831 19317 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:41:06.741575 19317 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s I0703 01:41:06.741627 19317 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal uuid: "0d2c312c9a074e7693e1d384090167f9" format_stamp: "Formatted at 2018-07-03 08:41:06 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:06.741745 19317 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:41:06.801200 19317 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:41:06.802711 19317 master_main.cc:83] Starting Master server... I0703 01:41:06.839813 19317 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.254:49860 I0703 01:41:06.840271 19317 webserver.cc:175] Starting webserver on 127.5.240.254:0 I0703 01:41:06.840312 19317 webserver.cc:186] Document root disabled I0703 01:41:06.841508 19317 webserver.cc:313] Webserver started. Bound to: http://127.5.240.254:42286/ I0703 01:41:06.842119 19317 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb I0703 01:41:06.842927 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master as pid 19317 I0703 01:41:06.843840 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.5.240.193:0 --local_ip_for_outbound_sockets=127.5.240.193 --webserver_interface=127.5.240.193 --webserver_port=0 --tserver_master_addrs=127.5.240.254:49860 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0703 01:41:06.846578 19376 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:41:06.859272 19376 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 0d2c312c9a074e7693e1d384090167f9: Bootstrap starting. I0703 01:41:06.861299 19376 tablet_bootstrap.cc:578] T 00000000000000000000000000000000 P 0d2c312c9a074e7693e1d384090167f9: No blocks or log segments found. Creating new log. I0703 01:41:06.862052 19376 log.cc:525] T 00000000000000000000000000000000 P 0d2c312c9a074e7693e1d384090167f9: Log is configured to *not* fsync() on all Append() calls I0703 01:41:06.864295 19376 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 0d2c312c9a074e7693e1d384090167f9: No bootstrap required, opened a new log I0703 01:41:06.870501 19376 raft_consensus.cc:325] T 00000000000000000000000000000000 P 0d2c312c9a074e7693e1d384090167f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2c312c9a074e7693e1d384090167f9" member_type: VOTER } I0703 01:41:06.870720 19376 raft_consensus.cc:351] T 00000000000000000000000000000000 P 0d2c312c9a074e7693e1d384090167f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:41:06.870779 19376 raft_consensus.cc:628] T 00000000000000000000000000000000 P 0d2c312c9a074e7693e1d384090167f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d2c312c9a074e7693e1d384090167f9, State: Initialized, Role: FOLLOWER I0703 01:41:06.871214 19376 consensus_queue.cc:227] T 00000000000000000000000000000000 P 0d2c312c9a074e7693e1d384090167f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2c312c9a074e7693e1d384090167f9" member_type: VOTER } I0703 01:41:06.871358 19376 raft_consensus.cc:365] T 00000000000000000000000000000000 P 0d2c312c9a074e7693e1d384090167f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:41:06.871433 19376 raft_consensus.cc:457] T 00000000000000000000000000000000 P 0d2c312c9a074e7693e1d384090167f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:41:06.871505 19376 raft_consensus.cc:2730] T 00000000000000000000000000000000 P 0d2c312c9a074e7693e1d384090167f9: Snoozing failure detection for 2.089s (starting election) I0703 01:41:06.871594 19376 raft_consensus.cc:2782] T 00000000000000000000000000000000 P 0d2c312c9a074e7693e1d384090167f9 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:41:06.872993 19376 raft_consensus.cc:479] T 00000000000000000000000000000000 P 0d2c312c9a074e7693e1d384090167f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2c312c9a074e7693e1d384090167f9" member_type: VOTER } I0703 01:41:06.873370 19376 leader_election.cc:270] T 00000000000000000000000000000000 P 0d2c312c9a074e7693e1d384090167f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:41:06.874374 19379 raft_consensus.cc:2730] T 00000000000000000000000000000000 P 0d2c312c9a074e7693e1d384090167f9: Snoozing failure detection for 1.736s (election complete) I0703 01:41:06.874644 19379 raft_consensus.cc:2537] T 00000000000000000000000000000000 P 0d2c312c9a074e7693e1d384090167f9 [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:41:06.876160 19379 raft_consensus.cc:593] T 00000000000000000000000000000000 P 0d2c312c9a074e7693e1d384090167f9 [term 1 LEADER]: Becoming Leader. State: Replica: 0d2c312c9a074e7693e1d384090167f9, State: Running, Role: LEADER I0703 01:41:06.876564 19379 consensus_queue.cc:204] T 00000000000000000000000000000000 P 0d2c312c9a074e7693e1d384090167f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2c312c9a074e7693e1d384090167f9" member_type: VOTER } I0703 01:41:06.878993 19376 sys_catalog.cc:337] T 00000000000000000000000000000000 P 0d2c312c9a074e7693e1d384090167f9 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "0d2c312c9a074e7693e1d384090167f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2c312c9a074e7693e1d384090167f9" member_type: VOTER } } I0703 01:41:06.879634 19376 sys_catalog.cc:340] T 00000000000000000000000000000000 P 0d2c312c9a074e7693e1d384090167f9 [sys.catalog]: This master's current role is: LEADER I0703 01:41:06.879181 19380 sys_catalog.cc:337] T 00000000000000000000000000000000 P 0d2c312c9a074e7693e1d384090167f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d2c312c9a074e7693e1d384090167f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2c312c9a074e7693e1d384090167f9" member_type: VOTER } } I0703 01:41:06.879179 19381 sys_catalog.cc:337] T 00000000000000000000000000000000 P 0d2c312c9a074e7693e1d384090167f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d2c312c9a074e7693e1d384090167f9. Latest consensus state: current_term: 1 leader_uuid: "0d2c312c9a074e7693e1d384090167f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d2c312c9a074e7693e1d384090167f9" member_type: VOTER } } I0703 01:41:06.880470 19381 sys_catalog.cc:340] T 00000000000000000000000000000000 P 0d2c312c9a074e7693e1d384090167f9 [sys.catalog]: This master's current role is: LEADER I0703 01:41:06.880475 19380 sys_catalog.cc:340] T 00000000000000000000000000000000 P 0d2c312c9a074e7693e1d384090167f9 [sys.catalog]: This master's current role is: LEADER I0703 01:41:06.883206 19376 sys_catalog.cc:424] T 00000000000000000000000000000000 P 0d2c312c9a074e7693e1d384090167f9 [sys.catalog]: configured and running, proceeding with master startup. I0703 01:41:06.883577 19383 catalog_manager.cc:1006] Loading table and tablet metadata into memory... I0703 01:41:06.885073 19317 master_main.cc:86] Master server successfully started. I0703 01:41:06.889962 19383 catalog_manager.cc:1017] Initializing Kudu internal certificate authority... I0703 01:41:06.953218 19383 catalog_manager.cc:904] Generated new certificate authority record I0703 01:41:06.955250 19383 catalog_manager.cc:1026] Loading token signing keys... I0703 01:41:06.964920 19383 catalog_manager.cc:4102] T 00000000000000000000000000000000 P 0d2c312c9a074e7693e1d384090167f9: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:41:07.028695 19377 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:41:07.028936 19377 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:41:07.030961 19377 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0703 01:41:07.031025 19377 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:41:07.031141 19377 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.193 I0703 01:41:07.038389 19377 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.5.240.193 --webserver_port=0 --tserver_master_addrs=127.5.240.254:49860 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.193 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:41:07.044945 19377 tablet_server_main.cc:85] Initializing tablet server... I0703 01:41:07.046489 19377 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 187208us I0703 01:41:07.047066 19377 fs_manager.cc:261] Metadata directory not provided I0703 01:41:07.047135 19377 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:41:07.047376 19377 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:41:07.047446 19377 server_base.cc:433] Attempting to create new FS layout instead I0703 01:41:07.051663 19377 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "376f9f4100594af09da21d7e8387bf57" format_stamp: "Formatted at 2018-07-03 08:41:07 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:07.052124 19377 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "376f9f4100594af09da21d7e8387bf57" format_stamp: "Formatted at 2018-07-03 08:41:07 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:07.060027 19377 fs_manager.cc:501] Time spent creating directory manager: real 0.008s user 0.005s sys 0.002s I0703 01:41:07.061228 19377 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:41:07.061661 19377 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:41:07.065726 19377 fs_manager.cc:417] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s I0703 01:41:07.065778 19377 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal uuid: "376f9f4100594af09da21d7e8387bf57" format_stamp: "Formatted at 2018-07-03 08:41:07 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:07.065914 19377 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:41:07.105520 19377 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:41:07.108108 19377 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:41:07.108525 19377 tablet_server_main.cc:90] Starting tablet server... I0703 01:41:07.203094 19377 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.193:41228 I0703 01:41:07.203562 19377 webserver.cc:175] Starting webserver on 127.5.240.193:0 I0703 01:41:07.203610 19377 webserver.cc:186] Document root disabled I0703 01:41:07.204933 19377 webserver.cc:313] Webserver started. Bound to: http://127.5.240.193:46087/ I0703 01:41:07.205461 19377 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb I0703 01:41:07.207478 19377 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:41:07.211360 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 19377 I0703 01:41:07.212222 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.5.240.194:0 --local_ip_for_outbound_sockets=127.5.240.194 --webserver_interface=127.5.240.194 --webserver_port=0 --tserver_master_addrs=127.5.240.254:49860 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0703 01:41:07.233288 19506 heartbeater.cc:346] Connected to a master server at 127.5.240.254:49860 I0703 01:41:07.233475 19506 heartbeater.cc:426] Registering TS with master... I0703 01:41:07.234377 19506 heartbeater.cc:475] Master 127.5.240.254:49860 requested a full tablet report, sending... I0703 01:41:07.235960 19343 ts_manager.cc:98] Registered new tserver with Master: 376f9f4100594af09da21d7e8387bf57 (127.5.240.193:41228) I0703 01:41:07.237967 19343 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.193:53916 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:41:07.383302 19510 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:41:07.383510 19510 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:41:07.385464 19510 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0703 01:41:07.385512 19510 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:41:07.385618 19510 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.194 I0703 01:41:07.392643 19510 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.194:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.5.240.194 --webserver_port=0 --tserver_master_addrs=127.5.240.254:49860 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.194 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:41:07.399152 19510 tablet_server_main.cc:85] Initializing tablet server... I0703 01:41:07.400676 19510 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 187208us I0703 01:41:07.401211 19510 fs_manager.cc:261] Metadata directory not provided I0703 01:41:07.401269 19510 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:41:07.401486 19510 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:41:07.401572 19510 server_base.cc:433] Attempting to create new FS layout instead I0703 01:41:07.405609 19510 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "2decc98fef5548efb47d8d15362d0fbd" format_stamp: "Formatted at 2018-07-03 08:41:07 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:07.406103 19510 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "2decc98fef5548efb47d8d15362d0fbd" format_stamp: "Formatted at 2018-07-03 08:41:07 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:07.413354 19510 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.003s sys 0.003s I0703 01:41:07.414528 19510 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:41:07.414921 19510 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:41:07.418543 19510 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s I0703 01:41:07.418612 19510 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal uuid: "2decc98fef5548efb47d8d15362d0fbd" format_stamp: "Formatted at 2018-07-03 08:41:07 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:07.418738 19510 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:41:07.441258 19510 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:41:07.443120 19510 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:41:07.443377 19510 tablet_server_main.cc:90] Starting tablet server... I0703 01:41:07.528352 19510 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.194:35509 I0703 01:41:07.528760 19510 webserver.cc:175] Starting webserver on 127.5.240.194:0 I0703 01:41:07.528800 19510 webserver.cc:186] Document root disabled I0703 01:41:07.529958 19510 webserver.cc:313] Webserver started. Bound to: http://127.5.240.194:57110/ I0703 01:41:07.530488 19510 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb I0703 01:41:07.532392 19510 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:41:07.532693 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 19510 I0703 01:41:07.533309 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.5.240.195:0 --local_ip_for_outbound_sockets=127.5.240.195 --webserver_interface=127.5.240.195 --webserver_port=0 --tserver_master_addrs=127.5.240.254:49860 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0703 01:41:07.544486 19629 heartbeater.cc:346] Connected to a master server at 127.5.240.254:49860 I0703 01:41:07.544653 19629 heartbeater.cc:426] Registering TS with master... I0703 01:41:07.545369 19629 heartbeater.cc:475] Master 127.5.240.254:49860 requested a full tablet report, sending... I0703 01:41:07.546368 19343 ts_manager.cc:98] Registered new tserver with Master: 2decc98fef5548efb47d8d15362d0fbd (127.5.240.194:35509) I0703 01:41:07.547677 19343 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.194:41813 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:41:07.694180 19632 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:41:07.694411 19632 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:41:07.696434 19632 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0703 01:41:07.696482 19632 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:41:07.696574 19632 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.195 I0703 01:41:07.703519 19632 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.195:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.5.240.195 --webserver_port=0 --tserver_master_addrs=127.5.240.254:49860 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.195 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:41:07.710089 19632 tablet_server_main.cc:85] Initializing tablet server... I0703 01:41:07.711591 19632 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 187208us I0703 01:41:07.712142 19632 fs_manager.cc:261] Metadata directory not provided I0703 01:41:07.712199 19632 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:41:07.712419 19632 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:41:07.712481 19632 server_base.cc:433] Attempting to create new FS layout instead I0703 01:41:07.716523 19632 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "0f8bacd3c4834508af3b9522493b3b46" format_stamp: "Formatted at 2018-07-03 08:41:07 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:07.717067 19632 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "0f8bacd3c4834508af3b9522493b3b46" format_stamp: "Formatted at 2018-07-03 08:41:07 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:07.724416 19632 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.003s sys 0.003s I0703 01:41:07.725661 19632 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:41:07.726061 19632 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:41:07.729624 19632 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s I0703 01:41:07.729667 19632 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal uuid: "0f8bacd3c4834508af3b9522493b3b46" format_stamp: "Formatted at 2018-07-03 08:41:07 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:07.729781 19632 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:41:07.761376 19632 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:41:07.763631 19632 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:41:07.763948 19632 tablet_server_main.cc:90] Starting tablet server... I0703 01:41:07.846879 19632 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.195:41626 I0703 01:41:07.847298 19632 webserver.cc:175] Starting webserver on 127.5.240.195:0 I0703 01:41:07.847339 19632 webserver.cc:186] Document root disabled I0703 01:41:07.848531 19632 webserver.cc:313] Webserver started. Bound to: http://127.5.240.195:42928/ I0703 01:41:07.849069 19632 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb I0703 01:41:07.853165 19632 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:41:07.857584 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 19632 I0703 01:41:07.858217 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.5.240.196:0 --local_ip_for_outbound_sockets=127.5.240.196 --webserver_interface=127.5.240.196 --webserver_port=0 --tserver_master_addrs=127.5.240.254:49860 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0703 01:41:07.862740 19751 heartbeater.cc:346] Connected to a master server at 127.5.240.254:49860 I0703 01:41:07.862888 19751 heartbeater.cc:426] Registering TS with master... I0703 01:41:07.863644 19751 heartbeater.cc:475] Master 127.5.240.254:49860 requested a full tablet report, sending... I0703 01:41:07.864753 19343 ts_manager.cc:98] Registered new tserver with Master: 0f8bacd3c4834508af3b9522493b3b46 (127.5.240.195:41626) I0703 01:41:07.865978 19343 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.195:46593 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:41:08.019444 19754 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:41:08.019682 19754 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:41:08.021713 19754 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0703 01:41:08.021760 19754 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:41:08.021844 19754 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.196 I0703 01:41:08.028811 19754 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.196:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb --webserver_interface=127.5.240.196 --webserver_port=0 --tserver_master_addrs=127.5.240.254:49860 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.196 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:41:08.035326 19754 tablet_server_main.cc:85] Initializing tablet server... I0703 01:41:08.036852 19754 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 187708us I0703 01:41:08.037415 19754 fs_manager.cc:261] Metadata directory not provided I0703 01:41:08.037472 19754 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:41:08.037720 19754 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:41:08.037781 19754 server_base.cc:433] Attempting to create new FS layout instead I0703 01:41:08.041913 19754 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/instance: uuid: "715952fe3be64ff99cf491fed834c714" format_stamp: "Formatted at 2018-07-03 08:41:08 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:08.042407 19754 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal/instance: uuid: "715952fe3be64ff99cf491fed834c714" format_stamp: "Formatted at 2018-07-03 08:41:08 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:08.049629 19754 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.003s sys 0.002s I0703 01:41:08.050865 19754 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:41:08.051244 19754 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:41:08.054314 19754 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0703 01:41:08.054359 19754 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal uuid: "715952fe3be64ff99cf491fed834c714" format_stamp: "Formatted at 2018-07-03 08:41:08 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:08.054476 19754 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:41:08.091799 19754 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:41:08.093597 19754 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:41:08.093895 19754 tablet_server_main.cc:90] Starting tablet server... I0703 01:41:08.179741 19754 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.196:41089 I0703 01:41:08.180125 19754 webserver.cc:175] Starting webserver on 127.5.240.196:0 I0703 01:41:08.180167 19754 webserver.cc:186] Document root disabled I0703 01:41:08.181318 19754 webserver.cc:313] Webserver started. Bound to: http://127.5.240.196:51240/ I0703 01:41:08.181866 19754 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb I0703 01:41:08.183147 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 19754 I0703 01:41:08.184039 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data --block_manager=log --rpc_bind_addresses=127.5.240.197:0 --local_ip_for_outbound_sockets=127.5.240.197 --webserver_interface=127.5.240.197 --webserver_port=0 --tserver_master_addrs=127.5.240.254:49860 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0703 01:41:08.191337 19754 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:41:08.203685 19873 heartbeater.cc:346] Connected to a master server at 127.5.240.254:49860 I0703 01:41:08.203830 19873 heartbeater.cc:426] Registering TS with master... I0703 01:41:08.204563 19873 heartbeater.cc:475] Master 127.5.240.254:49860 requested a full tablet report, sending... I0703 01:41:08.205531 19343 ts_manager.cc:98] Registered new tserver with Master: 715952fe3be64ff99cf491fed834c714 (127.5.240.196:41089) I0703 01:41:08.206388 19343 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.196:36307 I0703 01:41:08.240422 19506 heartbeater.cc:467] Master 127.5.240.254:49860 was elected leader, sending a full tablet report... WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:41:08.345240 19875 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:41:08.345444 19875 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:41:08.347407 19875 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0703 01:41:08.347455 19875 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:41:08.347539 19875 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.197 I0703 01:41:08.354504 19875 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.197:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb --webserver_interface=127.5.240.197 --webserver_port=0 --tserver_master_addrs=127.5.240.254:49860 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.197 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:41:08.361032 19875 tablet_server_main.cc:85] Initializing tablet server... I0703 01:41:08.362529 19875 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 187708us I0703 01:41:08.363088 19875 fs_manager.cc:261] Metadata directory not provided I0703 01:41:08.363147 19875 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:41:08.363389 19875 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:41:08.363451 19875 server_base.cc:433] Attempting to create new FS layout instead I0703 01:41:08.367472 19875 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/instance: uuid: "05cf55909ba346c59dc02479533b45c1" format_stamp: "Formatted at 2018-07-03 08:41:08 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:08.367983 19875 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal/instance: uuid: "05cf55909ba346c59dc02479533b45c1" format_stamp: "Formatted at 2018-07-03 08:41:08 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:08.375285 19875 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.003s sys 0.003s I0703 01:41:08.376480 19875 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:41:08.376894 19875 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:41:08.380482 19875 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s I0703 01:41:08.380528 19875 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal uuid: "05cf55909ba346c59dc02479533b45c1" format_stamp: "Formatted at 2018-07-03 08:41:08 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:08.380676 19875 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:41:08.410182 19875 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:41:08.411980 19875 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:41:08.412241 19875 tablet_server_main.cc:90] Starting tablet server... I0703 01:41:08.497442 19875 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.197:60809 I0703 01:41:08.497869 19875 webserver.cc:175] Starting webserver on 127.5.240.197:0 I0703 01:41:08.497906 19875 webserver.cc:186] Document root disabled I0703 01:41:08.499064 19875 webserver.cc:313] Webserver started. Bound to: http://127.5.240.197:47412/ I0703 01:41:08.499608 19875 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb I0703 01:41:08.501298 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 19875 I0703 01:41:08.506561 19875 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:41:08.517206 19995 heartbeater.cc:346] Connected to a master server at 127.5.240.254:49860 I0703 01:41:08.517375 19995 heartbeater.cc:426] Registering TS with master... I0703 01:41:08.518151 19995 heartbeater.cc:475] Master 127.5.240.254:49860 requested a full tablet report, sending... I0703 01:41:08.519081 19343 ts_manager.cc:98] Registered new tserver with Master: 05cf55909ba346c59dc02479533b45c1 (127.5.240.197:60809) I0703 01:41:08.520073 19343 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.197:44150 I0703 01:41:08.520766 6083 external_mini_cluster.cc:490] 5 TS(s) registered with all masters I0703 01:41:08.530994 19342 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.5.240.254:53932: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0703 01:41:08.549582 19629 heartbeater.cc:467] Master 127.5.240.254:49860 was elected leader, sending a full tablet report... I0703 01:41:08.556913 19566 tablet_service.cc:759] Processing CreateTablet for tablet 45c1b31759ad478cbe2483b2509fe145 (table=TestTable [id=1d098be4f2b94b1b851e3a764586cce6]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:41:08.557698 19566 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 45c1b31759ad478cbe2483b2509fe145. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:41:08.559259 19810 tablet_service.cc:759] Processing CreateTablet for tablet 45c1b31759ad478cbe2483b2509fe145 (table=TestTable [id=1d098be4f2b94b1b851e3a764586cce6]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:41:08.559842 19810 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 45c1b31759ad478cbe2483b2509fe145. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:41:08.559530 19688 tablet_service.cc:759] Processing CreateTablet for tablet 45c1b31759ad478cbe2483b2509fe145 (table=TestTable [id=1d098be4f2b94b1b851e3a764586cce6]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:41:08.560411 19688 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 45c1b31759ad478cbe2483b2509fe145. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:41:08.569826 19810 ts_tablet_manager.cc:1173] T 45c1b31759ad478cbe2483b2509fe145 P 715952fe3be64ff99cf491fed834c714: Registered tablet (data state: TABLET_DATA_READY) I0703 01:41:08.570588 20015 ts_tablet_manager.cc:1028] T 45c1b31759ad478cbe2483b2509fe145 P 715952fe3be64ff99cf491fed834c714: Bootstrapping tablet I0703 01:41:08.571142 20015 tablet_bootstrap.cc:427] T 45c1b31759ad478cbe2483b2509fe145 P 715952fe3be64ff99cf491fed834c714: Bootstrap starting. I0703 01:41:08.571245 19566 ts_tablet_manager.cc:1173] T 45c1b31759ad478cbe2483b2509fe145 P 2decc98fef5548efb47d8d15362d0fbd: Registered tablet (data state: TABLET_DATA_READY) I0703 01:41:08.572085 20016 ts_tablet_manager.cc:1028] T 45c1b31759ad478cbe2483b2509fe145 P 2decc98fef5548efb47d8d15362d0fbd: Bootstrapping tablet I0703 01:41:08.572427 20016 tablet_bootstrap.cc:427] T 45c1b31759ad478cbe2483b2509fe145 P 2decc98fef5548efb47d8d15362d0fbd: Bootstrap starting. I0703 01:41:08.573287 19688 ts_tablet_manager.cc:1173] T 45c1b31759ad478cbe2483b2509fe145 P 0f8bacd3c4834508af3b9522493b3b46: Registered tablet (data state: TABLET_DATA_READY) I0703 01:41:08.574012 20017 ts_tablet_manager.cc:1028] T 45c1b31759ad478cbe2483b2509fe145 P 0f8bacd3c4834508af3b9522493b3b46: Bootstrapping tablet I0703 01:41:08.574362 20017 tablet_bootstrap.cc:427] T 45c1b31759ad478cbe2483b2509fe145 P 0f8bacd3c4834508af3b9522493b3b46: Bootstrap starting. I0703 01:41:08.576263 20016 tablet_bootstrap.cc:578] T 45c1b31759ad478cbe2483b2509fe145 P 2decc98fef5548efb47d8d15362d0fbd: No blocks or log segments found. Creating new log. I0703 01:41:08.576732 20017 tablet_bootstrap.cc:578] T 45c1b31759ad478cbe2483b2509fe145 P 0f8bacd3c4834508af3b9522493b3b46: No blocks or log segments found. Creating new log. I0703 01:41:08.576777 20015 tablet_bootstrap.cc:578] T 45c1b31759ad478cbe2483b2509fe145 P 715952fe3be64ff99cf491fed834c714: No blocks or log segments found. Creating new log. I0703 01:41:08.577314 20016 log.cc:525] T 45c1b31759ad478cbe2483b2509fe145 P 2decc98fef5548efb47d8d15362d0fbd: Log is configured to *not* fsync() on all Append() calls I0703 01:41:08.577416 20017 log.cc:525] T 45c1b31759ad478cbe2483b2509fe145 P 0f8bacd3c4834508af3b9522493b3b46: Log is configured to *not* fsync() on all Append() calls I0703 01:41:08.577710 20015 log.cc:525] T 45c1b31759ad478cbe2483b2509fe145 P 715952fe3be64ff99cf491fed834c714: Log is configured to *not* fsync() on all Append() calls I0703 01:41:08.579962 20016 tablet_bootstrap.cc:427] T 45c1b31759ad478cbe2483b2509fe145 P 2decc98fef5548efb47d8d15362d0fbd: No bootstrap required, opened a new log I0703 01:41:08.580173 20016 ts_tablet_manager.cc:1045] T 45c1b31759ad478cbe2483b2509fe145 P 2decc98fef5548efb47d8d15362d0fbd: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s I0703 01:41:08.584260 20017 tablet_bootstrap.cc:427] T 45c1b31759ad478cbe2483b2509fe145 P 0f8bacd3c4834508af3b9522493b3b46: No bootstrap required, opened a new log I0703 01:41:08.584437 20017 ts_tablet_manager.cc:1045] T 45c1b31759ad478cbe2483b2509fe145 P 0f8bacd3c4834508af3b9522493b3b46: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.005s I0703 01:41:08.585069 20015 tablet_bootstrap.cc:427] T 45c1b31759ad478cbe2483b2509fe145 P 715952fe3be64ff99cf491fed834c714: No bootstrap required, opened a new log I0703 01:41:08.585265 20015 ts_tablet_manager.cc:1045] T 45c1b31759ad478cbe2483b2509fe145 P 715952fe3be64ff99cf491fed834c714: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.004s I0703 01:41:08.588706 20016 raft_consensus.cc:325] T 45c1b31759ad478cbe2483b2509fe145 P 2decc98fef5548efb47d8d15362d0fbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f8bacd3c4834508af3b9522493b3b46" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41626 } } peers { permanent_uuid: "715952fe3be64ff99cf491fed834c714" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 41089 } } peers { permanent_uuid: "2decc98fef5548efb47d8d15362d0fbd" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 35509 } } I0703 01:41:08.589125 20016 raft_consensus.cc:351] T 45c1b31759ad478cbe2483b2509fe145 P 2decc98fef5548efb47d8d15362d0fbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:41:08.589215 20016 raft_consensus.cc:628] T 45c1b31759ad478cbe2483b2509fe145 P 2decc98fef5548efb47d8d15362d0fbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2decc98fef5548efb47d8d15362d0fbd, State: Initialized, Role: FOLLOWER I0703 01:41:08.589764 20016 consensus_queue.cc:227] T 45c1b31759ad478cbe2483b2509fe145 P 2decc98fef5548efb47d8d15362d0fbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f8bacd3c4834508af3b9522493b3b46" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41626 } } peers { permanent_uuid: "715952fe3be64ff99cf491fed834c714" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 41089 } } peers { permanent_uuid: "2decc98fef5548efb47d8d15362d0fbd" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 35509 } } I0703 01:41:08.590410 20017 raft_consensus.cc:325] T 45c1b31759ad478cbe2483b2509fe145 P 0f8bacd3c4834508af3b9522493b3b46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f8bacd3c4834508af3b9522493b3b46" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41626 } } peers { permanent_uuid: "715952fe3be64ff99cf491fed834c714" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 41089 } } peers { permanent_uuid: "2decc98fef5548efb47d8d15362d0fbd" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 35509 } } I0703 01:41:08.590814 20017 raft_consensus.cc:351] T 45c1b31759ad478cbe2483b2509fe145 P 0f8bacd3c4834508af3b9522493b3b46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:41:08.590899 20017 raft_consensus.cc:628] T 45c1b31759ad478cbe2483b2509fe145 P 0f8bacd3c4834508af3b9522493b3b46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f8bacd3c4834508af3b9522493b3b46, State: Initialized, Role: FOLLOWER I0703 01:41:08.591295 20017 consensus_queue.cc:227] T 45c1b31759ad478cbe2483b2509fe145 P 0f8bacd3c4834508af3b9522493b3b46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f8bacd3c4834508af3b9522493b3b46" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41626 } } peers { permanent_uuid: "715952fe3be64ff99cf491fed834c714" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 41089 } } peers { permanent_uuid: "2decc98fef5548efb47d8d15362d0fbd" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 35509 } } I0703 01:41:08.591922 20016 ts_tablet_manager.cc:1073] T 45c1b31759ad478cbe2483b2509fe145 P 2decc98fef5548efb47d8d15362d0fbd: Time spent starting tablet: real 0.012s user 0.008s sys 0.003s I0703 01:41:08.592447 19751 heartbeater.cc:467] Master 127.5.240.254:49860 was elected leader, sending a full tablet report... I0703 01:41:08.592885 20017 ts_tablet_manager.cc:1073] T 45c1b31759ad478cbe2483b2509fe145 P 0f8bacd3c4834508af3b9522493b3b46: Time spent starting tablet: real 0.008s user 0.006s sys 0.003s I0703 01:41:08.595157 20015 raft_consensus.cc:325] T 45c1b31759ad478cbe2483b2509fe145 P 715952fe3be64ff99cf491fed834c714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f8bacd3c4834508af3b9522493b3b46" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41626 } } peers { permanent_uuid: "715952fe3be64ff99cf491fed834c714" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 41089 } } peers { permanent_uuid: "2decc98fef5548efb47d8d15362d0fbd" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 35509 } } I0703 01:41:08.595437 20015 raft_consensus.cc:351] T 45c1b31759ad478cbe2483b2509fe145 P 715952fe3be64ff99cf491fed834c714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:41:08.595499 20015 raft_consensus.cc:628] T 45c1b31759ad478cbe2483b2509fe145 P 715952fe3be64ff99cf491fed834c714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 715952fe3be64ff99cf491fed834c714, State: Initialized, Role: FOLLOWER I0703 01:41:08.595854 20015 consensus_queue.cc:227] T 45c1b31759ad478cbe2483b2509fe145 P 715952fe3be64ff99cf491fed834c714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f8bacd3c4834508af3b9522493b3b46" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41626 } } peers { permanent_uuid: "715952fe3be64ff99cf491fed834c714" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 41089 } } peers { permanent_uuid: "2decc98fef5548efb47d8d15362d0fbd" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 35509 } } I0703 01:41:08.596657 19873 heartbeater.cc:467] Master 127.5.240.254:49860 was elected leader, sending a full tablet report... I0703 01:41:08.597434 20015 ts_tablet_manager.cc:1073] T 45c1b31759ad478cbe2483b2509fe145 P 715952fe3be64ff99cf491fed834c714: Time spent starting tablet: real 0.012s user 0.007s sys 0.004s W0703 01:41:08.601544 19752 tablet.cc:1678] T 45c1b31759ad478cbe2483b2509fe145 P 0f8bacd3c4834508af3b9522493b3b46: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:41:08.636770 20023 raft_consensus.cc:457] T 45c1b31759ad478cbe2483b2509fe145 P 715952fe3be64ff99cf491fed834c714 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0703 01:41:08.636896 20023 raft_consensus.cc:2730] T 45c1b31759ad478cbe2483b2509fe145 P 715952fe3be64ff99cf491fed834c714: Snoozing failure detection for 1.973s (starting election) I0703 01:41:08.637022 20023 raft_consensus.cc:479] T 45c1b31759ad478cbe2483b2509fe145 P 715952fe3be64ff99cf491fed834c714 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f8bacd3c4834508af3b9522493b3b46" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41626 } } peers { permanent_uuid: "715952fe3be64ff99cf491fed834c714" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 41089 } } peers { permanent_uuid: "2decc98fef5548efb47d8d15362d0fbd" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 35509 } } I0703 01:41:08.637853 20023 leader_election.cc:243] T 45c1b31759ad478cbe2483b2509fe145 P 715952fe3be64ff99cf491fed834c714 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 0f8bacd3c4834508af3b9522493b3b46 (127.5.240.195:41626) I0703 01:41:08.638208 20023 leader_election.cc:243] T 45c1b31759ad478cbe2483b2509fe145 P 715952fe3be64ff99cf491fed834c714 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 2decc98fef5548efb47d8d15362d0fbd (127.5.240.194:35509) I0703 01:41:08.644780 19708 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "45c1b31759ad478cbe2483b2509fe145" candidate_uuid: "715952fe3be64ff99cf491fed834c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f8bacd3c4834508af3b9522493b3b46" is_pre_election: true I0703 01:41:08.645046 19708 raft_consensus.cc:2730] T 45c1b31759ad478cbe2483b2509fe145 P 0f8bacd3c4834508af3b9522493b3b46: Snoozing failure detection for 1.741s (vote granted) I0703 01:41:08.645148 19708 raft_consensus.cc:2243] T 45c1b31759ad478cbe2483b2509fe145 P 0f8bacd3c4834508af3b9522493b3b46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 715952fe3be64ff99cf491fed834c714 in term 0. I0703 01:41:08.645895 19764 leader_election.cc:379] T 45c1b31759ad478cbe2483b2509fe145 P 715952fe3be64ff99cf491fed834c714 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 0f8bacd3c4834508af3b9522493b3b46 (127.5.240.195:41626) I0703 01:41:08.645968 19764 leader_election.cc:270] T 45c1b31759ad478cbe2483b2509fe145 P 715952fe3be64ff99cf491fed834c714 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0703 01:41:08.646235 20023 raft_consensus.cc:2730] T 45c1b31759ad478cbe2483b2509fe145 P 715952fe3be64ff99cf491fed834c714: Snoozing failure detection for 1.682s (election complete) I0703 01:41:08.646311 20023 raft_consensus.cc:2537] T 45c1b31759ad478cbe2483b2509fe145 P 715952fe3be64ff99cf491fed834c714 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0703 01:41:08.646371 20023 raft_consensus.cc:457] T 45c1b31759ad478cbe2483b2509fe145 P 715952fe3be64ff99cf491fed834c714 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0703 01:41:08.646428 20023 raft_consensus.cc:2730] T 45c1b31759ad478cbe2483b2509fe145 P 715952fe3be64ff99cf491fed834c714: Snoozing failure detection for 1.902s (starting election) I0703 01:41:08.646486 20023 raft_consensus.cc:2782] T 45c1b31759ad478cbe2483b2509fe145 P 715952fe3be64ff99cf491fed834c714 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:41:08.646690 19586 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "45c1b31759ad478cbe2483b2509fe145" candidate_uuid: "715952fe3be64ff99cf491fed834c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2decc98fef5548efb47d8d15362d0fbd" is_pre_election: true I0703 01:41:08.646915 19586 raft_consensus.cc:2730] T 45c1b31759ad478cbe2483b2509fe145 P 2decc98fef5548efb47d8d15362d0fbd: Snoozing failure detection for 2.124s (vote granted) I0703 01:41:08.646994 19586 raft_consensus.cc:2243] T 45c1b31759ad478cbe2483b2509fe145 P 2decc98fef5548efb47d8d15362d0fbd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 715952fe3be64ff99cf491fed834c714 in term 0. I0703 01:41:08.647624 19762 leader_election.cc:379] T 45c1b31759ad478cbe2483b2509fe145 P 715952fe3be64ff99cf491fed834c714 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 2decc98fef5548efb47d8d15362d0fbd (127.5.240.194:35509) I0703 01:41:08.648241 20023 raft_consensus.cc:479] T 45c1b31759ad478cbe2483b2509fe145 P 715952fe3be64ff99cf491fed834c714 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f8bacd3c4834508af3b9522493b3b46" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41626 } } peers { permanent_uuid: "715952fe3be64ff99cf491fed834c714" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 41089 } } peers { permanent_uuid: "2decc98fef5548efb47d8d15362d0fbd" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 35509 } } I0703 01:41:08.648629 20023 leader_election.cc:243] T 45c1b31759ad478cbe2483b2509fe145 P 715952fe3be64ff99cf491fed834c714 [CANDIDATE]: Term 1 election: Requesting vote from peer 0f8bacd3c4834508af3b9522493b3b46 (127.5.240.195:41626) I0703 01:41:08.648809 20023 leader_election.cc:243] T 45c1b31759ad478cbe2483b2509fe145 P 715952fe3be64ff99cf491fed834c714 [CANDIDATE]: Term 1 election: Requesting vote from peer 2decc98fef5548efb47d8d15362d0fbd (127.5.240.194:35509) I0703 01:41:08.649380 19708 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "45c1b31759ad478cbe2483b2509fe145" candidate_uuid: "715952fe3be64ff99cf491fed834c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f8bacd3c4834508af3b9522493b3b46" I0703 01:41:08.649441 19586 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "45c1b31759ad478cbe2483b2509fe145" candidate_uuid: "715952fe3be64ff99cf491fed834c714" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2decc98fef5548efb47d8d15362d0fbd" I0703 01:41:08.649536 19586 raft_consensus.cc:2782] T 45c1b31759ad478cbe2483b2509fe145 P 2decc98fef5548efb47d8d15362d0fbd [term 0 FOLLOWER]: Advancing to term 1 I0703 01:41:08.649538 19708 raft_consensus.cc:2782] T 45c1b31759ad478cbe2483b2509fe145 P 0f8bacd3c4834508af3b9522493b3b46 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:41:08.649626 19708 raft_consensus.cc:2730] T 45c1b31759ad478cbe2483b2509fe145 P 0f8bacd3c4834508af3b9522493b3b46: Snoozing failure detection for 1.690s (vote granted) I0703 01:41:08.649643 19586 raft_consensus.cc:2730] T 45c1b31759ad478cbe2483b2509fe145 P 2decc98fef5548efb47d8d15362d0fbd: Snoozing failure detection for 2.230s (vote granted) I0703 01:41:08.651023 19708 raft_consensus.cc:2243] T 45c1b31759ad478cbe2483b2509fe145 P 0f8bacd3c4834508af3b9522493b3b46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 715952fe3be64ff99cf491fed834c714 in term 1. I0703 01:41:08.651059 19586 raft_consensus.cc:2243] T 45c1b31759ad478cbe2483b2509fe145 P 2decc98fef5548efb47d8d15362d0fbd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 715952fe3be64ff99cf491fed834c714 in term 1. I0703 01:41:08.651429 19764 leader_election.cc:379] T 45c1b31759ad478cbe2483b2509fe145 P 715952fe3be64ff99cf491fed834c714 [CANDIDATE]: Term 1 election: Vote granted by peer 0f8bacd3c4834508af3b9522493b3b46 (127.5.240.195:41626) I0703 01:41:08.651516 19764 leader_election.cc:270] T 45c1b31759ad478cbe2483b2509fe145 P 715952fe3be64ff99cf491fed834c714 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:41:08.651631 19762 leader_election.cc:379] T 45c1b31759ad478cbe2483b2509fe145 P 715952fe3be64ff99cf491fed834c714 [CANDIDATE]: Term 1 election: Vote granted by peer 2decc98fef5548efb47d8d15362d0fbd (127.5.240.194:35509) I0703 01:41:08.651708 20023 raft_consensus.cc:2730] T 45c1b31759ad478cbe2483b2509fe145 P 715952fe3be64ff99cf491fed834c714: Snoozing failure detection for 2.082s (election complete) I0703 01:41:08.651800 20023 raft_consensus.cc:2537] T 45c1b31759ad478cbe2483b2509fe145 P 715952fe3be64ff99cf491fed834c714 [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:41:08.652658 20023 raft_consensus.cc:593] T 45c1b31759ad478cbe2483b2509fe145 P 715952fe3be64ff99cf491fed834c714 [term 1 LEADER]: Becoming Leader. State: Replica: 715952fe3be64ff99cf491fed834c714, State: Running, Role: LEADER I0703 01:41:08.653115 20023 consensus_queue.cc:204] T 45c1b31759ad478cbe2483b2509fe145 P 715952fe3be64ff99cf491fed834c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f8bacd3c4834508af3b9522493b3b46" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41626 } } peers { permanent_uuid: "715952fe3be64ff99cf491fed834c714" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 41089 } } peers { permanent_uuid: "2decc98fef5548efb47d8d15362d0fbd" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 35509 } } I0703 01:41:08.656461 19343 catalog_manager.cc:3798] T 45c1b31759ad478cbe2483b2509fe145 P 715952fe3be64ff99cf491fed834c714 reported cstate change: term changed from 0 to 1, leader changed from to 715952fe3be64ff99cf491fed834c714 (127.5.240.196). New cstate: current_term: 1 leader_uuid: "715952fe3be64ff99cf491fed834c714" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f8bacd3c4834508af3b9522493b3b46" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41626 } } peers { permanent_uuid: "715952fe3be64ff99cf491fed834c714" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 41089 } } peers { permanent_uuid: "2decc98fef5548efb47d8d15362d0fbd" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 35509 } } } I0703 01:41:08.668191 6083 external_mini_cluster.cc:490] 5 TS(s) registered with all masters I0703 01:41:08.669293 6083 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 376f9f4100594af09da21d7e8387bf57 to finish bootstrapping I0703 01:41:08.675899 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 2decc98fef5548efb47d8d15362d0fbd to finish bootstrapping I0703 01:41:08.680660 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 0f8bacd3c4834508af3b9522493b3b46 to finish bootstrapping I0703 01:41:08.685721 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 715952fe3be64ff99cf491fed834c714 to finish bootstrapping I0703 01:41:08.690382 6083 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 05cf55909ba346c59dc02479533b45c1 to finish bootstrapping W0703 01:41:08.691735 19874 tablet.cc:1678] T 45c1b31759ad478cbe2483b2509fe145 P 715952fe3be64ff99cf491fed834c714: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:41:08.697294 6083 test_util.cc:208] Using random seed: 1183487164 I0703 01:41:08.705948 6083 test_workload.cc:306] TestWorkload: Skipping table creation because table TestTable already exists I0703 01:41:08.706372 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 19377 I0703 01:41:08.706969 19503 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:41:08.719123 19708 raft_consensus.cc:1081] T 45c1b31759ad478cbe2483b2509fe145 P 0f8bacd3c4834508af3b9522493b3b46 [term 1 FOLLOWER]: Refusing update from remote peer 715952fe3be64ff99cf491fed834c714: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0703 01:41:08.719326 19586 raft_consensus.cc:1081] T 45c1b31759ad478cbe2483b2509fe145 P 2decc98fef5548efb47d8d15362d0fbd [term 1 FOLLOWER]: Refusing update from remote peer 715952fe3be64ff99cf491fed834c714: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0703 01:41:08.720471 20025 consensus_queue.cc:962] T 45c1b31759ad478cbe2483b2509fe145 P 715952fe3be64ff99cf491fed834c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f8bacd3c4834508af3b9522493b3b46" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41626 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0703 01:41:08.720790 20023 consensus_queue.cc:962] T 45c1b31759ad478cbe2483b2509fe145 P 715952fe3be64ff99cf491fed834c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2decc98fef5548efb47d8d15362d0fbd" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0703 01:41:08.769975 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 19377 W0703 01:41:08.781247 19324 connection.cc:512] server connection from 127.5.240.193:53916 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:53916): Connection reset by peer (error 104) W0703 01:41:09.062675 20047 flags.cc:406] Enabled unsafe flag: --never_fsync=true I0703 01:41:09.345477 19830 consensus_peers.cc:455] T 45c1b31759ad478cbe2483b2509fe145 P 715952fe3be64ff99cf491fed834c714 -> Peer 2decc98fef5548efb47d8d15362d0fbd (127.5.240.194:35509): Closing peer: 2decc98fef5548efb47d8d15362d0fbd I0703 01:41:09.345580 19830 consensus_peers.cc:455] T 45c1b31759ad478cbe2483b2509fe145 P 715952fe3be64ff99cf491fed834c714 -> Peer 0f8bacd3c4834508af3b9522493b3b46 (127.5.240.195:41626): Closing peer: 0f8bacd3c4834508af3b9522493b3b46 I0703 01:41:09.345942 19830 consensus_queue.cc:204] T 45c1b31759ad478cbe2483b2509fe145 P 715952fe3be64ff99cf491fed834c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 111, Committed index: 111, Last appended: 1.111, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 112 OBSOLETE_local: false peers { permanent_uuid: "0f8bacd3c4834508af3b9522493b3b46" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41626 } } peers { permanent_uuid: "715952fe3be64ff99cf491fed834c714" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 41089 } } peers { permanent_uuid: "2decc98fef5548efb47d8d15362d0fbd" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 35509 } } peers { permanent_uuid: "05cf55909ba346c59dc02479533b45c1" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 60809 } } I0703 01:41:09.348690 19708 raft_consensus.cc:1081] T 45c1b31759ad478cbe2483b2509fe145 P 0f8bacd3c4834508af3b9522493b3b46 [term 1 FOLLOWER]: Refusing update from remote peer 715952fe3be64ff99cf491fed834c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 111. Preceding OpId from leader: term: 1 index: 112. (index mismatch) I0703 01:41:09.349059 19586 raft_consensus.cc:1081] T 45c1b31759ad478cbe2483b2509fe145 P 2decc98fef5548efb47d8d15362d0fbd [term 1 FOLLOWER]: Refusing update from remote peer 715952fe3be64ff99cf491fed834c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 111. Preceding OpId from leader: term: 1 index: 112. (index mismatch) I0703 01:41:09.349290 20046 consensus_queue.cc:962] T 45c1b31759ad478cbe2483b2509fe145 P 715952fe3be64ff99cf491fed834c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f8bacd3c4834508af3b9522493b3b46" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41626 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 112, Last known committed idx: 110, Time since last communication: 0.000s I0703 01:41:09.350106 20046 consensus_queue.cc:962] T 45c1b31759ad478cbe2483b2509fe145 P 715952fe3be64ff99cf491fed834c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2decc98fef5548efb47d8d15362d0fbd" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 112, Last known committed idx: 111, Time since last communication: 0.000s W0703 01:41:09.354387 19761 consensus_peers.cc:433] T 45c1b31759ad478cbe2483b2509fe145 P 715952fe3be64ff99cf491fed834c714 -> Peer 05cf55909ba346c59dc02479533b45c1 (127.5.240.197:60809): Couldn't send request to peer 05cf55909ba346c59dc02479533b45c1 for tablet 45c1b31759ad478cbe2483b2509fe145. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 45c1b31759ad478cbe2483b2509fe145. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:41:09.354091 20046 raft_consensus.cc:2684] T 45c1b31759ad478cbe2483b2509fe145 P 715952fe3be64ff99cf491fed834c714 [term 1 LEADER]: Committing config change with OpId 1.112: config changed from index -1 to 112, VOTER 05cf55909ba346c59dc02479533b45c1 (127.5.240.197) added. New config: { opid_index: 112 OBSOLETE_local: false peers { permanent_uuid: "0f8bacd3c4834508af3b9522493b3b46" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41626 } } peers { permanent_uuid: "715952fe3be64ff99cf491fed834c714" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 41089 } } peers { permanent_uuid: "2decc98fef5548efb47d8d15362d0fbd" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 35509 } } peers { permanent_uuid: "05cf55909ba346c59dc02479533b45c1" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 60809 } } } I0703 01:41:09.357622 19586 raft_consensus.cc:2684] T 45c1b31759ad478cbe2483b2509fe145 P 2decc98fef5548efb47d8d15362d0fbd [term 1 FOLLOWER]: Committing config change with OpId 1.112: config changed from index -1 to 112, VOTER 05cf55909ba346c59dc02479533b45c1 (127.5.240.197) added. New config: { opid_index: 112 OBSOLETE_local: false peers { permanent_uuid: "0f8bacd3c4834508af3b9522493b3b46" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41626 } } peers { permanent_uuid: "715952fe3be64ff99cf491fed834c714" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 41089 } } peers { permanent_uuid: "2decc98fef5548efb47d8d15362d0fbd" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 35509 } } peers { permanent_uuid: "05cf55909ba346c59dc02479533b45c1" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 60809 } } } I0703 01:41:09.360996 19343 catalog_manager.cc:3798] T 45c1b31759ad478cbe2483b2509fe145 P 715952fe3be64ff99cf491fed834c714 reported cstate change: config changed from index -1 to 112, VOTER 05cf55909ba346c59dc02479533b45c1 (127.5.240.197) added. New cstate: current_term: 1 leader_uuid: "715952fe3be64ff99cf491fed834c714" committed_config { opid_index: 112 OBSOLETE_local: false peers { permanent_uuid: "0f8bacd3c4834508af3b9522493b3b46" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41626 } } peers { permanent_uuid: "715952fe3be64ff99cf491fed834c714" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 41089 } } peers { permanent_uuid: "2decc98fef5548efb47d8d15362d0fbd" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 35509 } } peers { permanent_uuid: "05cf55909ba346c59dc02479533b45c1" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 60809 } } } I0703 01:41:09.360703 19708 raft_consensus.cc:2684] T 45c1b31759ad478cbe2483b2509fe145 P 0f8bacd3c4834508af3b9522493b3b46 [term 1 FOLLOWER]: Committing config change with OpId 1.112: config changed from index -1 to 112, VOTER 05cf55909ba346c59dc02479533b45c1 (127.5.240.197) added. New config: { opid_index: 112 OBSOLETE_local: false peers { permanent_uuid: "0f8bacd3c4834508af3b9522493b3b46" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41626 } } peers { permanent_uuid: "715952fe3be64ff99cf491fed834c714" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 41089 } } peers { permanent_uuid: "2decc98fef5548efb47d8d15362d0fbd" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 35509 } } peers { permanent_uuid: "05cf55909ba346c59dc02479533b45c1" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 60809 } } } I0703 01:41:09.522320 19995 heartbeater.cc:467] Master 127.5.240.254:49860 was elected leader, sending a full tablet report... I0703 01:41:09.966328 20131 ts_tablet_manager.cc:669] T 45c1b31759ad478cbe2483b2509fe145 P 05cf55909ba346c59dc02479533b45c1: Initiating tablet copy from peer 715952fe3be64ff99cf491fed834c714 (127.5.240.196:41089) I0703 01:41:09.967137 20131 tablet_copy_client.cc:239] T 45c1b31759ad478cbe2483b2509fe145 P 05cf55909ba346c59dc02479533b45c1: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.196:41089 I0703 01:41:09.981807 19850 tablet_copy_service.cc:135] P 715952fe3be64ff99cf491fed834c714: Received BeginTabletCopySession request for tablet 45c1b31759ad478cbe2483b2509fe145 from peer 05cf55909ba346c59dc02479533b45c1 ({username='jenkins'} at 127.5.240.197:42540) I0703 01:41:09.981950 19850 tablet_copy_service.cc:156] P 715952fe3be64ff99cf491fed834c714: Beginning new tablet copy session on tablet 45c1b31759ad478cbe2483b2509fe145 from peer 05cf55909ba346c59dc02479533b45c1 at {username='jenkins'} at 127.5.240.197:42540: session id = 05cf55909ba346c59dc02479533b45c1-45c1b31759ad478cbe2483b2509fe145 I0703 01:41:09.984181 19850 tablet_copy_source_session.cc:202] T 45c1b31759ad478cbe2483b2509fe145 P 715952fe3be64ff99cf491fed834c714: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:41:09.987284 20131 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 45c1b31759ad478cbe2483b2509fe145. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:41:10.003139 20131 ts_tablet_manager.cc:1173] T 45c1b31759ad478cbe2483b2509fe145 P 05cf55909ba346c59dc02479533b45c1: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:41:10.003906 20131 tablet_copy_client.cc:566] T 45c1b31759ad478cbe2483b2509fe145 P 05cf55909ba346c59dc02479533b45c1: tablet copy: Starting download of 0 data blocks... I0703 01:41:10.004194 20131 tablet_copy_client.cc:529] T 45c1b31759ad478cbe2483b2509fe145 P 05cf55909ba346c59dc02479533b45c1: tablet copy: Starting download of 1 WAL segments... I0703 01:41:10.011063 20131 tablet_copy_client.cc:405] T 45c1b31759ad478cbe2483b2509fe145 P 05cf55909ba346c59dc02479533b45c1: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:41:10.013370 20131 ts_tablet_manager.cc:1028] T 45c1b31759ad478cbe2483b2509fe145 P 05cf55909ba346c59dc02479533b45c1: Bootstrapping tablet I0703 01:41:10.013682 20131 tablet_bootstrap.cc:427] T 45c1b31759ad478cbe2483b2509fe145 P 05cf55909ba346c59dc02479533b45c1: Bootstrap starting. I0703 01:41:10.054185 20131 log.cc:525] T 45c1b31759ad478cbe2483b2509fe145 P 05cf55909ba346c59dc02479533b45c1: Log is configured to *not* fsync() on all Append() calls I0703 01:41:10.286928 20131 tablet_bootstrap.cc:427] T 45c1b31759ad478cbe2483b2509fe145 P 05cf55909ba346c59dc02479533b45c1: Bootstrap replayed 1/1 log segments. Stats: ops{read=234 overwritten=0 applied=233 ignored=0} inserts{seen=231 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates I0703 01:41:10.287381 20131 tablet_bootstrap.cc:427] T 45c1b31759ad478cbe2483b2509fe145 P 05cf55909ba346c59dc02479533b45c1: Bootstrap complete. I0703 01:41:10.287678 20131 ts_tablet_manager.cc:1045] T 45c1b31759ad478cbe2483b2509fe145 P 05cf55909ba346c59dc02479533b45c1: Time spent bootstrapping tablet: real 0.274s user 0.219s sys 0.018s I0703 01:41:10.292474 20131 raft_consensus.cc:325] T 45c1b31759ad478cbe2483b2509fe145 P 05cf55909ba346c59dc02479533b45c1 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending transactions. Active config: opid_index: 112 OBSOLETE_local: false peers { permanent_uuid: "0f8bacd3c4834508af3b9522493b3b46" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41626 } } peers { permanent_uuid: "715952fe3be64ff99cf491fed834c714" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 41089 } } peers { permanent_uuid: "2decc98fef5548efb47d8d15362d0fbd" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 35509 } } peers { permanent_uuid: "05cf55909ba346c59dc02479533b45c1" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 60809 } } I0703 01:41:10.295217 20131 raft_consensus.cc:628] T 45c1b31759ad478cbe2483b2509fe145 P 05cf55909ba346c59dc02479533b45c1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05cf55909ba346c59dc02479533b45c1, State: Initialized, Role: FOLLOWER I0703 01:41:10.295794 20131 consensus_queue.cc:227] T 45c1b31759ad478cbe2483b2509fe145 P 05cf55909ba346c59dc02479533b45c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 233, Last appended: 1.234, Last appended by leader: 234, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 112 OBSOLETE_local: false peers { permanent_uuid: "0f8bacd3c4834508af3b9522493b3b46" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41626 } } peers { permanent_uuid: "715952fe3be64ff99cf491fed834c714" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 41089 } } peers { permanent_uuid: "2decc98fef5548efb47d8d15362d0fbd" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 35509 } } peers { permanent_uuid: "05cf55909ba346c59dc02479533b45c1" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 60809 } } I0703 01:41:10.299093 20131 ts_tablet_manager.cc:1073] T 45c1b31759ad478cbe2483b2509fe145 P 05cf55909ba346c59dc02479533b45c1: Time spent starting tablet: real 0.011s user 0.008s sys 0.003s I0703 01:41:10.300186 19850 tablet_copy_service.cc:325] P 715952fe3be64ff99cf491fed834c714: Request end of tablet copy session 05cf55909ba346c59dc02479533b45c1-45c1b31759ad478cbe2483b2509fe145 received from {username='jenkins'} at 127.5.240.197:42540 I0703 01:41:10.300290 19850 tablet_copy_service.cc:413] P 715952fe3be64ff99cf491fed834c714: ending tablet copy session 05cf55909ba346c59dc02479533b45c1-45c1b31759ad478cbe2483b2509fe145 on tablet 45c1b31759ad478cbe2483b2509fe145 with peer 05cf55909ba346c59dc02479533b45c1 I0703 01:41:10.557363 19952 raft_consensus.cc:1026] T 45c1b31759ad478cbe2483b2509fe145 P 05cf55909ba346c59dc02479533b45c1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.111->[1.112-1.332] Dedup: 1.234->[1.235-1.332] I0703 01:41:11.013370 19830 consensus_peers.cc:455] T 45c1b31759ad478cbe2483b2509fe145 P 715952fe3be64ff99cf491fed834c714 -> Peer 05cf55909ba346c59dc02479533b45c1 (127.5.240.197:60809): Closing peer: 05cf55909ba346c59dc02479533b45c1 I0703 01:41:11.013504 19830 consensus_peers.cc:455] T 45c1b31759ad478cbe2483b2509fe145 P 715952fe3be64ff99cf491fed834c714 -> Peer 2decc98fef5548efb47d8d15362d0fbd (127.5.240.194:35509): Closing peer: 2decc98fef5548efb47d8d15362d0fbd I0703 01:41:11.013628 19830 consensus_peers.cc:455] T 45c1b31759ad478cbe2483b2509fe145 P 715952fe3be64ff99cf491fed834c714 -> Peer 0f8bacd3c4834508af3b9522493b3b46 (127.5.240.195:41626): Closing peer: 0f8bacd3c4834508af3b9522493b3b46 I0703 01:41:11.013865 19830 consensus_queue.cc:204] T 45c1b31759ad478cbe2483b2509fe145 P 715952fe3be64ff99cf491fed834c714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 400, Committed index: 400, Last appended: 1.401, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 402 OBSOLETE_local: false peers { permanent_uuid: "715952fe3be64ff99cf491fed834c714" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 41089 } } peers { permanent_uuid: "2decc98fef5548efb47d8d15362d0fbd" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 35509 } } peers { permanent_uuid: "05cf55909ba346c59dc02479533b45c1" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 60809 } } I0703 01:41:11.016703 19585 raft_consensus.cc:1081] T 45c1b31759ad478cbe2483b2509fe145 P 2decc98fef5548efb47d8d15362d0fbd [term 1 FOLLOWER]: Refusing update from remote peer 715952fe3be64ff99cf491fed834c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 401. Preceding OpId from leader: term: 1 index: 402. (index mismatch) I0703 01:41:11.017814 20023 consensus_queue.cc:962] T 45c1b31759ad478cbe2483b2509fe145 P 715952fe3be64ff99cf491fed834c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2decc98fef5548efb47d8d15362d0fbd" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 402, Last known committed idx: 400, Time since last communication: 0.000s I0703 01:41:11.018092 19952 raft_consensus.cc:1081] T 45c1b31759ad478cbe2483b2509fe145 P 05cf55909ba346c59dc02479533b45c1 [term 1 FOLLOWER]: Refusing update from remote peer 715952fe3be64ff99cf491fed834c714: Log matching property violated. Preceding OpId in replica: term: 1 index: 401. Preceding OpId from leader: term: 1 index: 402. (index mismatch) I0703 01:41:11.018697 20023 consensus_queue.cc:962] T 45c1b31759ad478cbe2483b2509fe145 P 715952fe3be64ff99cf491fed834c714 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05cf55909ba346c59dc02479533b45c1" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 60809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 402, Last known committed idx: 400, Time since last communication: 0.000s I0703 01:41:11.019861 20025 raft_consensus.cc:2684] T 45c1b31759ad478cbe2483b2509fe145 P 715952fe3be64ff99cf491fed834c714 [term 1 LEADER]: Committing config change with OpId 1.402: config changed from index 112 to 402, VOTER 0f8bacd3c4834508af3b9522493b3b46 (127.5.240.195) evicted. New config: { opid_index: 402 OBSOLETE_local: false peers { permanent_uuid: "715952fe3be64ff99cf491fed834c714" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 41089 } } peers { permanent_uuid: "2decc98fef5548efb47d8d15362d0fbd" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 35509 } } peers { permanent_uuid: "05cf55909ba346c59dc02479533b45c1" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 60809 } } } I0703 01:41:11.021204 19585 raft_consensus.cc:2684] T 45c1b31759ad478cbe2483b2509fe145 P 2decc98fef5548efb47d8d15362d0fbd [term 1 FOLLOWER]: Committing config change with OpId 1.402: config changed from index 112 to 402, VOTER 0f8bacd3c4834508af3b9522493b3b46 (127.5.240.195) evicted. New config: { opid_index: 402 OBSOLETE_local: false peers { permanent_uuid: "715952fe3be64ff99cf491fed834c714" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 41089 } } peers { permanent_uuid: "2decc98fef5548efb47d8d15362d0fbd" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 35509 } } peers { permanent_uuid: "05cf55909ba346c59dc02479533b45c1" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 60809 } } } I0703 01:41:11.024446 19343 catalog_manager.cc:3798] T 45c1b31759ad478cbe2483b2509fe145 P 715952fe3be64ff99cf491fed834c714 reported cstate change: config changed from index 112 to 402, VOTER 0f8bacd3c4834508af3b9522493b3b46 (127.5.240.195) evicted. New cstate: current_term: 1 leader_uuid: "715952fe3be64ff99cf491fed834c714" committed_config { opid_index: 402 OBSOLETE_local: false peers { permanent_uuid: "715952fe3be64ff99cf491fed834c714" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 41089 } } peers { permanent_uuid: "2decc98fef5548efb47d8d15362d0fbd" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 35509 } } peers { permanent_uuid: "05cf55909ba346c59dc02479533b45c1" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 60809 } } } I0703 01:41:11.027750 19952 raft_consensus.cc:2684] T 45c1b31759ad478cbe2483b2509fe145 P 05cf55909ba346c59dc02479533b45c1 [term 1 FOLLOWER]: Committing config change with OpId 1.402: config changed from index 112 to 402, VOTER 0f8bacd3c4834508af3b9522493b3b46 (127.5.240.195) evicted. New config: { opid_index: 402 OBSOLETE_local: false peers { permanent_uuid: "715952fe3be64ff99cf491fed834c714" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 41089 } } peers { permanent_uuid: "2decc98fef5548efb47d8d15362d0fbd" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 35509 } } peers { permanent_uuid: "05cf55909ba346c59dc02479533b45c1" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 60809 } } } I0703 01:41:11.041733 19343 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 45c1b31759ad478cbe2483b2509fe145 on 0f8bacd3c4834508af3b9522493b3b46 (127.5.240.195:41626) (TS 0f8bacd3c4834508af3b9522493b3b46 not found in new config with opid_index 402) I0703 01:41:11.042811 19688 tablet_service.cc:800] Processing DeleteTablet for tablet 45c1b31759ad478cbe2483b2509fe145 with delete_type TABLET_DATA_TOMBSTONED (TS 0f8bacd3c4834508af3b9522493b3b46 not found in new config with opid_index 402) from {username='jenkins'} at 127.5.240.195:37245 I0703 01:41:11.043457 20226 tablet_replica.cc:261] T 45c1b31759ad478cbe2483b2509fe145 P 0f8bacd3c4834508af3b9522493b3b46: stopping tablet replica I0703 01:41:11.043613 20226 maintenance_manager.cc:237] P 0f8bacd3c4834508af3b9522493b3b46: Unregistered op CompactRowSetsOp(45c1b31759ad478cbe2483b2509fe145) I0703 01:41:11.043690 20226 maintenance_manager.cc:237] P 0f8bacd3c4834508af3b9522493b3b46: Unregistered op MinorDeltaCompactionOp(45c1b31759ad478cbe2483b2509fe145) I0703 01:41:11.043738 20226 maintenance_manager.cc:237] P 0f8bacd3c4834508af3b9522493b3b46: Unregistered op MajorDeltaCompactionOp(45c1b31759ad478cbe2483b2509fe145) I0703 01:41:11.043803 20226 maintenance_manager.cc:237] P 0f8bacd3c4834508af3b9522493b3b46: Unregistered op UndoDeltaBlockGCOp(45c1b31759ad478cbe2483b2509fe145) I0703 01:41:11.043885 20226 maintenance_manager.cc:237] P 0f8bacd3c4834508af3b9522493b3b46: Unregistered op FlushMRSOp(45c1b31759ad478cbe2483b2509fe145) I0703 01:41:11.043946 20226 maintenance_manager.cc:237] P 0f8bacd3c4834508af3b9522493b3b46: Unregistered op FlushDeltaMemStoresOp(45c1b31759ad478cbe2483b2509fe145) I0703 01:41:11.043988 20226 maintenance_manager.cc:237] P 0f8bacd3c4834508af3b9522493b3b46: Unregistered op LogGCOp(45c1b31759ad478cbe2483b2509fe145) I0703 01:41:11.044050 20226 raft_consensus.cc:2040] T 45c1b31759ad478cbe2483b2509fe145 P 0f8bacd3c4834508af3b9522493b3b46 [term 1 FOLLOWER]: Raft consensus shutting down. I0703 01:41:11.044230 20226 pending_rounds.cc:63] T 45c1b31759ad478cbe2483b2509fe145 P 0f8bacd3c4834508af3b9522493b3b46: Trying to abort 1 pending transactions. I0703 01:41:11.044271 20226 pending_rounds.cc:68] T 45c1b31759ad478cbe2483b2509fe145 P 0f8bacd3c4834508af3b9522493b3b46: Aborting transaction as it isn't in flight: id { term: 1 index: 401 } timestamp: 6269367382067105792 op_type: WRITE_OP write_request { tablet_id: "45c1b31759ad478cbe2483b2509fe145" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true } } row_operations { rows: "\001\007\000\353\250\215D\300\264\027\017\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "d76070f6f930a695" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 6269367382039867392 } request_id { client_id: "7c7e5a6910a848fa802050b8cc3450b4" seq_no: 398 first_incomplete_seq_no: 398 attempt_no: 0 } I0703 01:41:11.044800 20226 raft_consensus.cc:2067] T 45c1b31759ad478cbe2483b2509fe145 P 0f8bacd3c4834508af3b9522493b3b46 [term 1 FOLLOWER]: Raft consensus is shut down! I0703 01:41:11.046257 20226 ts_tablet_manager.cc:1369] T 45c1b31759ad478cbe2483b2509fe145 P 0f8bacd3c4834508af3b9522493b3b46: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:41:11.049262 20226 ts_tablet_manager.cc:1382] T 45c1b31759ad478cbe2483b2509fe145 P 0f8bacd3c4834508af3b9522493b3b46: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.401 I0703 01:41:11.049355 20226 log.cc:986] T 45c1b31759ad478cbe2483b2509fe145 P 0f8bacd3c4834508af3b9522493b3b46: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal/wals/45c1b31759ad478cbe2483b2509fe145 I0703 01:41:11.050283 19327 catalog_manager.cc:3103] TS 0f8bacd3c4834508af3b9522493b3b46 (127.5.240.195:41626): tablet 45c1b31759ad478cbe2483b2509fe145 (table TestTable [id=1d098be4f2b94b1b851e3a764586cce6]) successfully deleted I0703 01:41:11.645157 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 19510 I0703 01:41:11.647279 19626 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:41:11.714418 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 19510 W0703 01:41:11.725867 19762 connection.cc:512] client connection to 127.5.240.194:35509 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:35509): Connection reset by peer (error 104) I0703 01:41:11.726020 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 19632 W0703 01:41:11.726301 19324 connection.cc:512] server connection from 127.5.240.194:41813 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:41813): Connection reset by peer (error 104) I0703 01:41:11.726734 19748 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:41:11.788518 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 19632 W0703 01:41:11.799444 19764 connection.cc:512] client connection to 127.5.240.195:41626 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.195:41626): Connection reset by peer (error 104) W0703 01:41:11.799654 19324 connection.cc:512] server connection from 127.5.240.195:46593 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.195:46593): Connection reset by peer (error 104) I0703 01:41:11.800182 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 19754 I0703 01:41:11.800886 19870 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:41:11.888885 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 19754 W0703 01:41:11.902505 19324 connection.cc:512] server connection from 127.5.240.196:36307 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.196:36307): Connection reset by peer (error 104) I0703 01:41:11.902909 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 19875 W0703 01:41:11.903162 19883 connection.cc:512] server connection from 127.5.240.196:56279 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.196:56279): Connection reset by peer (error 104) W0703 01:41:11.903302 19885 connection.cc:512] client connection to 127.5.240.196:41089 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.196:41089): Connection reset by peer (error 104) I0703 01:41:11.903736 19992 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:41:11.960309 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 19875 W0703 01:41:11.970103 19324 connection.cc:512] server connection from 127.5.240.197:44150 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.197:44150): Connection reset by peer (error 104) I0703 01:41:11.970263 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master pid 19317 I0703 01:41:11.971030 19373 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:41:12.037310 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master with pid 19317 [ OK ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/2 (5544 ms) [ RUN ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/3 I0703 01:41:12.060797 6083 test_util.cc:208] Using random seed: 1186850666 I0703 01:41:12.062512 6083 ts_itest-base.cc:113] Starting cluster with: I0703 01:41:12.062583 6083 ts_itest-base.cc:114] -------------- I0703 01:41:12.062657 6083 ts_itest-base.cc:115] 5 tablet servers I0703 01:41:12.062707 6083 ts_itest-base.cc:116] 3 replicas per TS I0703 01:41:12.062726 6083 ts_itest-base.cc:117] -------------- I0703 01:41:12.063796 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:45994 --webserver_interface=127.5.240.254 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:41:12.245731 20246 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:41:12.245934 20246 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:41:12.248122 20246 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0703 01:41:12.248170 20246 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0703 01:41:12.248198 20246 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:41:12.248219 20246 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0703 01:41:12.248247 20246 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0703 01:41:12.256189 20246 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:45994 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.5.240.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:41:12.262187 20246 master_main.cc:80] Initializing master server... I0703 01:41:12.262866 20246 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 189708us I0703 01:41:12.263890 20246 fs_manager.cc:261] Metadata directory not provided I0703 01:41:12.263948 20246 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:41:12.264164 20246 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:41:12.264226 20246 server_base.cc:433] Attempting to create new FS layout instead I0703 01:41:12.268374 20246 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/instance: uuid: "cab481b3b8e34cbbbbfec8ab4b0e9607" format_stamp: "Formatted at 2018-07-03 08:41:12 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:12.268890 20246 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "cab481b3b8e34cbbbbfec8ab4b0e9607" format_stamp: "Formatted at 2018-07-03 08:41:12 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:12.276893 20246 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.004s sys 0.003s I0703 01:41:12.278111 20246 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:41:12.278486 20246 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:41:12.282116 20246 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s I0703 01:41:12.282161 20246 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal uuid: "cab481b3b8e34cbbbbfec8ab4b0e9607" format_stamp: "Formatted at 2018-07-03 08:41:12 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:12.282261 20246 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:41:12.337057 20246 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:41:12.337958 20246 master_main.cc:83] Starting Master server... I0703 01:41:12.374372 20246 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.254:45994 I0703 01:41:12.374815 20246 webserver.cc:175] Starting webserver on 127.5.240.254:0 I0703 01:41:12.374850 20246 webserver.cc:186] Document root disabled I0703 01:41:12.376009 20246 webserver.cc:313] Webserver started. Bound to: http://127.5.240.254:36839/ I0703 01:41:12.376597 20246 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb I0703 01:41:12.380220 20305 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:41:12.386831 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master as pid 20246 I0703 01:41:12.387434 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.5.240.193:0 --local_ip_for_outbound_sockets=127.5.240.193 --webserver_interface=127.5.240.193 --webserver_port=0 --tserver_master_addrs=127.5.240.254:45994 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0703 01:41:12.391978 20305 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P cab481b3b8e34cbbbbfec8ab4b0e9607: Bootstrap starting. I0703 01:41:12.393999 20305 tablet_bootstrap.cc:578] T 00000000000000000000000000000000 P cab481b3b8e34cbbbbfec8ab4b0e9607: No blocks or log segments found. Creating new log. I0703 01:41:12.394753 20305 log.cc:525] T 00000000000000000000000000000000 P cab481b3b8e34cbbbbfec8ab4b0e9607: Log is configured to *not* fsync() on all Append() calls I0703 01:41:12.396468 20305 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P cab481b3b8e34cbbbbfec8ab4b0e9607: No bootstrap required, opened a new log I0703 01:41:12.402613 20305 raft_consensus.cc:325] T 00000000000000000000000000000000 P cab481b3b8e34cbbbbfec8ab4b0e9607 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab481b3b8e34cbbbbfec8ab4b0e9607" member_type: VOTER } I0703 01:41:12.402808 20305 raft_consensus.cc:351] T 00000000000000000000000000000000 P cab481b3b8e34cbbbbfec8ab4b0e9607 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:41:12.402869 20305 raft_consensus.cc:628] T 00000000000000000000000000000000 P cab481b3b8e34cbbbbfec8ab4b0e9607 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab481b3b8e34cbbbbfec8ab4b0e9607, State: Initialized, Role: FOLLOWER I0703 01:41:12.403311 20305 consensus_queue.cc:227] T 00000000000000000000000000000000 P cab481b3b8e34cbbbbfec8ab4b0e9607 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab481b3b8e34cbbbbfec8ab4b0e9607" member_type: VOTER } I0703 01:41:12.403453 20305 raft_consensus.cc:365] T 00000000000000000000000000000000 P cab481b3b8e34cbbbbfec8ab4b0e9607 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:41:12.403554 20305 raft_consensus.cc:457] T 00000000000000000000000000000000 P cab481b3b8e34cbbbbfec8ab4b0e9607 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:41:12.403643 20305 raft_consensus.cc:2730] T 00000000000000000000000000000000 P cab481b3b8e34cbbbbfec8ab4b0e9607: Snoozing failure detection for 1.665s (starting election) I0703 01:41:12.403736 20305 raft_consensus.cc:2782] T 00000000000000000000000000000000 P cab481b3b8e34cbbbbfec8ab4b0e9607 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:41:12.405068 20305 raft_consensus.cc:479] T 00000000000000000000000000000000 P cab481b3b8e34cbbbbfec8ab4b0e9607 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab481b3b8e34cbbbbfec8ab4b0e9607" member_type: VOTER } I0703 01:41:12.405434 20305 leader_election.cc:270] T 00000000000000000000000000000000 P cab481b3b8e34cbbbbfec8ab4b0e9607 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:41:12.406994 20308 raft_consensus.cc:2730] T 00000000000000000000000000000000 P cab481b3b8e34cbbbbfec8ab4b0e9607: Snoozing failure detection for 1.523s (election complete) I0703 01:41:12.407994 20308 raft_consensus.cc:2537] T 00000000000000000000000000000000 P cab481b3b8e34cbbbbfec8ab4b0e9607 [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:41:12.408751 20308 raft_consensus.cc:593] T 00000000000000000000000000000000 P cab481b3b8e34cbbbbfec8ab4b0e9607 [term 1 LEADER]: Becoming Leader. State: Replica: cab481b3b8e34cbbbbfec8ab4b0e9607, State: Running, Role: LEADER I0703 01:41:12.409123 20308 consensus_queue.cc:204] T 00000000000000000000000000000000 P cab481b3b8e34cbbbbfec8ab4b0e9607 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab481b3b8e34cbbbbfec8ab4b0e9607" member_type: VOTER } I0703 01:41:12.411407 20305 sys_catalog.cc:337] T 00000000000000000000000000000000 P cab481b3b8e34cbbbbfec8ab4b0e9607 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "cab481b3b8e34cbbbbfec8ab4b0e9607" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab481b3b8e34cbbbbfec8ab4b0e9607" member_type: VOTER } } I0703 01:41:12.412212 20305 sys_catalog.cc:340] T 00000000000000000000000000000000 P cab481b3b8e34cbbbbfec8ab4b0e9607 [sys.catalog]: This master's current role is: LEADER I0703 01:41:12.411638 20309 sys_catalog.cc:337] T 00000000000000000000000000000000 P cab481b3b8e34cbbbbfec8ab4b0e9607 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cab481b3b8e34cbbbbfec8ab4b0e9607" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab481b3b8e34cbbbbfec8ab4b0e9607" member_type: VOTER } } I0703 01:41:12.411620 20310 sys_catalog.cc:337] T 00000000000000000000000000000000 P cab481b3b8e34cbbbbfec8ab4b0e9607 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cab481b3b8e34cbbbbfec8ab4b0e9607. Latest consensus state: current_term: 1 leader_uuid: "cab481b3b8e34cbbbbfec8ab4b0e9607" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab481b3b8e34cbbbbfec8ab4b0e9607" member_type: VOTER } } I0703 01:41:12.413177 20310 sys_catalog.cc:340] T 00000000000000000000000000000000 P cab481b3b8e34cbbbbfec8ab4b0e9607 [sys.catalog]: This master's current role is: LEADER I0703 01:41:12.413385 20309 sys_catalog.cc:340] T 00000000000000000000000000000000 P cab481b3b8e34cbbbbfec8ab4b0e9607 [sys.catalog]: This master's current role is: LEADER I0703 01:41:12.415403 20305 sys_catalog.cc:424] T 00000000000000000000000000000000 P cab481b3b8e34cbbbbfec8ab4b0e9607 [sys.catalog]: configured and running, proceeding with master startup. I0703 01:41:12.415634 20312 catalog_manager.cc:1006] Loading table and tablet metadata into memory... I0703 01:41:12.417143 20246 master_main.cc:86] Master server successfully started. I0703 01:41:12.422226 20312 catalog_manager.cc:1017] Initializing Kudu internal certificate authority... I0703 01:41:12.469595 20312 catalog_manager.cc:904] Generated new certificate authority record I0703 01:41:12.471642 20312 catalog_manager.cc:1026] Loading token signing keys... I0703 01:41:12.479429 20312 catalog_manager.cc:4102] T 00000000000000000000000000000000 P cab481b3b8e34cbbbbfec8ab4b0e9607: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:41:12.563191 20306 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:41:12.563424 20306 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:41:12.565426 20306 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0703 01:41:12.565496 20306 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:41:12.565629 20306 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.193 I0703 01:41:12.572252 20306 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.5.240.193 --webserver_port=0 --tserver_master_addrs=127.5.240.254:45994 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.193 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:41:12.578950 20306 tablet_server_main.cc:85] Initializing tablet server... I0703 01:41:12.580538 20306 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 189708us I0703 01:41:12.581137 20306 fs_manager.cc:261] Metadata directory not provided I0703 01:41:12.581205 20306 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:41:12.581828 20306 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:41:12.581900 20306 server_base.cc:433] Attempting to create new FS layout instead I0703 01:41:12.586068 20306 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "86055af88ab04036878f3822fe571c07" format_stamp: "Formatted at 2018-07-03 08:41:12 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:12.586576 20306 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "86055af88ab04036878f3822fe571c07" format_stamp: "Formatted at 2018-07-03 08:41:12 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:12.595468 20306 fs_manager.cc:501] Time spent creating directory manager: real 0.009s user 0.004s sys 0.001s I0703 01:41:12.596688 20306 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:41:12.597080 20306 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:41:12.601241 20306 fs_manager.cc:417] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s I0703 01:41:12.601294 20306 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal uuid: "86055af88ab04036878f3822fe571c07" format_stamp: "Formatted at 2018-07-03 08:41:12 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:12.601430 20306 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:41:12.623111 20306 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:41:12.625619 20306 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:41:12.626026 20306 tablet_server_main.cc:90] Starting tablet server... I0703 01:41:12.729316 20306 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.193:42232 I0703 01:41:12.729809 20306 webserver.cc:175] Starting webserver on 127.5.240.193:0 I0703 01:41:12.729854 20306 webserver.cc:186] Document root disabled I0703 01:41:12.730650 20306 webserver.cc:313] Webserver started. Bound to: http://127.5.240.193:56855/ I0703 01:41:12.731184 20306 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb I0703 01:41:12.731863 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 20306 I0703 01:41:12.732741 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.5.240.194:0 --local_ip_for_outbound_sockets=127.5.240.194 --webserver_interface=127.5.240.194 --webserver_port=0 --tserver_master_addrs=127.5.240.254:45994 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0703 01:41:12.746969 20306 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:41:12.769873 20435 heartbeater.cc:346] Connected to a master server at 127.5.240.254:45994 I0703 01:41:12.770042 20435 heartbeater.cc:426] Registering TS with master... I0703 01:41:12.770872 20435 heartbeater.cc:475] Master 127.5.240.254:45994 requested a full tablet report, sending... I0703 01:41:12.772328 20272 ts_manager.cc:98] Registered new tserver with Master: 86055af88ab04036878f3822fe571c07 (127.5.240.193:42232) I0703 01:41:12.774144 20272 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.193:44264 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:41:12.901176 20436 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:41:12.901384 20436 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:41:12.903363 20436 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0703 01:41:12.903411 20436 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:41:12.903496 20436 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.194 I0703 01:41:12.909967 20436 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.194:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.5.240.194 --webserver_port=0 --tserver_master_addrs=127.5.240.254:45994 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.194 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:41:12.916118 20436 tablet_server_main.cc:85] Initializing tablet server... I0703 01:41:12.917639 20436 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 189708us I0703 01:41:12.918174 20436 fs_manager.cc:261] Metadata directory not provided I0703 01:41:12.918232 20436 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:41:12.918445 20436 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:41:12.918505 20436 server_base.cc:433] Attempting to create new FS layout instead I0703 01:41:12.922956 20436 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "8c3e06132b5b44fb8eb4fc451662d466" format_stamp: "Formatted at 2018-07-03 08:41:12 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:12.923480 20436 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "8c3e06132b5b44fb8eb4fc451662d466" format_stamp: "Formatted at 2018-07-03 08:41:12 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:12.930775 20436 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.003s sys 0.003s I0703 01:41:12.931947 20436 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:41:12.932305 20436 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:41:12.935851 20436 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0703 01:41:12.935896 20436 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal uuid: "8c3e06132b5b44fb8eb4fc451662d466" format_stamp: "Formatted at 2018-07-03 08:41:12 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:12.936008 20436 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:41:12.968732 20436 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:41:12.971022 20436 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:41:12.971366 20436 tablet_server_main.cc:90] Starting tablet server... I0703 01:41:13.055256 20436 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.194:57030 I0703 01:41:13.055672 20436 webserver.cc:175] Starting webserver on 127.5.240.194:0 I0703 01:41:13.055711 20436 webserver.cc:186] Document root disabled I0703 01:41:13.056386 20436 webserver.cc:313] Webserver started. Bound to: http://127.5.240.194:46758/ I0703 01:41:13.056936 20436 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb I0703 01:41:13.058665 20436 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:41:13.061605 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 20436 I0703 01:41:13.062357 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.5.240.195:0 --local_ip_for_outbound_sockets=127.5.240.195 --webserver_interface=127.5.240.195 --webserver_port=0 --tserver_master_addrs=127.5.240.254:45994 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0703 01:41:13.071211 20558 heartbeater.cc:346] Connected to a master server at 127.5.240.254:45994 I0703 01:41:13.071373 20558 heartbeater.cc:426] Registering TS with master... I0703 01:41:13.072178 20558 heartbeater.cc:475] Master 127.5.240.254:45994 requested a full tablet report, sending... I0703 01:41:13.073274 20272 ts_manager.cc:98] Registered new tserver with Master: 8c3e06132b5b44fb8eb4fc451662d466 (127.5.240.194:57030) I0703 01:41:13.074118 20272 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.194:59409 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:41:13.222311 20561 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:41:13.222517 20561 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:41:13.224519 20561 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0703 01:41:13.224587 20561 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:41:13.224691 20561 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.195 I0703 01:41:13.231571 20561 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.195:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.5.240.195 --webserver_port=0 --tserver_master_addrs=127.5.240.254:45994 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.195 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:41:13.238054 20561 tablet_server_main.cc:85] Initializing tablet server... I0703 01:41:13.239573 20561 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 190208us I0703 01:41:13.240125 20561 fs_manager.cc:261] Metadata directory not provided I0703 01:41:13.240181 20561 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:41:13.240387 20561 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:41:13.240447 20561 server_base.cc:433] Attempting to create new FS layout instead I0703 01:41:13.244518 20561 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "d7efcfaf2a624349a32682447014e68f" format_stamp: "Formatted at 2018-07-03 08:41:13 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:13.245071 20561 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "d7efcfaf2a624349a32682447014e68f" format_stamp: "Formatted at 2018-07-03 08:41:13 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:13.252246 20561 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.003s sys 0.003s I0703 01:41:13.253433 20561 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:41:13.253811 20561 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:41:13.257354 20561 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s I0703 01:41:13.257398 20561 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal uuid: "d7efcfaf2a624349a32682447014e68f" format_stamp: "Formatted at 2018-07-03 08:41:13 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:13.257508 20561 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:41:13.305833 20561 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:41:13.308068 20561 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:41:13.308411 20561 tablet_server_main.cc:90] Starting tablet server... I0703 01:41:13.394290 20561 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.195:41493 I0703 01:41:13.394707 20561 webserver.cc:175] Starting webserver on 127.5.240.195:0 I0703 01:41:13.394747 20561 webserver.cc:186] Document root disabled I0703 01:41:13.395946 20561 webserver.cc:313] Webserver started. Bound to: http://127.5.240.195:55557/ I0703 01:41:13.396505 20561 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb I0703 01:41:13.396919 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 20561 I0703 01:41:13.397809 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.5.240.196:0 --local_ip_for_outbound_sockets=127.5.240.196 --webserver_interface=127.5.240.196 --webserver_port=0 --tserver_master_addrs=127.5.240.254:45994 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0703 01:41:13.402829 20561 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:41:13.410717 20680 heartbeater.cc:346] Connected to a master server at 127.5.240.254:45994 I0703 01:41:13.410850 20680 heartbeater.cc:426] Registering TS with master... I0703 01:41:13.411636 20680 heartbeater.cc:475] Master 127.5.240.254:45994 requested a full tablet report, sending... I0703 01:41:13.412590 20272 ts_manager.cc:98] Registered new tserver with Master: d7efcfaf2a624349a32682447014e68f (127.5.240.195:41493) I0703 01:41:13.413400 20272 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.195:42571 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:41:13.557399 20683 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:41:13.557641 20683 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:41:13.559722 20683 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0703 01:41:13.559780 20683 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:41:13.559881 20683 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.196 I0703 01:41:13.566434 20683 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.196:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb --webserver_interface=127.5.240.196 --webserver_port=0 --tserver_master_addrs=127.5.240.254:45994 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.196 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:41:13.573032 20683 tablet_server_main.cc:85] Initializing tablet server... I0703 01:41:13.574545 20683 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 190208us I0703 01:41:13.575129 20683 fs_manager.cc:261] Metadata directory not provided I0703 01:41:13.575191 20683 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:41:13.575811 20683 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:41:13.575876 20683 server_base.cc:433] Attempting to create new FS layout instead I0703 01:41:13.579946 20683 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/instance: uuid: "a1e2e6b86a1747a4ae340383599dcff4" format_stamp: "Formatted at 2018-07-03 08:41:13 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:13.580488 20683 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal/instance: uuid: "a1e2e6b86a1747a4ae340383599dcff4" format_stamp: "Formatted at 2018-07-03 08:41:13 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:13.588239 20683 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.004s sys 0.002s I0703 01:41:13.589453 20683 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:41:13.589870 20683 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:41:13.592988 20683 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0703 01:41:13.593036 20683 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal uuid: "a1e2e6b86a1747a4ae340383599dcff4" format_stamp: "Formatted at 2018-07-03 08:41:13 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:13.593161 20683 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:41:13.631361 20683 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:41:13.633627 20683 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:41:13.633935 20683 tablet_server_main.cc:90] Starting tablet server... I0703 01:41:13.718804 20683 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.196:36708 I0703 01:41:13.719202 20683 webserver.cc:175] Starting webserver on 127.5.240.196:0 I0703 01:41:13.719239 20683 webserver.cc:186] Document root disabled I0703 01:41:13.719943 20683 webserver.cc:313] Webserver started. Bound to: http://127.5.240.196:37197/ I0703 01:41:13.720472 20683 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb I0703 01:41:13.722506 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 20683 I0703 01:41:13.723390 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data --block_manager=log --rpc_bind_addresses=127.5.240.197:0 --local_ip_for_outbound_sockets=127.5.240.197 --webserver_interface=127.5.240.197 --webserver_port=0 --tserver_master_addrs=127.5.240.254:45994 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0703 01:41:13.732154 20683 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:41:13.743602 20802 heartbeater.cc:346] Connected to a master server at 127.5.240.254:45994 I0703 01:41:13.743784 20802 heartbeater.cc:426] Registering TS with master... I0703 01:41:13.744709 20802 heartbeater.cc:475] Master 127.5.240.254:45994 requested a full tablet report, sending... I0703 01:41:13.745889 20272 ts_manager.cc:98] Registered new tserver with Master: a1e2e6b86a1747a4ae340383599dcff4 (127.5.240.196:36708) I0703 01:41:13.747189 20272 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.196:49233 I0703 01:41:13.776317 20435 heartbeater.cc:467] Master 127.5.240.254:45994 was elected leader, sending a full tablet report... WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:41:13.888659 20804 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:41:13.888866 20804 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:41:13.890872 20804 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0703 01:41:13.890920 20804 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:41:13.891031 20804 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.197 I0703 01:41:13.897979 20804 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.197:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb --webserver_interface=127.5.240.197 --webserver_port=0 --tserver_master_addrs=127.5.240.254:45994 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.197 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:41:13.904673 20804 tablet_server_main.cc:85] Initializing tablet server... I0703 01:41:13.906229 20804 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 190208us I0703 01:41:13.906817 20804 fs_manager.cc:261] Metadata directory not provided I0703 01:41:13.906878 20804 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:41:13.907094 20804 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:41:13.907156 20804 server_base.cc:433] Attempting to create new FS layout instead I0703 01:41:13.911345 20804 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/instance: uuid: "406b32afe710485ba1ca267c6629542e" format_stamp: "Formatted at 2018-07-03 08:41:13 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:13.911903 20804 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal/instance: uuid: "406b32afe710485ba1ca267c6629542e" format_stamp: "Formatted at 2018-07-03 08:41:13 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:13.919414 20804 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.003s sys 0.003s I0703 01:41:13.920662 20804 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:41:13.921032 20804 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:41:13.924603 20804 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s I0703 01:41:13.924654 20804 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal uuid: "406b32afe710485ba1ca267c6629542e" format_stamp: "Formatted at 2018-07-03 08:41:13 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:13.924782 20804 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:41:13.953685 20804 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:41:13.955943 20804 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:41:13.956279 20804 tablet_server_main.cc:90] Starting tablet server... I0703 01:41:14.041887 20804 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.197:43025 I0703 01:41:14.042316 20804 webserver.cc:175] Starting webserver on 127.5.240.197:0 I0703 01:41:14.042361 20804 webserver.cc:186] Document root disabled I0703 01:41:14.043066 20804 webserver.cc:313] Webserver started. Bound to: http://127.5.240.197:52624/ I0703 01:41:14.043642 20804 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb I0703 01:41:14.047662 20804 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:41:14.051311 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 20804 I0703 01:41:14.058260 20924 heartbeater.cc:346] Connected to a master server at 127.5.240.254:45994 I0703 01:41:14.058398 20924 heartbeater.cc:426] Registering TS with master... I0703 01:41:14.059303 20924 heartbeater.cc:475] Master 127.5.240.254:45994 requested a full tablet report, sending... I0703 01:41:14.060292 20272 ts_manager.cc:98] Registered new tserver with Master: 406b32afe710485ba1ca267c6629542e (127.5.240.197:43025) I0703 01:41:14.061166 20272 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.197:39644 I0703 01:41:14.061471 6083 external_mini_cluster.cc:490] 5 TS(s) registered with all masters I0703 01:41:14.070958 20272 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.5.240.254:56629: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0703 01:41:14.076037 20558 heartbeater.cc:467] Master 127.5.240.254:45994 was elected leader, sending a full tablet report... I0703 01:41:14.099244 20495 tablet_service.cc:759] Processing CreateTablet for tablet 8058ebbaf0444a03a06a5cc261243a60 (table=TestTable [id=cca11239bdb84b6ca083a7a4f8892b07]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:41:14.100163 20495 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 8058ebbaf0444a03a06a5cc261243a60. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:41:14.099959 20617 tablet_service.cc:759] Processing CreateTablet for tablet 8058ebbaf0444a03a06a5cc261243a60 (table=TestTable [id=cca11239bdb84b6ca083a7a4f8892b07]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:41:14.100677 20617 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 8058ebbaf0444a03a06a5cc261243a60. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:41:14.102483 20739 tablet_service.cc:759] Processing CreateTablet for tablet 8058ebbaf0444a03a06a5cc261243a60 (table=TestTable [id=cca11239bdb84b6ca083a7a4f8892b07]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:41:14.103044 20739 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 8058ebbaf0444a03a06a5cc261243a60. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:41:14.113448 20739 ts_tablet_manager.cc:1173] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4: Registered tablet (data state: TABLET_DATA_READY) I0703 01:41:14.114254 20944 ts_tablet_manager.cc:1028] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4: Bootstrapping tablet I0703 01:41:14.114300 20495 ts_tablet_manager.cc:1173] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466: Registered tablet (data state: TABLET_DATA_READY) I0703 01:41:14.114456 20617 ts_tablet_manager.cc:1173] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f: Registered tablet (data state: TABLET_DATA_READY) I0703 01:41:14.114941 20944 tablet_bootstrap.cc:427] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4: Bootstrap starting. I0703 01:41:14.117527 20944 tablet_bootstrap.cc:578] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4: No blocks or log segments found. Creating new log. I0703 01:41:14.118512 20944 log.cc:525] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4: Log is configured to *not* fsync() on all Append() calls I0703 01:41:14.119369 20945 ts_tablet_manager.cc:1028] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466: Bootstrapping tablet I0703 01:41:14.119649 20945 tablet_bootstrap.cc:427] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466: Bootstrap starting. I0703 01:41:14.119868 20946 ts_tablet_manager.cc:1028] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f: Bootstrapping tablet I0703 01:41:14.120189 20946 tablet_bootstrap.cc:427] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f: Bootstrap starting. I0703 01:41:14.121702 20944 tablet_bootstrap.cc:427] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4: No bootstrap required, opened a new log I0703 01:41:14.122004 20944 ts_tablet_manager.cc:1045] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.004s I0703 01:41:14.122483 20945 tablet_bootstrap.cc:578] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466: No blocks or log segments found. Creating new log. I0703 01:41:14.123497 20945 log.cc:525] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466: Log is configured to *not* fsync() on all Append() calls I0703 01:41:14.123920 20946 tablet_bootstrap.cc:578] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f: No blocks or log segments found. Creating new log. I0703 01:41:14.125063 20946 log.cc:525] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f: Log is configured to *not* fsync() on all Append() calls I0703 01:41:14.126139 20945 tablet_bootstrap.cc:427] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466: No bootstrap required, opened a new log I0703 01:41:14.126312 20945 ts_tablet_manager.cc:1045] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.001s I0703 01:41:14.128366 20944 raft_consensus.cc:325] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } } peers { permanent_uuid: "a1e2e6b86a1747a4ae340383599dcff4" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 36708 } } peers { permanent_uuid: "8c3e06132b5b44fb8eb4fc451662d466" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 57030 } } I0703 01:41:14.128677 20944 raft_consensus.cc:351] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:41:14.128726 20944 raft_consensus.cc:628] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1e2e6b86a1747a4ae340383599dcff4, State: Initialized, Role: FOLLOWER I0703 01:41:14.129112 20944 consensus_queue.cc:227] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } } peers { permanent_uuid: "a1e2e6b86a1747a4ae340383599dcff4" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 36708 } } peers { permanent_uuid: "8c3e06132b5b44fb8eb4fc451662d466" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 57030 } } I0703 01:41:14.129940 20802 heartbeater.cc:467] Master 127.5.240.254:45994 was elected leader, sending a full tablet report... I0703 01:41:14.130901 20944 ts_tablet_manager.cc:1073] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4: Time spent starting tablet: real 0.009s user 0.007s sys 0.002s I0703 01:41:14.135342 20946 tablet_bootstrap.cc:427] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f: No bootstrap required, opened a new log I0703 01:41:14.135056 20945 raft_consensus.cc:325] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } } peers { permanent_uuid: "a1e2e6b86a1747a4ae340383599dcff4" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 36708 } } peers { permanent_uuid: "8c3e06132b5b44fb8eb4fc451662d466" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 57030 } } I0703 01:41:14.135426 20945 raft_consensus.cc:351] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:41:14.135498 20945 raft_consensus.cc:628] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c3e06132b5b44fb8eb4fc451662d466, State: Initialized, Role: FOLLOWER I0703 01:41:14.135577 20946 ts_tablet_manager.cc:1045] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.011s I0703 01:41:14.135988 20945 consensus_queue.cc:227] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } } peers { permanent_uuid: "a1e2e6b86a1747a4ae340383599dcff4" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 36708 } } peers { permanent_uuid: "8c3e06132b5b44fb8eb4fc451662d466" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 57030 } } I0703 01:41:14.137331 20945 ts_tablet_manager.cc:1073] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466: Time spent starting tablet: real 0.011s user 0.005s sys 0.005s I0703 01:41:14.142204 20946 raft_consensus.cc:325] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } } peers { permanent_uuid: "a1e2e6b86a1747a4ae340383599dcff4" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 36708 } } peers { permanent_uuid: "8c3e06132b5b44fb8eb4fc451662d466" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 57030 } } I0703 01:41:14.142458 20946 raft_consensus.cc:351] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:41:14.142536 20946 raft_consensus.cc:628] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7efcfaf2a624349a32682447014e68f, State: Initialized, Role: FOLLOWER I0703 01:41:14.142853 20946 consensus_queue.cc:227] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } } peers { permanent_uuid: "a1e2e6b86a1747a4ae340383599dcff4" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 36708 } } peers { permanent_uuid: "8c3e06132b5b44fb8eb4fc451662d466" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 57030 } } I0703 01:41:14.143604 20680 heartbeater.cc:467] Master 127.5.240.254:45994 was elected leader, sending a full tablet report... I0703 01:41:14.144124 20946 ts_tablet_manager.cc:1073] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s W0703 01:41:14.149011 20681 tablet.cc:1678] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f: Can't schedule compaction. Clean time has not been advanced past its initial value. W0703 01:41:14.231775 20803 tablet.cc:1678] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4: Can't schedule compaction. Clean time has not been advanced past its initial value. W0703 01:41:14.309653 20559 tablet.cc:1678] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:41:14.404587 20951 raft_consensus.cc:457] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0703 01:41:14.404697 20951 raft_consensus.cc:2730] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466: Snoozing failure detection for 1.532s (starting election) I0703 01:41:14.404796 20951 raft_consensus.cc:479] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } } peers { permanent_uuid: "a1e2e6b86a1747a4ae340383599dcff4" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 36708 } } peers { permanent_uuid: "8c3e06132b5b44fb8eb4fc451662d466" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 57030 } } I0703 01:41:14.405963 20951 leader_election.cc:243] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer d7efcfaf2a624349a32682447014e68f (127.5.240.195:41493) I0703 01:41:14.406301 20951 leader_election.cc:243] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer a1e2e6b86a1747a4ae340383599dcff4 (127.5.240.196:36708) I0703 01:41:14.415707 20759 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "8058ebbaf0444a03a06a5cc261243a60" candidate_uuid: "8c3e06132b5b44fb8eb4fc451662d466" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1e2e6b86a1747a4ae340383599dcff4" is_pre_election: true I0703 01:41:14.415930 20759 raft_consensus.cc:2730] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4: Snoozing failure detection for 1.532s (vote granted) I0703 01:41:14.416013 20759 raft_consensus.cc:2243] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c3e06132b5b44fb8eb4fc451662d466 in term 0. I0703 01:41:14.416632 20446 leader_election.cc:379] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 [CANDIDATE]: Term 1 pre-election: Vote granted by peer a1e2e6b86a1747a4ae340383599dcff4 (127.5.240.196:36708) I0703 01:41:14.416693 20446 leader_election.cc:270] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0703 01:41:14.416270 20637 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "8058ebbaf0444a03a06a5cc261243a60" candidate_uuid: "8c3e06132b5b44fb8eb4fc451662d466" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7efcfaf2a624349a32682447014e68f" is_pre_election: true I0703 01:41:14.416934 20637 raft_consensus.cc:2730] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f: Snoozing failure detection for 1.513s (vote granted) I0703 01:41:14.416965 20951 raft_consensus.cc:2730] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466: Snoozing failure detection for 1.888s (election complete) I0703 01:41:14.417021 20951 raft_consensus.cc:2537] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0703 01:41:14.417024 20637 raft_consensus.cc:2243] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c3e06132b5b44fb8eb4fc451662d466 in term 0. I0703 01:41:14.417071 20951 raft_consensus.cc:457] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0703 01:41:14.417109 20951 raft_consensus.cc:2730] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466: Snoozing failure detection for 2.219s (starting election) I0703 01:41:14.417151 20951 raft_consensus.cc:2782] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:41:14.417649 20448 leader_election.cc:379] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 [CANDIDATE]: Term 1 pre-election: Vote granted by peer d7efcfaf2a624349a32682447014e68f (127.5.240.195:41493) I0703 01:41:14.418797 20951 raft_consensus.cc:479] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } } peers { permanent_uuid: "a1e2e6b86a1747a4ae340383599dcff4" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 36708 } } peers { permanent_uuid: "8c3e06132b5b44fb8eb4fc451662d466" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 57030 } } I0703 01:41:14.419117 20951 leader_election.cc:243] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 [CANDIDATE]: Term 1 election: Requesting vote from peer d7efcfaf2a624349a32682447014e68f (127.5.240.195:41493) I0703 01:41:14.419263 20951 leader_election.cc:243] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 [CANDIDATE]: Term 1 election: Requesting vote from peer a1e2e6b86a1747a4ae340383599dcff4 (127.5.240.196:36708) I0703 01:41:14.419735 20637 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "8058ebbaf0444a03a06a5cc261243a60" candidate_uuid: "8c3e06132b5b44fb8eb4fc451662d466" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7efcfaf2a624349a32682447014e68f" I0703 01:41:14.419847 20637 raft_consensus.cc:2782] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f [term 0 FOLLOWER]: Advancing to term 1 I0703 01:41:14.419862 20759 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "8058ebbaf0444a03a06a5cc261243a60" candidate_uuid: "8c3e06132b5b44fb8eb4fc451662d466" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1e2e6b86a1747a4ae340383599dcff4" I0703 01:41:14.419926 20637 raft_consensus.cc:2730] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f: Snoozing failure detection for 2.099s (vote granted) I0703 01:41:14.419972 20759 raft_consensus.cc:2782] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:41:14.420037 20759 raft_consensus.cc:2730] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4: Snoozing failure detection for 2.175s (vote granted) I0703 01:41:14.421259 20637 raft_consensus.cc:2243] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c3e06132b5b44fb8eb4fc451662d466 in term 1. I0703 01:41:14.421401 20759 raft_consensus.cc:2243] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c3e06132b5b44fb8eb4fc451662d466 in term 1. I0703 01:41:14.421653 20448 leader_election.cc:379] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 [CANDIDATE]: Term 1 election: Vote granted by peer d7efcfaf2a624349a32682447014e68f (127.5.240.195:41493) I0703 01:41:14.421756 20448 leader_election.cc:270] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:41:14.421955 20446 leader_election.cc:379] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 [CANDIDATE]: Term 1 election: Vote granted by peer a1e2e6b86a1747a4ae340383599dcff4 (127.5.240.196:36708) I0703 01:41:14.421957 20951 raft_consensus.cc:2730] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466: Snoozing failure detection for 2.123s (election complete) I0703 01:41:14.422076 20951 raft_consensus.cc:2537] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:41:14.422456 20951 raft_consensus.cc:593] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 [term 1 LEADER]: Becoming Leader. State: Replica: 8c3e06132b5b44fb8eb4fc451662d466, State: Running, Role: LEADER I0703 01:41:14.422828 20951 consensus_queue.cc:204] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } } peers { permanent_uuid: "a1e2e6b86a1747a4ae340383599dcff4" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 36708 } } peers { permanent_uuid: "8c3e06132b5b44fb8eb4fc451662d466" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 57030 } } I0703 01:41:14.427425 20271 catalog_manager.cc:3798] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 reported cstate change: term changed from 0 to 1, leader changed from to 8c3e06132b5b44fb8eb4fc451662d466 (127.5.240.194). New cstate: current_term: 1 leader_uuid: "8c3e06132b5b44fb8eb4fc451662d466" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1e2e6b86a1747a4ae340383599dcff4" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 36708 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c3e06132b5b44fb8eb4fc451662d466" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 57030 } health_report { overall_health: HEALTHY } } } I0703 01:41:14.441931 6083 external_mini_cluster.cc:490] 5 TS(s) registered with all masters I0703 01:41:14.443075 6083 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 86055af88ab04036878f3822fe571c07 to finish bootstrapping I0703 01:41:14.449530 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 8c3e06132b5b44fb8eb4fc451662d466 to finish bootstrapping I0703 01:41:14.454277 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver d7efcfaf2a624349a32682447014e68f to finish bootstrapping I0703 01:41:14.458811 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver a1e2e6b86a1747a4ae340383599dcff4 to finish bootstrapping I0703 01:41:14.464025 6083 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 406b32afe710485ba1ca267c6629542e to finish bootstrapping I0703 01:41:14.469568 6083 test_util.cc:208] Using random seed: 1189259443 I0703 01:41:14.477721 6083 test_workload.cc:306] TestWorkload: Skipping table creation because table TestTable already exists I0703 01:41:14.497980 20637 raft_consensus.cc:1081] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f [term 1 FOLLOWER]: Refusing update from remote peer 8c3e06132b5b44fb8eb4fc451662d466: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0703 01:41:14.498054 20759 raft_consensus.cc:1081] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [term 1 FOLLOWER]: Refusing update from remote peer 8c3e06132b5b44fb8eb4fc451662d466: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0703 01:41:14.499007 20955 consensus_queue.cc:962] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0703 01:41:14.499270 20951 consensus_queue.cc:962] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1e2e6b86a1747a4ae340383599dcff4" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 36708 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s W0703 01:41:14.741998 20965 flags.cc:406] Enabled unsafe flag: --never_fsync=true I0703 01:41:14.949321 20515 consensus_peers.cc:455] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 -> Peer a1e2e6b86a1747a4ae340383599dcff4 (127.5.240.196:36708): Closing peer: a1e2e6b86a1747a4ae340383599dcff4 I0703 01:41:14.949395 20515 consensus_peers.cc:455] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 -> Peer d7efcfaf2a624349a32682447014e68f (127.5.240.195:41493): Closing peer: d7efcfaf2a624349a32682447014e68f I0703 01:41:14.949635 20515 consensus_queue.cc:204] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 77, Committed index: 77, Last appended: 1.77, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 78 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } } peers { permanent_uuid: "a1e2e6b86a1747a4ae340383599dcff4" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 36708 } } peers { permanent_uuid: "8c3e06132b5b44fb8eb4fc451662d466" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 57030 } attrs { replace: true } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: NON_VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: true } } I0703 01:41:14.952086 20759 raft_consensus.cc:1081] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [term 1 FOLLOWER]: Refusing update from remote peer 8c3e06132b5b44fb8eb4fc451662d466: Log matching property violated. Preceding OpId in replica: term: 1 index: 77. Preceding OpId from leader: term: 1 index: 78. (index mismatch) I0703 01:41:14.952332 20637 raft_consensus.cc:1081] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f [term 1 FOLLOWER]: Refusing update from remote peer 8c3e06132b5b44fb8eb4fc451662d466: Log matching property violated. Preceding OpId in replica: term: 1 index: 77. Preceding OpId from leader: term: 1 index: 78. (index mismatch) I0703 01:41:14.952626 20951 consensus_queue.cc:962] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1e2e6b86a1747a4ae340383599dcff4" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 36708 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 78, Last known committed idx: 77, Time since last communication: 0.000s I0703 01:41:14.953507 20951 consensus_queue.cc:962] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 78, Last known committed idx: 77, Time since last communication: 0.000s I0703 01:41:14.955818 20951 raft_consensus.cc:2684] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 [term 1 LEADER]: Committing config change with OpId 1.78: config changed from index -1 to 78, NON_VOTER 406b32afe710485ba1ca267c6629542e (127.5.240.197) added. New config: { opid_index: 78 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } } peers { permanent_uuid: "a1e2e6b86a1747a4ae340383599dcff4" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 36708 } } peers { permanent_uuid: "8c3e06132b5b44fb8eb4fc451662d466" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 57030 } attrs { replace: true } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: NON_VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: true } } } I0703 01:41:14.957305 20637 raft_consensus.cc:2684] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f [term 1 FOLLOWER]: Committing config change with OpId 1.78: config changed from index -1 to 78, NON_VOTER 406b32afe710485ba1ca267c6629542e (127.5.240.197) added. New config: { opid_index: 78 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } } peers { permanent_uuid: "a1e2e6b86a1747a4ae340383599dcff4" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 36708 } } peers { permanent_uuid: "8c3e06132b5b44fb8eb4fc451662d466" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 57030 } attrs { replace: true } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: NON_VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: true } } } I0703 01:41:14.960093 20759 raft_consensus.cc:2684] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [term 1 FOLLOWER]: Committing config change with OpId 1.78: config changed from index -1 to 78, NON_VOTER 406b32afe710485ba1ca267c6629542e (127.5.240.197) added. New config: { opid_index: 78 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } } peers { permanent_uuid: "a1e2e6b86a1747a4ae340383599dcff4" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 36708 } } peers { permanent_uuid: "8c3e06132b5b44fb8eb4fc451662d466" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 57030 } attrs { replace: true } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: NON_VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: true } } } I0703 01:41:14.962100 20272 catalog_manager.cc:3798] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 reported cstate change: config changed from index -1 to 78, NON_VOTER 406b32afe710485ba1ca267c6629542e (127.5.240.197) added. New cstate: current_term: 1 leader_uuid: "8c3e06132b5b44fb8eb4fc451662d466" committed_config { opid_index: 78 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1e2e6b86a1747a4ae340383599dcff4" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 36708 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c3e06132b5b44fb8eb4fc451662d466" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 57030 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: NON_VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } W0703 01:41:14.962843 20448 consensus_peers.cc:433] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 -> Peer 406b32afe710485ba1ca267c6629542e (127.5.240.197:43025): Couldn't send request to peer 406b32afe710485ba1ca267c6629542e for tablet 8058ebbaf0444a03a06a5cc261243a60. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8058ebbaf0444a03a06a5cc261243a60. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:41:15.063122 20924 heartbeater.cc:467] Master 127.5.240.254:45994 was elected leader, sending a full tablet report... I0703 01:41:15.453279 21017 ts_tablet_manager.cc:669] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e: Initiating tablet copy from peer 8c3e06132b5b44fb8eb4fc451662d466 (127.5.240.194:57030) I0703 01:41:15.454254 21017 tablet_copy_client.cc:239] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.194:57030 I0703 01:41:15.464045 20535 tablet_copy_service.cc:135] P 8c3e06132b5b44fb8eb4fc451662d466: Received BeginTabletCopySession request for tablet 8058ebbaf0444a03a06a5cc261243a60 from peer 406b32afe710485ba1ca267c6629542e ({username='jenkins'} at 127.5.240.197:40303) I0703 01:41:15.464272 20535 tablet_copy_service.cc:156] P 8c3e06132b5b44fb8eb4fc451662d466: Beginning new tablet copy session on tablet 8058ebbaf0444a03a06a5cc261243a60 from peer 406b32afe710485ba1ca267c6629542e at {username='jenkins'} at 127.5.240.197:40303: session id = 406b32afe710485ba1ca267c6629542e-8058ebbaf0444a03a06a5cc261243a60 I0703 01:41:15.468298 20535 tablet_copy_source_session.cc:202] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:41:15.471880 21017 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 8058ebbaf0444a03a06a5cc261243a60. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:41:15.485157 21017 ts_tablet_manager.cc:1173] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:41:15.485538 21017 tablet_copy_client.cc:566] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e: tablet copy: Starting download of 0 data blocks... I0703 01:41:15.485855 21017 tablet_copy_client.cc:529] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e: tablet copy: Starting download of 1 WAL segments... I0703 01:41:15.491806 21017 tablet_copy_client.cc:405] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:41:15.494192 21017 ts_tablet_manager.cc:1028] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e: Bootstrapping tablet I0703 01:41:15.494411 21017 tablet_bootstrap.cc:427] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e: Bootstrap starting. I0703 01:41:15.527945 21017 log.cc:525] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e: Log is configured to *not* fsync() on all Append() calls I0703 01:41:15.689790 21017 tablet_bootstrap.cc:427] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e: Bootstrap replayed 1/1 log segments. Stats: ops{read=190 overwritten=0 applied=189 ignored=0} inserts{seen=187 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates I0703 01:41:15.690258 21017 tablet_bootstrap.cc:427] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e: Bootstrap complete. I0703 01:41:15.690574 21017 ts_tablet_manager.cc:1045] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e: Time spent bootstrapping tablet: real 0.196s user 0.164s sys 0.015s I0703 01:41:15.695416 21017 raft_consensus.cc:325] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [term 1 LEARNER]: Replica starting. Triggering 1 pending transactions. Active config: opid_index: 78 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } } peers { permanent_uuid: "a1e2e6b86a1747a4ae340383599dcff4" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 36708 } } peers { permanent_uuid: "8c3e06132b5b44fb8eb4fc451662d466" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 57030 } attrs { replace: true } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: NON_VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: true } } I0703 01:41:15.697989 21017 raft_consensus.cc:628] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 406b32afe710485ba1ca267c6629542e, State: Initialized, Role: LEARNER I0703 01:41:15.698436 21017 consensus_queue.cc:227] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 189, Last appended: 1.190, Last appended by leader: 190, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 78 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } } peers { permanent_uuid: "a1e2e6b86a1747a4ae340383599dcff4" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 36708 } } peers { permanent_uuid: "8c3e06132b5b44fb8eb4fc451662d466" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 57030 } attrs { replace: true } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: NON_VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: true } } I0703 01:41:15.703060 21017 ts_tablet_manager.cc:1073] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s I0703 01:41:15.704028 20535 tablet_copy_service.cc:325] P 8c3e06132b5b44fb8eb4fc451662d466: Request end of tablet copy session 406b32afe710485ba1ca267c6629542e-8058ebbaf0444a03a06a5cc261243a60 received from {username='jenkins'} at 127.5.240.197:40303 I0703 01:41:15.704162 20535 tablet_copy_service.cc:413] P 8c3e06132b5b44fb8eb4fc451662d466: ending tablet copy session 406b32afe710485ba1ca267c6629542e-8058ebbaf0444a03a06a5cc261243a60 on tablet 8058ebbaf0444a03a06a5cc261243a60 with peer 406b32afe710485ba1ca267c6629542e I0703 01:41:15.907775 20881 raft_consensus.cc:1026] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.77->[1.78-1.276] Dedup: 1.190->[1.191-1.276] I0703 01:41:16.020730 20975 raft_consensus.cc:890] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466: attempting to promote NON_VOTER 406b32afe710485ba1ca267c6629542e to VOTER I0703 01:41:16.021464 20975 consensus_peers.cc:455] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 -> Peer 406b32afe710485ba1ca267c6629542e (127.5.240.197:43025): Closing peer: 406b32afe710485ba1ca267c6629542e I0703 01:41:16.021569 20975 consensus_peers.cc:455] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 -> Peer a1e2e6b86a1747a4ae340383599dcff4 (127.5.240.196:36708): Closing peer: a1e2e6b86a1747a4ae340383599dcff4 I0703 01:41:16.021657 20975 consensus_peers.cc:455] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 -> Peer d7efcfaf2a624349a32682447014e68f (127.5.240.195:41493): Closing peer: d7efcfaf2a624349a32682447014e68f I0703 01:41:16.021986 20975 consensus_queue.cc:204] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 296, Committed index: 296, Last appended: 1.297, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } } peers { permanent_uuid: "a1e2e6b86a1747a4ae340383599dcff4" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 36708 } } peers { permanent_uuid: "8c3e06132b5b44fb8eb4fc451662d466" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 57030 } attrs { replace: true } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false } } I0703 01:41:16.024370 20636 raft_consensus.cc:1081] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f [term 1 FOLLOWER]: Refusing update from remote peer 8c3e06132b5b44fb8eb4fc451662d466: Log matching property violated. Preceding OpId in replica: term: 1 index: 296. Preceding OpId from leader: term: 1 index: 298. (index mismatch) I0703 01:41:16.024924 20759 raft_consensus.cc:1081] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [term 1 FOLLOWER]: Refusing update from remote peer 8c3e06132b5b44fb8eb4fc451662d466: Log matching property violated. Preceding OpId in replica: term: 1 index: 297. Preceding OpId from leader: term: 1 index: 298. (index mismatch) I0703 01:41:16.025518 20951 consensus_queue.cc:962] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 298, Last known committed idx: 296, Time since last communication: 0.000s I0703 01:41:16.025971 20955 consensus_queue.cc:962] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1e2e6b86a1747a4ae340383599dcff4" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 36708 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 298, Last known committed idx: 296, Time since last communication: 0.000s I0703 01:41:16.026463 20880 raft_consensus.cc:1081] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [term 1 LEARNER]: Refusing update from remote peer 8c3e06132b5b44fb8eb4fc451662d466: Log matching property violated. Preceding OpId in replica: term: 1 index: 297. Preceding OpId from leader: term: 1 index: 298. (index mismatch) I0703 01:41:16.027318 20951 consensus_queue.cc:962] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 298, Last known committed idx: 296, Time since last communication: 0.000s I0703 01:41:16.028668 20955 raft_consensus.cc:2684] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 [term 1 LEADER]: Committing config change with OpId 1.298: config changed from index 78 to 298, 406b32afe710485ba1ca267c6629542e (127.5.240.197) changed from NON_VOTER to VOTER. New config: { opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } } peers { permanent_uuid: "a1e2e6b86a1747a4ae340383599dcff4" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 36708 } } peers { permanent_uuid: "8c3e06132b5b44fb8eb4fc451662d466" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 57030 } attrs { replace: true } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false } } } I0703 01:41:16.030529 20881 raft_consensus.cc:2684] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [term 1 FOLLOWER]: Committing config change with OpId 1.298: config changed from index 78 to 298, 406b32afe710485ba1ca267c6629542e (127.5.240.197) changed from NON_VOTER to VOTER. New config: { opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } } peers { permanent_uuid: "a1e2e6b86a1747a4ae340383599dcff4" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 36708 } } peers { permanent_uuid: "8c3e06132b5b44fb8eb4fc451662d466" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 57030 } attrs { replace: true } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false } } } I0703 01:41:16.031437 20759 raft_consensus.cc:2684] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [term 1 FOLLOWER]: Committing config change with OpId 1.298: config changed from index 78 to 298, 406b32afe710485ba1ca267c6629542e (127.5.240.197) changed from NON_VOTER to VOTER. New config: { opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } } peers { permanent_uuid: "a1e2e6b86a1747a4ae340383599dcff4" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 36708 } } peers { permanent_uuid: "8c3e06132b5b44fb8eb4fc451662d466" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 57030 } attrs { replace: true } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false } } } I0703 01:41:16.033761 20271 catalog_manager.cc:3798] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 reported cstate change: config changed from index 78 to 298, 406b32afe710485ba1ca267c6629542e (127.5.240.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8c3e06132b5b44fb8eb4fc451662d466" committed_config { opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1e2e6b86a1747a4ae340383599dcff4" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 36708 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c3e06132b5b44fb8eb4fc451662d466" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 57030 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I0703 01:41:16.034368 20636 raft_consensus.cc:2684] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f [term 1 FOLLOWER]: Committing config change with OpId 1.298: config changed from index 78 to 298, 406b32afe710485ba1ca267c6629542e (127.5.240.197) changed from NON_VOTER to VOTER. New config: { opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } } peers { permanent_uuid: "a1e2e6b86a1747a4ae340383599dcff4" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 36708 } } peers { permanent_uuid: "8c3e06132b5b44fb8eb4fc451662d466" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 57030 } attrs { replace: true } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false } } } I0703 01:41:16.088321 20515 tablet_service.cc:1140] Received LeaderStepDown RPC: tablet_id: "8058ebbaf0444a03a06a5cc261243a60" dest_uuid: "8c3e06132b5b44fb8eb4fc451662d466" from {username='jenkins'} at 127.5.240.194:44732 I0703 01:41:16.088488 20515 raft_consensus.cc:552] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 [term 1 LEADER]: Received request to step down I0703 01:41:16.088537 20515 raft_consensus.cc:628] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8c3e06132b5b44fb8eb4fc451662d466, State: Running, Role: LEADER I0703 01:41:16.088855 20515 consensus_queue.cc:227] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 306, Committed index: 306, Last appended: 1.306, Last appended by leader: 306, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } } peers { permanent_uuid: "a1e2e6b86a1747a4ae340383599dcff4" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 36708 } } peers { permanent_uuid: "8c3e06132b5b44fb8eb4fc451662d466" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 57030 } attrs { replace: true } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false } } I0703 01:41:16.089169 20515 consensus_peers.cc:455] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 -> Peer 406b32afe710485ba1ca267c6629542e (127.5.240.197:43025): Closing peer: 406b32afe710485ba1ca267c6629542e I0703 01:41:16.089277 20515 consensus_peers.cc:455] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 -> Peer a1e2e6b86a1747a4ae340383599dcff4 (127.5.240.196:36708): Closing peer: a1e2e6b86a1747a4ae340383599dcff4 I0703 01:41:16.089339 20515 consensus_peers.cc:455] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 -> Peer d7efcfaf2a624349a32682447014e68f (127.5.240.195:41493): Closing peer: d7efcfaf2a624349a32682447014e68f I0703 01:41:16.089902 20515 raft_consensus.cc:2730] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466: Snoozing failure detection for 3.000s (explicit stepdown request) I0703 01:41:17.589404 21068 raft_consensus.cc:457] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c3e06132b5b44fb8eb4fc451662d466) I0703 01:41:17.589485 21068 raft_consensus.cc:2730] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4: Snoozing failure detection for 1.927s (starting election) I0703 01:41:17.589573 21069 raft_consensus.cc:457] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c3e06132b5b44fb8eb4fc451662d466) I0703 01:41:17.589674 21069 raft_consensus.cc:2730] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e: Snoozing failure detection for 1.806s (starting election) I0703 01:41:17.589588 21068 raft_consensus.cc:479] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } } peers { permanent_uuid: "a1e2e6b86a1747a4ae340383599dcff4" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 36708 } } peers { permanent_uuid: "8c3e06132b5b44fb8eb4fc451662d466" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 57030 } attrs { replace: true } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false } } I0703 01:41:17.589777 21069 raft_consensus.cc:479] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } } peers { permanent_uuid: "a1e2e6b86a1747a4ae340383599dcff4" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 36708 } } peers { permanent_uuid: "8c3e06132b5b44fb8eb4fc451662d466" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 57030 } attrs { replace: true } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false } } I0703 01:41:17.590376 21068 leader_election.cc:243] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer d7efcfaf2a624349a32682447014e68f (127.5.240.195:41493) I0703 01:41:17.590502 21069 leader_election.cc:243] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer d7efcfaf2a624349a32682447014e68f (127.5.240.195:41493) I0703 01:41:17.590641 21068 leader_election.cc:243] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 8c3e06132b5b44fb8eb4fc451662d466 (127.5.240.194:57030) I0703 01:41:17.590749 21068 leader_election.cc:243] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 406b32afe710485ba1ca267c6629542e (127.5.240.197:43025) I0703 01:41:17.595657 21069 leader_election.cc:243] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer a1e2e6b86a1747a4ae340383599dcff4 (127.5.240.196:36708) I0703 01:41:17.598376 21074 raft_consensus.cc:457] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c3e06132b5b44fb8eb4fc451662d466) I0703 01:41:17.598485 21074 raft_consensus.cc:2730] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f: Snoozing failure detection for 1.513s (starting election) I0703 01:41:17.598686 21074 raft_consensus.cc:479] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } } peers { permanent_uuid: "a1e2e6b86a1747a4ae340383599dcff4" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 36708 } } peers { permanent_uuid: "8c3e06132b5b44fb8eb4fc451662d466" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 57030 } attrs { replace: true } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false } } I0703 01:41:17.599581 21074 leader_election.cc:243] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer a1e2e6b86a1747a4ae340383599dcff4 (127.5.240.196:36708) I0703 01:41:17.603410 21069 leader_election.cc:243] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 8c3e06132b5b44fb8eb4fc451662d466 (127.5.240.194:57030) I0703 01:41:17.604892 21074 leader_election.cc:243] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 8c3e06132b5b44fb8eb4fc451662d466 (127.5.240.194:57030) I0703 01:41:17.605109 21074 leader_election.cc:243] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 406b32afe710485ba1ca267c6629542e (127.5.240.197:43025) I0703 01:41:17.615080 20515 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "8058ebbaf0444a03a06a5cc261243a60" candidate_uuid: "406b32afe710485ba1ca267c6629542e" candidate_term: 2 candidate_status { last_received { term: 1 index: 306 } } ignore_live_leader: false dest_uuid: "8c3e06132b5b44fb8eb4fc451662d466" is_pre_election: true I0703 01:41:17.615300 20515 raft_consensus.cc:2730] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466: Snoozing failure detection for 1.981s (vote granted) I0703 01:41:17.615375 20515 raft_consensus.cc:2243] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 406b32afe710485ba1ca267c6629542e in term 1. I0703 01:41:17.616170 20814 leader_election.cc:379] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [CANDIDATE]: Term 2 pre-election: Vote granted by peer 8c3e06132b5b44fb8eb4fc451662d466 (127.5.240.194:57030) I0703 01:41:17.616719 20515 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "8058ebbaf0444a03a06a5cc261243a60" candidate_uuid: "a1e2e6b86a1747a4ae340383599dcff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 306 } } ignore_live_leader: false dest_uuid: "8c3e06132b5b44fb8eb4fc451662d466" is_pre_election: true I0703 01:41:17.616895 20515 raft_consensus.cc:2730] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466: Snoozing failure detection for 1.504s (vote granted) I0703 01:41:17.616976 20515 raft_consensus.cc:2243] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1e2e6b86a1747a4ae340383599dcff4 in term 1. I0703 01:41:17.618209 20881 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "8058ebbaf0444a03a06a5cc261243a60" candidate_uuid: "a1e2e6b86a1747a4ae340383599dcff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 306 } } ignore_live_leader: false dest_uuid: "406b32afe710485ba1ca267c6629542e" is_pre_election: true I0703 01:41:17.618412 20881 raft_consensus.cc:2730] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e: Snoozing failure detection for 1.532s (vote granted) I0703 01:41:17.618489 20881 raft_consensus.cc:2243] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1e2e6b86a1747a4ae340383599dcff4 in term 1. I0703 01:41:17.619942 20692 leader_election.cc:379] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [CANDIDATE]: Term 2 pre-election: Vote granted by peer 8c3e06132b5b44fb8eb4fc451662d466 (127.5.240.194:57030) I0703 01:41:17.620209 20692 leader_election.cc:379] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [CANDIDATE]: Term 2 pre-election: Vote granted by peer 406b32afe710485ba1ca267c6629542e (127.5.240.197:43025) I0703 01:41:17.620357 20692 leader_election.cc:270] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. I0703 01:41:17.621013 21068 raft_consensus.cc:2730] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4: Snoozing failure detection for 1.539s (election complete) I0703 01:41:17.621083 21068 raft_consensus.cc:2537] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0703 01:41:17.621176 21068 raft_consensus.cc:457] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8c3e06132b5b44fb8eb4fc451662d466) I0703 01:41:17.621228 21068 raft_consensus.cc:2730] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4: Snoozing failure detection for 2.034s (starting election) I0703 01:41:17.621342 21068 raft_consensus.cc:2782] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [term 1 FOLLOWER]: Advancing to term 2 I0703 01:41:17.623683 21068 raft_consensus.cc:479] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } } peers { permanent_uuid: "a1e2e6b86a1747a4ae340383599dcff4" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 36708 } } peers { permanent_uuid: "8c3e06132b5b44fb8eb4fc451662d466" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 57030 } attrs { replace: true } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false } } I0703 01:41:17.624347 21068 leader_election.cc:243] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [CANDIDATE]: Term 2 election: Requesting vote from peer d7efcfaf2a624349a32682447014e68f (127.5.240.195:41493) I0703 01:41:17.624649 21068 leader_election.cc:243] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [CANDIDATE]: Term 2 election: Requesting vote from peer 8c3e06132b5b44fb8eb4fc451662d466 (127.5.240.194:57030) I0703 01:41:17.625023 21068 leader_election.cc:243] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [CANDIDATE]: Term 2 election: Requesting vote from peer 406b32afe710485ba1ca267c6629542e (127.5.240.197:43025) I0703 01:41:17.626655 20515 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "8058ebbaf0444a03a06a5cc261243a60" candidate_uuid: "a1e2e6b86a1747a4ae340383599dcff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 306 } } ignore_live_leader: false dest_uuid: "8c3e06132b5b44fb8eb4fc451662d466" I0703 01:41:17.626788 20515 raft_consensus.cc:2782] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 [term 1 FOLLOWER]: Advancing to term 2 I0703 01:41:17.626871 20515 raft_consensus.cc:2730] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466: Snoozing failure detection for 2.155s (vote granted) I0703 01:41:17.627027 20881 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "8058ebbaf0444a03a06a5cc261243a60" candidate_uuid: "a1e2e6b86a1747a4ae340383599dcff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 306 } } ignore_live_leader: false dest_uuid: "406b32afe710485ba1ca267c6629542e" I0703 01:41:17.627171 20881 raft_consensus.cc:2782] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [term 1 FOLLOWER]: Advancing to term 2 I0703 01:41:17.627271 20881 raft_consensus.cc:2730] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e: Snoozing failure detection for 2.013s (vote granted) I0703 01:41:17.628959 20515 raft_consensus.cc:2243] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1e2e6b86a1747a4ae340383599dcff4 in term 2. I0703 01:41:17.629097 20759 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "8058ebbaf0444a03a06a5cc261243a60" candidate_uuid: "406b32afe710485ba1ca267c6629542e" candidate_term: 2 candidate_status { last_received { term: 1 index: 306 } } ignore_live_leader: false dest_uuid: "a1e2e6b86a1747a4ae340383599dcff4" is_pre_election: true I0703 01:41:17.629287 20881 raft_consensus.cc:2243] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1e2e6b86a1747a4ae340383599dcff4 in term 2. I0703 01:41:17.629326 20759 raft_consensus.cc:2173] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 406b32afe710485ba1ca267c6629542e in current term 2: Already voted for candidate a1e2e6b86a1747a4ae340383599dcff4 in this term. I0703 01:41:17.629545 20692 leader_election.cc:379] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [CANDIDATE]: Term 2 election: Vote granted by peer 8c3e06132b5b44fb8eb4fc451662d466 (127.5.240.194:57030) I0703 01:41:17.630010 20692 leader_election.cc:379] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [CANDIDATE]: Term 2 election: Vote granted by peer 406b32afe710485ba1ca267c6629542e (127.5.240.197:43025) I0703 01:41:17.630074 20692 leader_election.cc:270] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0703 01:41:17.630046 20812 leader_election.cc:393] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [CANDIDATE]: Term 2 pre-election: Vote denied by peer a1e2e6b86a1747a4ae340383599dcff4 (127.5.240.196:36708). Message: Invalid argument: T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 406b32afe710485ba1ca267c6629542e in current term 2: Already voted for candidate a1e2e6b86a1747a4ae340383599dcff4 in this term. I0703 01:41:17.635283 21068 raft_consensus.cc:2730] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4: Snoozing failure detection for 2.125s (election complete) I0703 01:41:17.636181 20636 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "8058ebbaf0444a03a06a5cc261243a60" candidate_uuid: "406b32afe710485ba1ca267c6629542e" candidate_term: 2 candidate_status { last_received { term: 1 index: 306 } } ignore_live_leader: false dest_uuid: "d7efcfaf2a624349a32682447014e68f" is_pre_election: true I0703 01:41:17.636370 20636 raft_consensus.cc:2730] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f: Snoozing failure detection for 1.830s (vote granted) I0703 01:41:17.636391 21068 raft_consensus.cc:2537] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [term 2 FOLLOWER]: Leader election won for term 2 I0703 01:41:17.636479 20636 raft_consensus.cc:2243] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 406b32afe710485ba1ca267c6629542e in term 1. I0703 01:41:17.637395 20759 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "8058ebbaf0444a03a06a5cc261243a60" candidate_uuid: "d7efcfaf2a624349a32682447014e68f" candidate_term: 2 candidate_status { last_received { term: 1 index: 306 } } ignore_live_leader: false dest_uuid: "a1e2e6b86a1747a4ae340383599dcff4" is_pre_election: true I0703 01:41:17.637984 20814 leader_election.cc:379] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [CANDIDATE]: Term 2 pre-election: Vote granted by peer d7efcfaf2a624349a32682447014e68f (127.5.240.195:41493) I0703 01:41:17.638048 20814 leader_election.cc:270] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. I0703 01:41:17.638309 20515 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "8058ebbaf0444a03a06a5cc261243a60" candidate_uuid: "d7efcfaf2a624349a32682447014e68f" candidate_term: 2 candidate_status { last_received { term: 1 index: 306 } } ignore_live_leader: false dest_uuid: "8c3e06132b5b44fb8eb4fc451662d466" is_pre_election: true I0703 01:41:17.638486 20515 raft_consensus.cc:2173] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7efcfaf2a624349a32682447014e68f in current term 2: Already voted for candidate a1e2e6b86a1747a4ae340383599dcff4 in this term. I0703 01:41:17.638577 21069 raft_consensus.cc:2730] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e: Snoozing failure detection for 1.563s (election complete) I0703 01:41:17.638681 21069 raft_consensus.cc:2500] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won I0703 01:41:17.639400 20636 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "8058ebbaf0444a03a06a5cc261243a60" candidate_uuid: "a1e2e6b86a1747a4ae340383599dcff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 306 } } ignore_live_leader: false dest_uuid: "d7efcfaf2a624349a32682447014e68f" is_pre_election: true I0703 01:41:17.639533 20636 raft_consensus.cc:2730] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f: Snoozing failure detection for 2.019s (vote granted) I0703 01:41:17.639587 20636 raft_consensus.cc:2243] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1e2e6b86a1747a4ae340383599dcff4 in term 1. I0703 01:41:17.639825 20881 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "8058ebbaf0444a03a06a5cc261243a60" candidate_uuid: "d7efcfaf2a624349a32682447014e68f" candidate_term: 2 candidate_status { last_received { term: 1 index: 306 } } ignore_live_leader: false dest_uuid: "406b32afe710485ba1ca267c6629542e" is_pre_election: true I0703 01:41:17.639855 21068 raft_consensus.cc:593] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [term 2 LEADER]: Becoming Leader. State: Replica: a1e2e6b86a1747a4ae340383599dcff4, State: Running, Role: LEADER I0703 01:41:17.640009 20881 raft_consensus.cc:2173] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7efcfaf2a624349a32682447014e68f in current term 2: Already voted for candidate a1e2e6b86a1747a4ae340383599dcff4 in this term. I0703 01:41:17.640082 20570 leader_election.cc:393] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f [CANDIDATE]: Term 2 pre-election: Vote denied by peer 8c3e06132b5b44fb8eb4fc451662d466 (127.5.240.194:57030). Message: Invalid argument: T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7efcfaf2a624349a32682447014e68f in current term 2: Already voted for candidate a1e2e6b86a1747a4ae340383599dcff4 in this term. I0703 01:41:17.640485 21068 consensus_queue.cc:204] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 306, Committed index: 306, Last appended: 1.306, Last appended by leader: 306, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } } peers { permanent_uuid: "a1e2e6b86a1747a4ae340383599dcff4" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 36708 } } peers { permanent_uuid: "8c3e06132b5b44fb8eb4fc451662d466" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 57030 } attrs { replace: true } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false } } I0703 01:41:17.640616 20636 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "8058ebbaf0444a03a06a5cc261243a60" candidate_uuid: "a1e2e6b86a1747a4ae340383599dcff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 306 } } ignore_live_leader: false dest_uuid: "d7efcfaf2a624349a32682447014e68f" I0703 01:41:17.640962 20692 leader_election.cc:379] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [CANDIDATE]: Term 2 pre-election: Vote granted by peer d7efcfaf2a624349a32682447014e68f (127.5.240.195:41493) I0703 01:41:17.640975 20636 raft_consensus.cc:2782] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f [term 1 FOLLOWER]: Advancing to term 2 I0703 01:41:17.641077 20636 raft_consensus.cc:2730] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f: Snoozing failure detection for 1.644s (vote granted) I0703 01:41:17.640975 20570 leader_election.cc:393] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f [CANDIDATE]: Term 2 pre-election: Vote denied by peer 406b32afe710485ba1ca267c6629542e (127.5.240.197:43025). Message: Invalid argument: T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7efcfaf2a624349a32682447014e68f in current term 2: Already voted for candidate a1e2e6b86a1747a4ae340383599dcff4 in this term. I0703 01:41:17.642433 20570 leader_election.cc:270] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0703 01:41:17.642791 20636 raft_consensus.cc:2243] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1e2e6b86a1747a4ae340383599dcff4 in term 2. I0703 01:41:17.642979 21074 raft_consensus.cc:2730] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f: Snoozing failure detection for 2.102s (election complete) I0703 01:41:17.643081 21074 raft_consensus.cc:2485] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0703 01:41:17.643597 20692 leader_election.cc:379] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [CANDIDATE]: Term 2 election: Vote granted by peer d7efcfaf2a624349a32682447014e68f (127.5.240.195:41493) I0703 01:41:17.644227 20759 raft_consensus.cc:2204] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate d7efcfaf2a624349a32682447014e68f for term 2 because replica is either leader or believes a valid leader to be alive. I0703 01:41:17.644922 20568 leader_election.cc:393] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f [CANDIDATE]: Term 2 pre-election: Vote denied by peer a1e2e6b86a1747a4ae340383599dcff4 (127.5.240.196:36708). Message: Invalid argument: T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate d7efcfaf2a624349a32682447014e68f for term 2 because replica is either leader or believes a valid leader to be alive. I0703 01:41:17.645267 20270 catalog_manager.cc:3798] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 reported cstate change: term changed from 1 to 2, leader changed from 8c3e06132b5b44fb8eb4fc451662d466 (127.5.240.194) to a1e2e6b86a1747a4ae340383599dcff4 (127.5.240.196). New cstate: current_term: 2 leader_uuid: "a1e2e6b86a1747a4ae340383599dcff4" committed_config { opid_index: 298 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1e2e6b86a1747a4ae340383599dcff4" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 36708 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c3e06132b5b44fb8eb4fc451662d466" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 57030 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false } health_report { overall_health: UNKNOWN } } } I0703 01:41:17.705188 20881 raft_consensus.cc:1081] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [term 2 FOLLOWER]: Refusing update from remote peer a1e2e6b86a1747a4ae340383599dcff4: Log matching property violated. Preceding OpId in replica: term: 1 index: 306. Preceding OpId from leader: term: 2 index: 308. (index mismatch) I0703 01:41:17.705269 20515 raft_consensus.cc:1081] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 [term 2 FOLLOWER]: Refusing update from remote peer a1e2e6b86a1747a4ae340383599dcff4: Log matching property violated. Preceding OpId in replica: term: 1 index: 306. Preceding OpId from leader: term: 2 index: 308. (index mismatch) I0703 01:41:17.705446 20636 raft_consensus.cc:1081] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f [term 2 FOLLOWER]: Refusing update from remote peer a1e2e6b86a1747a4ae340383599dcff4: Log matching property violated. Preceding OpId in replica: term: 1 index: 306. Preceding OpId from leader: term: 2 index: 308. (index mismatch) I0703 01:41:17.705950 21097 consensus_queue.cc:962] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 307, Last known committed idx: 306, Time since last communication: 0.000s I0703 01:41:17.706416 21090 consensus_queue.cc:962] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 307, Last known committed idx: 306, Time since last communication: 0.000s I0703 01:41:17.706784 21068 consensus_queue.cc:962] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c3e06132b5b44fb8eb4fc451662d466" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 57030 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 307, Last known committed idx: 306, Time since last communication: 0.000s I0703 01:41:17.718005 20270 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 8058ebbaf0444a03a06a5cc261243a60 (attempt 1) I0703 01:41:17.719769 20759 consensus_peers.cc:455] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 -> Peer 406b32afe710485ba1ca267c6629542e (127.5.240.197:43025): Closing peer: 406b32afe710485ba1ca267c6629542e I0703 01:41:17.719840 20759 consensus_peers.cc:455] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 -> Peer 8c3e06132b5b44fb8eb4fc451662d466 (127.5.240.194:57030): Closing peer: 8c3e06132b5b44fb8eb4fc451662d466 I0703 01:41:17.719883 20759 consensus_peers.cc:455] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 -> Peer d7efcfaf2a624349a32682447014e68f (127.5.240.195:41493): Closing peer: d7efcfaf2a624349a32682447014e68f I0703 01:41:17.720145 20759 consensus_queue.cc:204] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 308, Committed index: 308, Last appended: 2.308, Last appended by leader: 306, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 309 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } } peers { permanent_uuid: "a1e2e6b86a1747a4ae340383599dcff4" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 36708 } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false } } I0703 01:41:17.722824 20637 raft_consensus.cc:1081] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f [term 2 FOLLOWER]: Refusing update from remote peer a1e2e6b86a1747a4ae340383599dcff4: Log matching property violated. Preceding OpId in replica: term: 2 index: 308. Preceding OpId from leader: term: 2 index: 309. (index mismatch) I0703 01:41:17.723040 20881 raft_consensus.cc:1081] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [term 2 FOLLOWER]: Refusing update from remote peer a1e2e6b86a1747a4ae340383599dcff4: Log matching property violated. Preceding OpId in replica: term: 2 index: 308. Preceding OpId from leader: term: 2 index: 309. (index mismatch) I0703 01:41:17.724182 21090 consensus_queue.cc:962] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 309, Last known committed idx: 308, Time since last communication: 0.000s I0703 01:41:17.724360 21097 consensus_queue.cc:962] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 309, Last known committed idx: 308, Time since last communication: 0.000s I0703 01:41:17.726462 21090 raft_consensus.cc:2684] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [term 2 LEADER]: Committing config change with OpId 2.309: config changed from index 298 to 309, VOTER 8c3e06132b5b44fb8eb4fc451662d466 (127.5.240.194) evicted. New config: { opid_index: 309 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } } peers { permanent_uuid: "a1e2e6b86a1747a4ae340383599dcff4" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 36708 } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false } } } I0703 01:41:17.727036 20881 raft_consensus.cc:2684] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [term 2 FOLLOWER]: Committing config change with OpId 2.309: config changed from index 298 to 309, VOTER 8c3e06132b5b44fb8eb4fc451662d466 (127.5.240.194) evicted. New config: { opid_index: 309 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } } peers { permanent_uuid: "a1e2e6b86a1747a4ae340383599dcff4" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 36708 } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false } } } I0703 01:41:17.727972 20636 raft_consensus.cc:2684] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f [term 2 FOLLOWER]: Committing config change with OpId 2.309: config changed from index 298 to 309, VOTER 8c3e06132b5b44fb8eb4fc451662d466 (127.5.240.194) evicted. New config: { opid_index: 309 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } } peers { permanent_uuid: "a1e2e6b86a1747a4ae340383599dcff4" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 36708 } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false } } } I0703 01:41:17.728906 20253 catalog_manager.cc:3352] ChangeConfig:REMOVE_PEER RPC for tablet 8058ebbaf0444a03a06a5cc261243a60 with cas_config_opid_index 298: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0703 01:41:17.729869 20272 catalog_manager.cc:3798] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 reported cstate change: config changed from index 298 to 309, VOTER 8c3e06132b5b44fb8eb4fc451662d466 (127.5.240.194) evicted. New cstate: current_term: 2 leader_uuid: "a1e2e6b86a1747a4ae340383599dcff4" committed_config { opid_index: 309 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1e2e6b86a1747a4ae340383599dcff4" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 36708 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I0703 01:41:17.732959 20272 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 8058ebbaf0444a03a06a5cc261243a60 on 8c3e06132b5b44fb8eb4fc451662d466 (127.5.240.194:57030) (TS 8c3e06132b5b44fb8eb4fc451662d466 not found in new config with opid_index 309) I0703 01:41:17.733651 20495 tablet_service.cc:800] Processing DeleteTablet for tablet 8058ebbaf0444a03a06a5cc261243a60 with delete_type TABLET_DATA_TOMBSTONED (TS 8c3e06132b5b44fb8eb4fc451662d466 not found in new config with opid_index 309) from {username='jenkins'} at 127.5.240.194:44711 I0703 01:41:17.734905 21108 tablet_replica.cc:261] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466: stopping tablet replica I0703 01:41:17.735090 21108 maintenance_manager.cc:237] P 8c3e06132b5b44fb8eb4fc451662d466: Unregistered op CompactRowSetsOp(8058ebbaf0444a03a06a5cc261243a60) I0703 01:41:17.735208 21108 maintenance_manager.cc:237] P 8c3e06132b5b44fb8eb4fc451662d466: Unregistered op MinorDeltaCompactionOp(8058ebbaf0444a03a06a5cc261243a60) I0703 01:41:17.735292 21108 maintenance_manager.cc:237] P 8c3e06132b5b44fb8eb4fc451662d466: Unregistered op MajorDeltaCompactionOp(8058ebbaf0444a03a06a5cc261243a60) I0703 01:41:17.735378 21108 maintenance_manager.cc:237] P 8c3e06132b5b44fb8eb4fc451662d466: Unregistered op UndoDeltaBlockGCOp(8058ebbaf0444a03a06a5cc261243a60) I0703 01:41:17.735500 21108 maintenance_manager.cc:237] P 8c3e06132b5b44fb8eb4fc451662d466: Unregistered op FlushMRSOp(8058ebbaf0444a03a06a5cc261243a60) I0703 01:41:17.735602 21108 maintenance_manager.cc:237] P 8c3e06132b5b44fb8eb4fc451662d466: Unregistered op FlushDeltaMemStoresOp(8058ebbaf0444a03a06a5cc261243a60) I0703 01:41:17.735678 21108 maintenance_manager.cc:237] P 8c3e06132b5b44fb8eb4fc451662d466: Unregistered op LogGCOp(8058ebbaf0444a03a06a5cc261243a60) I0703 01:41:17.736482 21108 raft_consensus.cc:2040] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 [term 2 FOLLOWER]: Raft consensus shutting down. I0703 01:41:17.736737 21108 pending_rounds.cc:63] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466: Trying to abort 2 pending transactions. I0703 01:41:17.736820 21108 pending_rounds.cc:68] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466: Aborting transaction as it isn't in flight: id { term: 2 index: 307 } timestamp: 6269367409220935680 op_type: NO_OP noop_request { } I0703 01:41:17.737005 21108 raft_consensus.cc:2620] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Transaction aborted I0703 01:41:17.737123 21108 pending_rounds.cc:68] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466: Aborting transaction as it isn't in flight: id { term: 2 index: 308 } timestamp: 6269367409469513728 op_type: WRITE_OP write_request { tablet_id: "8058ebbaf0444a03a06a5cc261243a60" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true } } row_operations { rows: "\001\007\000\030]h*=\031\373h\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "5773d5dd6bbb5864" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 6269367402838704128 } request_id { client_id: "2499667f6bcc4c699e4f6bf5bd2dde19" seq_no: 303 first_incomplete_seq_no: 303 attempt_no: 51 } I0703 01:41:17.737720 21108 raft_consensus.cc:2067] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 [term 2 FOLLOWER]: Raft consensus is shut down! I0703 01:41:17.740270 21108 ts_tablet_manager.cc:1369] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:41:17.745317 21108 ts_tablet_manager.cc:1382] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.308 I0703 01:41:17.745410 21108 log.cc:986] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal/wals/8058ebbaf0444a03a06a5cc261243a60 I0703 01:41:17.746130 20255 catalog_manager.cc:3103] TS 8c3e06132b5b44fb8eb4fc451662d466 (127.5.240.194:57030): tablet 8058ebbaf0444a03a06a5cc261243a60 (table TestTable [id=cca11239bdb84b6ca083a7a4f8892b07]) successfully deleted W0703 01:41:18.474483 21118 flags.cc:406] Enabled unsafe flag: --never_fsync=true I0703 01:41:18.702781 20759 consensus_peers.cc:455] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 -> Peer 406b32afe710485ba1ca267c6629542e (127.5.240.197:43025): Closing peer: 406b32afe710485ba1ca267c6629542e I0703 01:41:18.702883 20759 consensus_peers.cc:455] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 -> Peer d7efcfaf2a624349a32682447014e68f (127.5.240.195:41493): Closing peer: d7efcfaf2a624349a32682447014e68f I0703 01:41:18.703109 20759 consensus_queue.cc:204] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 488, Committed index: 488, Last appended: 2.489, Last appended by leader: 306, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 490 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } } peers { permanent_uuid: "a1e2e6b86a1747a4ae340383599dcff4" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 36708 } attrs { replace: true } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false } } peers { permanent_uuid: "86055af88ab04036878f3822fe571c07" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 42232 } attrs { promote: true } } I0703 01:41:18.705775 20637 raft_consensus.cc:1081] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f [term 2 FOLLOWER]: Refusing update from remote peer a1e2e6b86a1747a4ae340383599dcff4: Log matching property violated. Preceding OpId in replica: term: 2 index: 489. Preceding OpId from leader: term: 2 index: 490. (index mismatch) I0703 01:41:18.706615 20881 raft_consensus.cc:1081] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [term 2 FOLLOWER]: Refusing update from remote peer a1e2e6b86a1747a4ae340383599dcff4: Log matching property violated. Preceding OpId in replica: term: 2 index: 489. Preceding OpId from leader: term: 2 index: 490. (index mismatch) I0703 01:41:18.706542 21097 consensus_queue.cc:962] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 490, Last known committed idx: 488, Time since last communication: 0.000s I0703 01:41:18.707446 21097 consensus_queue.cc:962] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 490, Last known committed idx: 488, Time since last communication: 0.000s I0703 01:41:18.708964 21090 raft_consensus.cc:2684] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [term 2 LEADER]: Committing config change with OpId 2.490: config changed from index 309 to 490, NON_VOTER 86055af88ab04036878f3822fe571c07 (127.5.240.193) added. New config: { opid_index: 490 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } } peers { permanent_uuid: "a1e2e6b86a1747a4ae340383599dcff4" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 36708 } attrs { replace: true } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false } } peers { permanent_uuid: "86055af88ab04036878f3822fe571c07" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 42232 } attrs { promote: true } } } I0703 01:41:18.711132 20637 raft_consensus.cc:2684] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f [term 2 FOLLOWER]: Committing config change with OpId 2.490: config changed from index 309 to 490, NON_VOTER 86055af88ab04036878f3822fe571c07 (127.5.240.193) added. New config: { opid_index: 490 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } } peers { permanent_uuid: "a1e2e6b86a1747a4ae340383599dcff4" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 36708 } attrs { replace: true } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false } } peers { permanent_uuid: "86055af88ab04036878f3822fe571c07" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 42232 } attrs { promote: true } } } W0703 01:41:18.711745 20691 consensus_peers.cc:433] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 -> Peer 86055af88ab04036878f3822fe571c07 (127.5.240.193:42232): Couldn't send request to peer 86055af88ab04036878f3822fe571c07 for tablet 8058ebbaf0444a03a06a5cc261243a60. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8058ebbaf0444a03a06a5cc261243a60. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:41:18.714741 20270 catalog_manager.cc:3798] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 reported cstate change: config changed from index 309 to 490, NON_VOTER 86055af88ab04036878f3822fe571c07 (127.5.240.193) added. New cstate: current_term: 2 leader_uuid: "a1e2e6b86a1747a4ae340383599dcff4" committed_config { opid_index: 490 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1e2e6b86a1747a4ae340383599dcff4" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 36708 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86055af88ab04036878f3822fe571c07" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 42232 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I0703 01:41:18.716261 20881 raft_consensus.cc:2684] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [term 2 FOLLOWER]: Committing config change with OpId 2.490: config changed from index 309 to 490, NON_VOTER 86055af88ab04036878f3822fe571c07 (127.5.240.193) added. New config: { opid_index: 490 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } } peers { permanent_uuid: "a1e2e6b86a1747a4ae340383599dcff4" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 36708 } attrs { replace: true } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false } } peers { permanent_uuid: "86055af88ab04036878f3822fe571c07" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 42232 } attrs { promote: true } } } I0703 01:41:19.115427 21167 ts_tablet_manager.cc:669] T 8058ebbaf0444a03a06a5cc261243a60 P 86055af88ab04036878f3822fe571c07: Initiating tablet copy from peer a1e2e6b86a1747a4ae340383599dcff4 (127.5.240.196:36708) I0703 01:41:19.115778 21167 tablet_copy_client.cc:239] T 8058ebbaf0444a03a06a5cc261243a60 P 86055af88ab04036878f3822fe571c07: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.196:36708 I0703 01:41:19.125681 20779 tablet_copy_service.cc:135] P a1e2e6b86a1747a4ae340383599dcff4: Received BeginTabletCopySession request for tablet 8058ebbaf0444a03a06a5cc261243a60 from peer 86055af88ab04036878f3822fe571c07 ({username='jenkins'} at 127.5.240.193:45823) I0703 01:41:19.125900 20779 tablet_copy_service.cc:156] P a1e2e6b86a1747a4ae340383599dcff4: Beginning new tablet copy session on tablet 8058ebbaf0444a03a06a5cc261243a60 from peer 86055af88ab04036878f3822fe571c07 at {username='jenkins'} at 127.5.240.193:45823: session id = 86055af88ab04036878f3822fe571c07-8058ebbaf0444a03a06a5cc261243a60 I0703 01:41:19.128971 20779 tablet_copy_source_session.cc:202] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:41:19.132587 21167 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 8058ebbaf0444a03a06a5cc261243a60. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:41:19.147529 21167 ts_tablet_manager.cc:1173] T 8058ebbaf0444a03a06a5cc261243a60 P 86055af88ab04036878f3822fe571c07: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:41:19.147828 21167 tablet_copy_client.cc:566] T 8058ebbaf0444a03a06a5cc261243a60 P 86055af88ab04036878f3822fe571c07: tablet copy: Starting download of 0 data blocks... I0703 01:41:19.148064 21167 tablet_copy_client.cc:529] T 8058ebbaf0444a03a06a5cc261243a60 P 86055af88ab04036878f3822fe571c07: tablet copy: Starting download of 1 WAL segments... I0703 01:41:19.157207 21167 tablet_copy_client.cc:405] T 8058ebbaf0444a03a06a5cc261243a60 P 86055af88ab04036878f3822fe571c07: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:41:19.158988 21167 ts_tablet_manager.cc:1028] T 8058ebbaf0444a03a06a5cc261243a60 P 86055af88ab04036878f3822fe571c07: Bootstrapping tablet I0703 01:41:19.159210 21167 tablet_bootstrap.cc:427] T 8058ebbaf0444a03a06a5cc261243a60 P 86055af88ab04036878f3822fe571c07: Bootstrap starting. I0703 01:41:19.219925 21167 log.cc:525] T 8058ebbaf0444a03a06a5cc261243a60 P 86055af88ab04036878f3822fe571c07: Log is configured to *not* fsync() on all Append() calls W0703 01:41:19.534994 20691 consensus_peers.cc:433] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 -> Peer 86055af88ab04036878f3822fe571c07 (127.5.240.193:42232): Couldn't send request to peer 86055af88ab04036878f3822fe571c07 for tablet 8058ebbaf0444a03a06a5cc261243a60. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 2 times. I0703 01:41:19.609557 21167 tablet_bootstrap.cc:427] T 8058ebbaf0444a03a06a5cc261243a60 P 86055af88ab04036878f3822fe571c07: Bootstrap replayed 1/1 log segments. Stats: ops{read=571 overwritten=0 applied=570 ignored=0} inserts{seen=564 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates I0703 01:41:19.610127 21167 tablet_bootstrap.cc:427] T 8058ebbaf0444a03a06a5cc261243a60 P 86055af88ab04036878f3822fe571c07: Bootstrap complete. I0703 01:41:19.610404 21167 ts_tablet_manager.cc:1045] T 8058ebbaf0444a03a06a5cc261243a60 P 86055af88ab04036878f3822fe571c07: Time spent bootstrapping tablet: real 0.451s user 0.426s sys 0.022s I0703 01:41:19.614349 21167 raft_consensus.cc:325] T 8058ebbaf0444a03a06a5cc261243a60 P 86055af88ab04036878f3822fe571c07 [term 2 LEARNER]: Replica starting. Triggering 1 pending transactions. Active config: opid_index: 490 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } } peers { permanent_uuid: "a1e2e6b86a1747a4ae340383599dcff4" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 36708 } attrs { replace: true } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false } } peers { permanent_uuid: "86055af88ab04036878f3822fe571c07" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 42232 } attrs { promote: true } } I0703 01:41:19.616911 21167 raft_consensus.cc:628] T 8058ebbaf0444a03a06a5cc261243a60 P 86055af88ab04036878f3822fe571c07 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 86055af88ab04036878f3822fe571c07, State: Initialized, Role: LEARNER I0703 01:41:19.617328 21167 consensus_queue.cc:227] T 8058ebbaf0444a03a06a5cc261243a60 P 86055af88ab04036878f3822fe571c07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 570, Last appended: 2.571, Last appended by leader: 571, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 490 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } } peers { permanent_uuid: "a1e2e6b86a1747a4ae340383599dcff4" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 36708 } attrs { replace: true } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false } } peers { permanent_uuid: "86055af88ab04036878f3822fe571c07" member_type: NON_VOTER last_known_addr { host: "127.5.240.193" port: 42232 } attrs { promote: true } } I0703 01:41:19.619501 21167 ts_tablet_manager.cc:1073] T 8058ebbaf0444a03a06a5cc261243a60 P 86055af88ab04036878f3822fe571c07: Time spent starting tablet: real 0.009s user 0.008s sys 0.001s I0703 01:41:19.620275 20779 tablet_copy_service.cc:325] P a1e2e6b86a1747a4ae340383599dcff4: Request end of tablet copy session 86055af88ab04036878f3822fe571c07-8058ebbaf0444a03a06a5cc261243a60 received from {username='jenkins'} at 127.5.240.193:45823 I0703 01:41:19.620358 20779 tablet_copy_service.cc:413] P a1e2e6b86a1747a4ae340383599dcff4: ending tablet copy session 86055af88ab04036878f3822fe571c07-8058ebbaf0444a03a06a5cc261243a60 on tablet 8058ebbaf0444a03a06a5cc261243a60 with peer 86055af88ab04036878f3822fe571c07 I0703 01:41:20.139768 20392 raft_consensus.cc:1026] T 8058ebbaf0444a03a06a5cc261243a60 P 86055af88ab04036878f3822fe571c07 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.489->[2.490-2.752] Dedup: 2.571->[2.572-2.752] I0703 01:41:20.267192 21097 raft_consensus.cc:890] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4: attempting to promote NON_VOTER 86055af88ab04036878f3822fe571c07 to VOTER I0703 01:41:20.267597 21097 consensus_peers.cc:455] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 -> Peer 86055af88ab04036878f3822fe571c07 (127.5.240.193:42232): Closing peer: 86055af88ab04036878f3822fe571c07 I0703 01:41:20.267669 21097 consensus_peers.cc:455] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 -> Peer 406b32afe710485ba1ca267c6629542e (127.5.240.197:43025): Closing peer: 406b32afe710485ba1ca267c6629542e I0703 01:41:20.267715 21097 consensus_peers.cc:455] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 -> Peer d7efcfaf2a624349a32682447014e68f (127.5.240.195:41493): Closing peer: d7efcfaf2a624349a32682447014e68f I0703 01:41:20.267930 21097 consensus_queue.cc:204] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 771, Committed index: 771, Last appended: 2.772, Last appended by leader: 306, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 773 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } } peers { permanent_uuid: "a1e2e6b86a1747a4ae340383599dcff4" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 36708 } attrs { replace: true } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false } } peers { permanent_uuid: "86055af88ab04036878f3822fe571c07" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42232 } attrs { promote: false } } I0703 01:41:20.270459 20881 raft_consensus.cc:1081] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [term 2 FOLLOWER]: Refusing update from remote peer a1e2e6b86a1747a4ae340383599dcff4: Log matching property violated. Preceding OpId in replica: term: 2 index: 772. Preceding OpId from leader: term: 2 index: 773. (index mismatch) I0703 01:41:20.270959 20637 raft_consensus.cc:1081] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f [term 2 FOLLOWER]: Refusing update from remote peer a1e2e6b86a1747a4ae340383599dcff4: Log matching property violated. Preceding OpId in replica: term: 2 index: 772. Preceding OpId from leader: term: 2 index: 773. (index mismatch) I0703 01:41:20.271493 21166 consensus_queue.cc:962] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 773, Last known committed idx: 771, Time since last communication: 0.000s I0703 01:41:20.271873 21097 consensus_queue.cc:962] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 773, Last known committed idx: 771, Time since last communication: 0.000s I0703 01:41:20.272423 20391 raft_consensus.cc:1081] T 8058ebbaf0444a03a06a5cc261243a60 P 86055af88ab04036878f3822fe571c07 [term 2 LEARNER]: Refusing update from remote peer a1e2e6b86a1747a4ae340383599dcff4: Log matching property violated. Preceding OpId in replica: term: 2 index: 772. Preceding OpId from leader: term: 2 index: 773. (index mismatch) I0703 01:41:20.273149 21097 consensus_queue.cc:962] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86055af88ab04036878f3822fe571c07" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42232 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 773, Last known committed idx: 771, Time since last communication: 0.000s I0703 01:41:20.274628 21090 raft_consensus.cc:2684] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [term 2 LEADER]: Committing config change with OpId 2.773: config changed from index 490 to 773, 86055af88ab04036878f3822fe571c07 (127.5.240.193) changed from NON_VOTER to VOTER. New config: { opid_index: 773 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } } peers { permanent_uuid: "a1e2e6b86a1747a4ae340383599dcff4" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 36708 } attrs { replace: true } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false } } peers { permanent_uuid: "86055af88ab04036878f3822fe571c07" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42232 } attrs { promote: false } } } I0703 01:41:20.276312 20392 raft_consensus.cc:2684] T 8058ebbaf0444a03a06a5cc261243a60 P 86055af88ab04036878f3822fe571c07 [term 2 FOLLOWER]: Committing config change with OpId 2.773: config changed from index 490 to 773, 86055af88ab04036878f3822fe571c07 (127.5.240.193) changed from NON_VOTER to VOTER. New config: { opid_index: 773 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } } peers { permanent_uuid: "a1e2e6b86a1747a4ae340383599dcff4" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 36708 } attrs { replace: true } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false } } peers { permanent_uuid: "86055af88ab04036878f3822fe571c07" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42232 } attrs { promote: false } } } I0703 01:41:20.278327 20637 raft_consensus.cc:2684] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f [term 2 FOLLOWER]: Committing config change with OpId 2.773: config changed from index 490 to 773, 86055af88ab04036878f3822fe571c07 (127.5.240.193) changed from NON_VOTER to VOTER. New config: { opid_index: 773 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } } peers { permanent_uuid: "a1e2e6b86a1747a4ae340383599dcff4" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 36708 } attrs { replace: true } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false } } peers { permanent_uuid: "86055af88ab04036878f3822fe571c07" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42232 } attrs { promote: false } } } I0703 01:41:20.279047 20272 catalog_manager.cc:3798] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 reported cstate change: config changed from index 490 to 773, 86055af88ab04036878f3822fe571c07 (127.5.240.193) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "a1e2e6b86a1747a4ae340383599dcff4" committed_config { opid_index: 773 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1e2e6b86a1747a4ae340383599dcff4" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 36708 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "86055af88ab04036878f3822fe571c07" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42232 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I0703 01:41:20.281708 20881 raft_consensus.cc:2684] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [term 2 FOLLOWER]: Committing config change with OpId 2.773: config changed from index 490 to 773, 86055af88ab04036878f3822fe571c07 (127.5.240.193) changed from NON_VOTER to VOTER. New config: { opid_index: 773 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } } peers { permanent_uuid: "a1e2e6b86a1747a4ae340383599dcff4" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 36708 } attrs { replace: true } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false } } peers { permanent_uuid: "86055af88ab04036878f3822fe571c07" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42232 } attrs { promote: false } } } I0703 01:41:20.337893 20759 tablet_service.cc:1140] Received LeaderStepDown RPC: tablet_id: "8058ebbaf0444a03a06a5cc261243a60" dest_uuid: "a1e2e6b86a1747a4ae340383599dcff4" from {username='jenkins'} at 127.5.240.196:41436 I0703 01:41:20.338405 20759 raft_consensus.cc:552] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [term 2 LEADER]: Received request to step down I0703 01:41:20.338479 20759 raft_consensus.cc:628] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a1e2e6b86a1747a4ae340383599dcff4, State: Running, Role: LEADER I0703 01:41:20.338917 20759 consensus_queue.cc:227] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 780, Committed index: 780, Last appended: 2.781, Last appended by leader: 781, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 773 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } } peers { permanent_uuid: "a1e2e6b86a1747a4ae340383599dcff4" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 36708 } attrs { replace: true } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false } } peers { permanent_uuid: "86055af88ab04036878f3822fe571c07" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42232 } attrs { promote: false } } I0703 01:41:20.339339 20759 consensus_peers.cc:455] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 -> Peer 86055af88ab04036878f3822fe571c07 (127.5.240.193:42232): Closing peer: 86055af88ab04036878f3822fe571c07 I0703 01:41:20.339507 20759 consensus_peers.cc:455] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 -> Peer 406b32afe710485ba1ca267c6629542e (127.5.240.197:43025): Closing peer: 406b32afe710485ba1ca267c6629542e W0703 01:41:20.339504 21166 consensus_queue.cc:1041] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "86055af88ab04036878f3822fe571c07" responder_term: 2 status { last_received { term: 2 index: 780 } last_committed_idx: 780 last_received_current_leader { term: 2 index: 780 } } I0703 01:41:20.339751 20759 consensus_peers.cc:455] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 -> Peer d7efcfaf2a624349a32682447014e68f (127.5.240.195:41493): Closing peer: d7efcfaf2a624349a32682447014e68f I0703 01:41:20.339922 20759 raft_consensus.cc:2730] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4: Snoozing failure detection for 3.000s (explicit stepdown request) I0703 01:41:21.837278 21219 raft_consensus.cc:457] T 8058ebbaf0444a03a06a5cc261243a60 P 86055af88ab04036878f3822fe571c07 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a1e2e6b86a1747a4ae340383599dcff4) I0703 01:41:21.837469 21219 raft_consensus.cc:2730] T 8058ebbaf0444a03a06a5cc261243a60 P 86055af88ab04036878f3822fe571c07: Snoozing failure detection for 1.651s (starting election) I0703 01:41:21.837544 21220 raft_consensus.cc:457] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a1e2e6b86a1747a4ae340383599dcff4) I0703 01:41:21.837635 21220 raft_consensus.cc:2730] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f: Snoozing failure detection for 2.001s (starting election) I0703 01:41:21.837620 21219 raft_consensus.cc:479] T 8058ebbaf0444a03a06a5cc261243a60 P 86055af88ab04036878f3822fe571c07 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 773 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } } peers { permanent_uuid: "a1e2e6b86a1747a4ae340383599dcff4" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 36708 } attrs { replace: true } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false } } peers { permanent_uuid: "86055af88ab04036878f3822fe571c07" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42232 } attrs { promote: false } } I0703 01:41:21.837733 21220 raft_consensus.cc:479] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 773 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } } peers { permanent_uuid: "a1e2e6b86a1747a4ae340383599dcff4" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 36708 } attrs { replace: true } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false } } peers { permanent_uuid: "86055af88ab04036878f3822fe571c07" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42232 } attrs { promote: false } } I0703 01:41:21.838223 21220 leader_election.cc:243] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer a1e2e6b86a1747a4ae340383599dcff4 (127.5.240.196:36708) I0703 01:41:21.838399 21220 leader_election.cc:243] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 406b32afe710485ba1ca267c6629542e (127.5.240.197:43025) I0703 01:41:21.838464 21219 leader_election.cc:243] T 8058ebbaf0444a03a06a5cc261243a60 P 86055af88ab04036878f3822fe571c07 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer d7efcfaf2a624349a32682447014e68f (127.5.240.195:41493) I0703 01:41:21.838631 21220 leader_election.cc:243] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 86055af88ab04036878f3822fe571c07 (127.5.240.193:42232) I0703 01:41:21.838773 21219 leader_election.cc:243] T 8058ebbaf0444a03a06a5cc261243a60 P 86055af88ab04036878f3822fe571c07 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer a1e2e6b86a1747a4ae340383599dcff4 (127.5.240.196:36708) I0703 01:41:21.839313 20881 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "8058ebbaf0444a03a06a5cc261243a60" candidate_uuid: "d7efcfaf2a624349a32682447014e68f" candidate_term: 3 candidate_status { last_received { term: 2 index: 780 } } ignore_live_leader: false dest_uuid: "406b32afe710485ba1ca267c6629542e" is_pre_election: true I0703 01:41:21.839499 20881 raft_consensus.cc:2204] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7efcfaf2a624349a32682447014e68f for term 3 because replica is either leader or believes a valid leader to be alive. I0703 01:41:21.839686 20759 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "8058ebbaf0444a03a06a5cc261243a60" candidate_uuid: "d7efcfaf2a624349a32682447014e68f" candidate_term: 3 candidate_status { last_received { term: 2 index: 780 } } ignore_live_leader: false dest_uuid: "a1e2e6b86a1747a4ae340383599dcff4" is_pre_election: true I0703 01:41:21.839993 20759 raft_consensus.cc:2190] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7efcfaf2a624349a32682447014e68f for term 3 because replica has last-logged OpId of term: 2 index: 781, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 780. I0703 01:41:21.840262 20758 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "8058ebbaf0444a03a06a5cc261243a60" candidate_uuid: "86055af88ab04036878f3822fe571c07" candidate_term: 3 candidate_status { last_received { term: 2 index: 780 } } ignore_live_leader: false dest_uuid: "a1e2e6b86a1747a4ae340383599dcff4" is_pre_election: true I0703 01:41:21.840477 20758 raft_consensus.cc:2190] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 86055af88ab04036878f3822fe571c07 for term 3 because replica has last-logged OpId of term: 2 index: 781, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 780. I0703 01:41:21.840692 20570 leader_election.cc:393] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f [CANDIDATE]: Term 3 pre-election: Vote denied by peer 406b32afe710485ba1ca267c6629542e (127.5.240.197:43025). Message: Invalid argument: T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7efcfaf2a624349a32682447014e68f for term 3 because replica is either leader or believes a valid leader to be alive. I0703 01:41:21.841747 20568 leader_election.cc:393] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f [CANDIDATE]: Term 3 pre-election: Vote denied by peer a1e2e6b86a1747a4ae340383599dcff4 (127.5.240.196:36708). Message: Invalid argument: T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7efcfaf2a624349a32682447014e68f for term 3 because replica has last-logged OpId of term: 2 index: 781, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 780. I0703 01:41:21.841867 20568 leader_election.cc:270] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. I0703 01:41:21.841936 20323 leader_election.cc:393] T 8058ebbaf0444a03a06a5cc261243a60 P 86055af88ab04036878f3822fe571c07 [CANDIDATE]: Term 3 pre-election: Vote denied by peer a1e2e6b86a1747a4ae340383599dcff4 (127.5.240.196:36708). Message: Invalid argument: T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 86055af88ab04036878f3822fe571c07 for term 3 because replica has last-logged OpId of term: 2 index: 781, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 780. I0703 01:41:21.842048 21225 raft_consensus.cc:457] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a1e2e6b86a1747a4ae340383599dcff4) I0703 01:41:21.839172 21219 leader_election.cc:243] T 8058ebbaf0444a03a06a5cc261243a60 P 86055af88ab04036878f3822fe571c07 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 406b32afe710485ba1ca267c6629542e (127.5.240.197:43025) I0703 01:41:21.842142 21225 raft_consensus.cc:2730] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e: Snoozing failure detection for 1.799s (starting election) I0703 01:41:21.842254 21225 raft_consensus.cc:479] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 773 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } } peers { permanent_uuid: "a1e2e6b86a1747a4ae340383599dcff4" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 36708 } attrs { replace: true } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false } } peers { permanent_uuid: "86055af88ab04036878f3822fe571c07" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42232 } attrs { promote: false } } I0703 01:41:21.842893 21225 leader_election.cc:243] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer d7efcfaf2a624349a32682447014e68f (127.5.240.195:41493) I0703 01:41:21.842927 21220 raft_consensus.cc:2730] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f: Snoozing failure detection for 2.227s (election complete) I0703 01:41:21.843047 21220 raft_consensus.cc:2485] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority I0703 01:41:21.843991 20637 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "8058ebbaf0444a03a06a5cc261243a60" candidate_uuid: "406b32afe710485ba1ca267c6629542e" candidate_term: 3 candidate_status { last_received { term: 2 index: 781 } } ignore_live_leader: false dest_uuid: "d7efcfaf2a624349a32682447014e68f" is_pre_election: true I0703 01:41:21.844132 20637 raft_consensus.cc:2730] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f: Snoozing failure detection for 1.739s (vote granted) I0703 01:41:21.844184 20637 raft_consensus.cc:2243] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 406b32afe710485ba1ca267c6629542e in term 2. I0703 01:41:21.844919 20814 leader_election.cc:379] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [CANDIDATE]: Term 3 pre-election: Vote granted by peer d7efcfaf2a624349a32682447014e68f (127.5.240.195:41493) I0703 01:41:21.846324 21225 leader_election.cc:243] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer a1e2e6b86a1747a4ae340383599dcff4 (127.5.240.196:36708) I0703 01:41:21.848206 21225 leader_election.cc:243] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 86055af88ab04036878f3822fe571c07 (127.5.240.193:42232) I0703 01:41:21.848275 20758 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "8058ebbaf0444a03a06a5cc261243a60" candidate_uuid: "406b32afe710485ba1ca267c6629542e" candidate_term: 3 candidate_status { last_received { term: 2 index: 781 } } ignore_live_leader: false dest_uuid: "a1e2e6b86a1747a4ae340383599dcff4" is_pre_election: true I0703 01:41:21.848418 20758 raft_consensus.cc:2730] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4: Snoozing failure detection for 1.648s (vote granted) I0703 01:41:21.848497 20758 raft_consensus.cc:2243] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 406b32afe710485ba1ca267c6629542e in term 2. I0703 01:41:21.849071 20812 leader_election.cc:379] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [CANDIDATE]: Term 3 pre-election: Vote granted by peer a1e2e6b86a1747a4ae340383599dcff4 (127.5.240.196:36708) I0703 01:41:21.849172 20812 leader_election.cc:270] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. I0703 01:41:21.849426 21225 raft_consensus.cc:2730] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e: Snoozing failure detection for 1.716s (election complete) I0703 01:41:21.850127 21225 raft_consensus.cc:2537] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [term 2 FOLLOWER]: Leader pre-election won for term 3 I0703 01:41:21.850273 21225 raft_consensus.cc:457] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [term 2 FOLLOWER]: Starting leader election (detected failure of leader a1e2e6b86a1747a4ae340383599dcff4) I0703 01:41:21.850416 21225 raft_consensus.cc:2730] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e: Snoozing failure detection for 1.535s (starting election) I0703 01:41:21.850560 21225 raft_consensus.cc:2782] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [term 2 FOLLOWER]: Advancing to term 3 I0703 01:41:21.852733 20637 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "8058ebbaf0444a03a06a5cc261243a60" candidate_uuid: "86055af88ab04036878f3822fe571c07" candidate_term: 3 candidate_status { last_received { term: 2 index: 780 } } ignore_live_leader: false dest_uuid: "d7efcfaf2a624349a32682447014e68f" is_pre_election: true I0703 01:41:21.852924 20637 raft_consensus.cc:2730] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f: Snoozing failure detection for 3.263s (vote granted) I0703 01:41:21.853027 20637 raft_consensus.cc:2243] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86055af88ab04036878f3822fe571c07 in term 2. I0703 01:41:21.853909 21225 raft_consensus.cc:479] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 773 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } } peers { permanent_uuid: "a1e2e6b86a1747a4ae340383599dcff4" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 36708 } attrs { replace: true } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false } } peers { permanent_uuid: "86055af88ab04036878f3822fe571c07" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42232 } attrs { promote: false } } I0703 01:41:21.854286 20325 leader_election.cc:379] T 8058ebbaf0444a03a06a5cc261243a60 P 86055af88ab04036878f3822fe571c07 [CANDIDATE]: Term 3 pre-election: Vote granted by peer d7efcfaf2a624349a32682447014e68f (127.5.240.195:41493) I0703 01:41:21.854220 20392 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "8058ebbaf0444a03a06a5cc261243a60" candidate_uuid: "d7efcfaf2a624349a32682447014e68f" candidate_term: 3 candidate_status { last_received { term: 2 index: 780 } } ignore_live_leader: false dest_uuid: "86055af88ab04036878f3822fe571c07" is_pre_election: true I0703 01:41:21.854502 20392 raft_consensus.cc:2730] T 8058ebbaf0444a03a06a5cc261243a60 P 86055af88ab04036878f3822fe571c07: Snoozing failure detection for 2.032s (vote granted) I0703 01:41:21.854581 20392 raft_consensus.cc:2243] T 8058ebbaf0444a03a06a5cc261243a60 P 86055af88ab04036878f3822fe571c07 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7efcfaf2a624349a32682447014e68f in term 2. I0703 01:41:21.855195 20569 leader_election.cc:379] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f [CANDIDATE]: Term 3 pre-election: Vote granted by peer 86055af88ab04036878f3822fe571c07 (127.5.240.193:42232) I0703 01:41:21.856858 21225 leader_election.cc:243] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [CANDIDATE]: Term 3 election: Requesting vote from peer d7efcfaf2a624349a32682447014e68f (127.5.240.195:41493) I0703 01:41:21.857551 20881 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "8058ebbaf0444a03a06a5cc261243a60" candidate_uuid: "86055af88ab04036878f3822fe571c07" candidate_term: 3 candidate_status { last_received { term: 2 index: 780 } } ignore_live_leader: false dest_uuid: "406b32afe710485ba1ca267c6629542e" is_pre_election: true I0703 01:41:21.857682 20637 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "8058ebbaf0444a03a06a5cc261243a60" candidate_uuid: "406b32afe710485ba1ca267c6629542e" candidate_term: 3 candidate_status { last_received { term: 2 index: 781 } } ignore_live_leader: false dest_uuid: "d7efcfaf2a624349a32682447014e68f" I0703 01:41:21.857811 20637 raft_consensus.cc:2782] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f [term 2 FOLLOWER]: Advancing to term 3 I0703 01:41:21.857836 20881 raft_consensus.cc:2173] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 86055af88ab04036878f3822fe571c07 in current term 3: Already voted for candidate 406b32afe710485ba1ca267c6629542e in this term. I0703 01:41:21.857877 20637 raft_consensus.cc:2730] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f: Snoozing failure detection for 1.749s (vote granted) I0703 01:41:21.858521 20325 leader_election.cc:393] T 8058ebbaf0444a03a06a5cc261243a60 P 86055af88ab04036878f3822fe571c07 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 406b32afe710485ba1ca267c6629542e (127.5.240.197:43025). Message: Invalid argument: T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 86055af88ab04036878f3822fe571c07 in current term 3: Already voted for candidate 406b32afe710485ba1ca267c6629542e in this term. I0703 01:41:21.858628 20325 leader_election.cc:270] T 8058ebbaf0444a03a06a5cc261243a60 P 86055af88ab04036878f3822fe571c07 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. I0703 01:41:21.858927 21219 raft_consensus.cc:2730] T 8058ebbaf0444a03a06a5cc261243a60 P 86055af88ab04036878f3822fe571c07: Snoozing failure detection for 1.798s (election complete) I0703 01:41:21.859017 21219 raft_consensus.cc:2782] T 8058ebbaf0444a03a06a5cc261243a60 P 86055af88ab04036878f3822fe571c07 [term 2 FOLLOWER]: Advancing to term 3 I0703 01:41:21.859910 20637 raft_consensus.cc:2243] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 406b32afe710485ba1ca267c6629542e in term 3. I0703 01:41:21.860484 20814 leader_election.cc:379] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [CANDIDATE]: Term 3 election: Vote granted by peer d7efcfaf2a624349a32682447014e68f (127.5.240.195:41493) I0703 01:41:21.860494 21219 raft_consensus.cc:2485] T 8058ebbaf0444a03a06a5cc261243a60 P 86055af88ab04036878f3822fe571c07 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority I0703 01:41:21.860606 21225 leader_election.cc:243] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [CANDIDATE]: Term 3 election: Requesting vote from peer a1e2e6b86a1747a4ae340383599dcff4 (127.5.240.196:36708) I0703 01:41:21.860816 21225 leader_election.cc:243] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [CANDIDATE]: Term 3 election: Requesting vote from peer 86055af88ab04036878f3822fe571c07 (127.5.240.193:42232) I0703 01:41:21.861378 20758 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "8058ebbaf0444a03a06a5cc261243a60" candidate_uuid: "406b32afe710485ba1ca267c6629542e" candidate_term: 3 candidate_status { last_received { term: 2 index: 781 } } ignore_live_leader: false dest_uuid: "a1e2e6b86a1747a4ae340383599dcff4" I0703 01:41:21.861524 20758 raft_consensus.cc:2782] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [term 2 FOLLOWER]: Advancing to term 3 I0703 01:41:21.861621 20758 raft_consensus.cc:2730] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4: Snoozing failure detection for 1.681s (vote granted) I0703 01:41:21.862681 20392 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "8058ebbaf0444a03a06a5cc261243a60" candidate_uuid: "406b32afe710485ba1ca267c6629542e" candidate_term: 3 candidate_status { last_received { term: 2 index: 781 } } ignore_live_leader: false dest_uuid: "86055af88ab04036878f3822fe571c07" is_pre_election: true I0703 01:41:21.862838 20392 raft_consensus.cc:2730] T 8058ebbaf0444a03a06a5cc261243a60 P 86055af88ab04036878f3822fe571c07: Snoozing failure detection for 2.007s (vote granted) I0703 01:41:21.862896 20392 raft_consensus.cc:2243] T 8058ebbaf0444a03a06a5cc261243a60 P 86055af88ab04036878f3822fe571c07 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 406b32afe710485ba1ca267c6629542e in term 3. I0703 01:41:21.863458 20392 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "8058ebbaf0444a03a06a5cc261243a60" candidate_uuid: "406b32afe710485ba1ca267c6629542e" candidate_term: 3 candidate_status { last_received { term: 2 index: 781 } } ignore_live_leader: false dest_uuid: "86055af88ab04036878f3822fe571c07" I0703 01:41:21.863601 20392 raft_consensus.cc:2730] T 8058ebbaf0444a03a06a5cc261243a60 P 86055af88ab04036878f3822fe571c07: Snoozing failure detection for 2.035s (vote granted) I0703 01:41:21.863622 20758 raft_consensus.cc:2243] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 406b32afe710485ba1ca267c6629542e in term 3. I0703 01:41:21.864188 20812 leader_election.cc:379] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [CANDIDATE]: Term 3 election: Vote granted by peer a1e2e6b86a1747a4ae340383599dcff4 (127.5.240.196:36708) I0703 01:41:21.864257 20812 leader_election.cc:270] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. I0703 01:41:21.864451 21225 raft_consensus.cc:2730] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e: Snoozing failure detection for 1.503s (election complete) I0703 01:41:21.864526 21225 raft_consensus.cc:2537] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [term 3 FOLLOWER]: Leader election won for term 3 I0703 01:41:21.864928 20813 leader_election.cc:379] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [CANDIDATE]: Term 3 pre-election: Vote granted by peer 86055af88ab04036878f3822fe571c07 (127.5.240.193:42232) I0703 01:41:21.865607 20392 raft_consensus.cc:2243] T 8058ebbaf0444a03a06a5cc261243a60 P 86055af88ab04036878f3822fe571c07 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 406b32afe710485ba1ca267c6629542e in term 3. I0703 01:41:21.865736 21225 raft_consensus.cc:593] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [term 3 LEADER]: Becoming Leader. State: Replica: 406b32afe710485ba1ca267c6629542e, State: Running, Role: LEADER I0703 01:41:21.866188 20813 leader_election.cc:379] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [CANDIDATE]: Term 3 election: Vote granted by peer 86055af88ab04036878f3822fe571c07 (127.5.240.193:42232) I0703 01:41:21.866370 21225 consensus_queue.cc:204] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 780, Committed index: 780, Last appended: 2.781, Last appended by leader: 781, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 773 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } } peers { permanent_uuid: "a1e2e6b86a1747a4ae340383599dcff4" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 36708 } attrs { replace: true } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false } } peers { permanent_uuid: "86055af88ab04036878f3822fe571c07" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42232 } attrs { promote: false } } I0703 01:41:21.869151 20270 catalog_manager.cc:3798] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e reported cstate change: term changed from 2 to 3, leader changed from a1e2e6b86a1747a4ae340383599dcff4 (127.5.240.196) to 406b32afe710485ba1ca267c6629542e (127.5.240.197). New cstate: current_term: 3 leader_uuid: "406b32afe710485ba1ca267c6629542e" committed_config { opid_index: 773 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1e2e6b86a1747a4ae340383599dcff4" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 36708 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "86055af88ab04036878f3822fe571c07" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42232 } attrs { promote: false } health_report { overall_health: UNKNOWN } } } I0703 01:41:22.320895 20758 raft_consensus.cc:1081] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [term 3 FOLLOWER]: Refusing update from remote peer 406b32afe710485ba1ca267c6629542e: Log matching property violated. Preceding OpId in replica: term: 2 index: 781. Preceding OpId from leader: term: 3 index: 782. (index mismatch) I0703 01:41:22.321561 21225 consensus_queue.cc:962] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1e2e6b86a1747a4ae340383599dcff4" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 36708 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 782, Last known committed idx: 780, Time since last communication: 0.000s I0703 01:41:22.381292 20392 raft_consensus.cc:1081] T 8058ebbaf0444a03a06a5cc261243a60 P 86055af88ab04036878f3822fe571c07 [term 3 FOLLOWER]: Refusing update from remote peer 406b32afe710485ba1ca267c6629542e: Log matching property violated. Preceding OpId in replica: term: 2 index: 780. Preceding OpId from leader: term: 3 index: 782. (index mismatch) I0703 01:41:22.381909 21225 consensus_queue.cc:962] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [LEADER]: Connected to new peer: Peer: permanent_uuid: "86055af88ab04036878f3822fe571c07" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42232 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 782, Last known committed idx: 780, Time since last communication: 0.000s I0703 01:41:22.388741 20637 raft_consensus.cc:1081] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f [term 3 FOLLOWER]: Refusing update from remote peer 406b32afe710485ba1ca267c6629542e: Log matching property violated. Preceding OpId in replica: term: 2 index: 780. Preceding OpId from leader: term: 3 index: 782. (index mismatch) I0703 01:41:22.389827 21254 consensus_queue.cc:962] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 782, Last known committed idx: 780, Time since last communication: 0.000s I0703 01:41:22.393328 21257 rpcz_store.cc:265] Call kudu.tserver.TabletServerService.Write from 127.5.240.196:41415 (ReqId={client: 2499667f6bcc4c699e4f6bf5bd2dde19, seq_no=773, attempt_no=0}) took 2057ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":4923,"num_ops":1,"prepare.queue_time_us":1055,"prepare.run_cpu_time_us":682,"prepare.run_wall_time_us":710,"raft.queue_time_us":38,"raft.run_cpu_time_us":240,"raft.run_wall_time_us":292,"replication_time_us":2050085,"spinlock_wait_cycles":259328,"thread_start_us":4782,"threads_started":1}]]} I0703 01:41:22.423089 20269 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 8058ebbaf0444a03a06a5cc261243a60 (attempt 1) I0703 01:41:22.436552 20881 consensus_peers.cc:455] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e -> Peer 86055af88ab04036878f3822fe571c07 (127.5.240.193:42232): Closing peer: 86055af88ab04036878f3822fe571c07 I0703 01:41:22.436695 20881 consensus_peers.cc:455] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e -> Peer a1e2e6b86a1747a4ae340383599dcff4 (127.5.240.196:36708): Closing peer: a1e2e6b86a1747a4ae340383599dcff4 I0703 01:41:22.436785 20881 consensus_peers.cc:455] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e -> Peer d7efcfaf2a624349a32682447014e68f (127.5.240.195:41493): Closing peer: d7efcfaf2a624349a32682447014e68f I0703 01:41:22.437067 20881 consensus_queue.cc:204] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 785, Committed index: 785, Last appended: 3.785, Last appended by leader: 781, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 786 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false } } peers { permanent_uuid: "86055af88ab04036878f3822fe571c07" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42232 } attrs { promote: false } } I0703 01:41:22.439029 20392 raft_consensus.cc:1081] T 8058ebbaf0444a03a06a5cc261243a60 P 86055af88ab04036878f3822fe571c07 [term 3 FOLLOWER]: Refusing update from remote peer 406b32afe710485ba1ca267c6629542e: Log matching property violated. Preceding OpId in replica: term: 3 index: 785. Preceding OpId from leader: term: 3 index: 786. (index mismatch) I0703 01:41:22.439492 20637 raft_consensus.cc:1081] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f [term 3 FOLLOWER]: Refusing update from remote peer 406b32afe710485ba1ca267c6629542e: Log matching property violated. Preceding OpId in replica: term: 3 index: 785. Preceding OpId from leader: term: 3 index: 786. (index mismatch) I0703 01:41:22.439658 21263 consensus_queue.cc:962] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [LEADER]: Connected to new peer: Peer: permanent_uuid: "86055af88ab04036878f3822fe571c07" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42232 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 786, Last known committed idx: 785, Time since last communication: 0.000s I0703 01:41:22.440104 21254 consensus_queue.cc:962] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 786, Last known committed idx: 785, Time since last communication: 0.000s I0703 01:41:22.441834 21263 raft_consensus.cc:2684] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [term 3 LEADER]: Committing config change with OpId 3.786: config changed from index 773 to 786, VOTER a1e2e6b86a1747a4ae340383599dcff4 (127.5.240.196) evicted. New config: { opid_index: 786 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false } } peers { permanent_uuid: "86055af88ab04036878f3822fe571c07" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42232 } attrs { promote: false } } } I0703 01:41:22.442736 20392 raft_consensus.cc:2684] T 8058ebbaf0444a03a06a5cc261243a60 P 86055af88ab04036878f3822fe571c07 [term 3 FOLLOWER]: Committing config change with OpId 3.786: config changed from index 773 to 786, VOTER a1e2e6b86a1747a4ae340383599dcff4 (127.5.240.196) evicted. New config: { opid_index: 786 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false } } peers { permanent_uuid: "86055af88ab04036878f3822fe571c07" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42232 } attrs { promote: false } } } I0703 01:41:22.443064 20637 raft_consensus.cc:2684] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f [term 3 FOLLOWER]: Committing config change with OpId 3.786: config changed from index 773 to 786, VOTER a1e2e6b86a1747a4ae340383599dcff4 (127.5.240.196) evicted. New config: { opid_index: 786 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false } } peers { permanent_uuid: "86055af88ab04036878f3822fe571c07" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42232 } attrs { promote: false } } } I0703 01:41:22.445545 20255 catalog_manager.cc:3352] ChangeConfig:REMOVE_PEER RPC for tablet 8058ebbaf0444a03a06a5cc261243a60 with cas_config_opid_index 773: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0703 01:41:22.446810 20269 catalog_manager.cc:3798] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f reported cstate change: config changed from index 773 to 786, VOTER a1e2e6b86a1747a4ae340383599dcff4 (127.5.240.196) evicted. New cstate: current_term: 3 leader_uuid: "406b32afe710485ba1ca267c6629542e" committed_config { opid_index: 786 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false } } peers { permanent_uuid: "86055af88ab04036878f3822fe571c07" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42232 } attrs { promote: false } } } I0703 01:41:22.450855 20269 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 8058ebbaf0444a03a06a5cc261243a60 on a1e2e6b86a1747a4ae340383599dcff4 (127.5.240.196:36708) (TS a1e2e6b86a1747a4ae340383599dcff4 not found in new config with opid_index 786) I0703 01:41:22.452538 20739 tablet_service.cc:800] Processing DeleteTablet for tablet 8058ebbaf0444a03a06a5cc261243a60 with delete_type TABLET_DATA_TOMBSTONED (TS a1e2e6b86a1747a4ae340383599dcff4 not found in new config with opid_index 786) from {username='jenkins'} at 127.5.240.196:41391 I0703 01:41:22.453979 21269 tablet_replica.cc:261] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4: stopping tablet replica I0703 01:41:22.454167 21269 maintenance_manager.cc:237] P a1e2e6b86a1747a4ae340383599dcff4: Unregistered op CompactRowSetsOp(8058ebbaf0444a03a06a5cc261243a60) I0703 01:41:22.454257 21269 maintenance_manager.cc:237] P a1e2e6b86a1747a4ae340383599dcff4: Unregistered op MinorDeltaCompactionOp(8058ebbaf0444a03a06a5cc261243a60) I0703 01:41:22.454318 21269 maintenance_manager.cc:237] P a1e2e6b86a1747a4ae340383599dcff4: Unregistered op MajorDeltaCompactionOp(8058ebbaf0444a03a06a5cc261243a60) I0703 01:41:22.454381 21269 maintenance_manager.cc:237] P a1e2e6b86a1747a4ae340383599dcff4: Unregistered op UndoDeltaBlockGCOp(8058ebbaf0444a03a06a5cc261243a60) I0703 01:41:22.454475 21269 maintenance_manager.cc:237] P a1e2e6b86a1747a4ae340383599dcff4: Unregistered op FlushMRSOp(8058ebbaf0444a03a06a5cc261243a60) I0703 01:41:22.454555 21269 maintenance_manager.cc:237] P a1e2e6b86a1747a4ae340383599dcff4: Unregistered op FlushDeltaMemStoresOp(8058ebbaf0444a03a06a5cc261243a60) I0703 01:41:22.454607 21269 maintenance_manager.cc:237] P a1e2e6b86a1747a4ae340383599dcff4: Unregistered op LogGCOp(8058ebbaf0444a03a06a5cc261243a60) I0703 01:41:22.454706 21269 raft_consensus.cc:2040] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [term 3 FOLLOWER]: Raft consensus shutting down. I0703 01:41:22.454939 21269 raft_consensus.cc:2067] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4 [term 3 FOLLOWER]: Raft consensus is shut down! I0703 01:41:22.456851 21269 ts_tablet_manager.cc:1369] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:41:22.462785 21269 ts_tablet_manager.cc:1382] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.785 I0703 01:41:22.462913 21269 log.cc:986] T 8058ebbaf0444a03a06a5cc261243a60 P a1e2e6b86a1747a4ae340383599dcff4: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal/wals/8058ebbaf0444a03a06a5cc261243a60 I0703 01:41:22.463728 20253 catalog_manager.cc:3103] TS a1e2e6b86a1747a4ae340383599dcff4 (127.5.240.196:36708): tablet 8058ebbaf0444a03a06a5cc261243a60 (table TestTable [id=cca11239bdb84b6ca083a7a4f8892b07]) successfully deleted W0703 01:41:23.261831 21279 flags.cc:406] Enabled unsafe flag: --never_fsync=true I0703 01:41:23.538110 20881 consensus_peers.cc:455] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e -> Peer 86055af88ab04036878f3822fe571c07 (127.5.240.193:42232): Closing peer: 86055af88ab04036878f3822fe571c07 I0703 01:41:23.538206 20881 consensus_peers.cc:455] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e -> Peer d7efcfaf2a624349a32682447014e68f (127.5.240.195:41493): Closing peer: d7efcfaf2a624349a32682447014e68f I0703 01:41:23.538378 20881 consensus_queue.cc:204] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1004, Committed index: 1004, Last appended: 3.1004, Last appended by leader: 781, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1005 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } attrs { replace: true } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false } } peers { permanent_uuid: "86055af88ab04036878f3822fe571c07" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42232 } attrs { promote: false } } peers { permanent_uuid: "8c3e06132b5b44fb8eb4fc451662d466" member_type: NON_VOTER last_known_addr { host: "127.5.240.194" port: 57030 } attrs { promote: true } } I0703 01:41:23.540369 20392 raft_consensus.cc:1081] T 8058ebbaf0444a03a06a5cc261243a60 P 86055af88ab04036878f3822fe571c07 [term 3 FOLLOWER]: Refusing update from remote peer 406b32afe710485ba1ca267c6629542e: Log matching property violated. Preceding OpId in replica: term: 3 index: 1004. Preceding OpId from leader: term: 3 index: 1005. (index mismatch) I0703 01:41:23.540963 21263 consensus_queue.cc:962] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [LEADER]: Connected to new peer: Peer: permanent_uuid: "86055af88ab04036878f3822fe571c07" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42232 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1005, Last known committed idx: 1004, Time since last communication: 0.000s I0703 01:41:23.541373 20636 raft_consensus.cc:1081] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f [term 3 FOLLOWER]: Refusing update from remote peer 406b32afe710485ba1ca267c6629542e: Log matching property violated. Preceding OpId in replica: term: 3 index: 1004. Preceding OpId from leader: term: 3 index: 1005. (index mismatch) W0703 01:41:23.541611 20814 consensus_peers.cc:433] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e -> Peer 8c3e06132b5b44fb8eb4fc451662d466 (127.5.240.194:57030): Couldn't send request to peer 8c3e06132b5b44fb8eb4fc451662d466 for tablet 8058ebbaf0444a03a06a5cc261243a60. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:41:23.541972 21263 consensus_queue.cc:962] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1005, Last known committed idx: 1004, Time since last communication: 0.000s I0703 01:41:23.544052 21254 raft_consensus.cc:2684] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [term 3 LEADER]: Committing config change with OpId 3.1005: config changed from index 786 to 1005, NON_VOTER 8c3e06132b5b44fb8eb4fc451662d466 (127.5.240.194) added. New config: { opid_index: 1005 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } attrs { replace: true } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false } } peers { permanent_uuid: "86055af88ab04036878f3822fe571c07" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42232 } attrs { promote: false } } peers { permanent_uuid: "8c3e06132b5b44fb8eb4fc451662d466" member_type: NON_VOTER last_known_addr { host: "127.5.240.194" port: 57030 } attrs { promote: true } } } I0703 01:41:23.545110 20392 raft_consensus.cc:2684] T 8058ebbaf0444a03a06a5cc261243a60 P 86055af88ab04036878f3822fe571c07 [term 3 FOLLOWER]: Committing config change with OpId 3.1005: config changed from index 786 to 1005, NON_VOTER 8c3e06132b5b44fb8eb4fc451662d466 (127.5.240.194) added. New config: { opid_index: 1005 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } attrs { replace: true } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false } } peers { permanent_uuid: "86055af88ab04036878f3822fe571c07" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42232 } attrs { promote: false } } peers { permanent_uuid: "8c3e06132b5b44fb8eb4fc451662d466" member_type: NON_VOTER last_known_addr { host: "127.5.240.194" port: 57030 } attrs { promote: true } } } I0703 01:41:23.548243 20636 raft_consensus.cc:2684] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f [term 3 FOLLOWER]: Committing config change with OpId 3.1005: config changed from index 786 to 1005, NON_VOTER 8c3e06132b5b44fb8eb4fc451662d466 (127.5.240.194) added. New config: { opid_index: 1005 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } attrs { replace: true } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false } } peers { permanent_uuid: "86055af88ab04036878f3822fe571c07" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42232 } attrs { promote: false } } peers { permanent_uuid: "8c3e06132b5b44fb8eb4fc451662d466" member_type: NON_VOTER last_known_addr { host: "127.5.240.194" port: 57030 } attrs { promote: true } } } I0703 01:41:23.553149 20270 catalog_manager.cc:3798] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f reported cstate change: config changed from index 786 to 1005, NON_VOTER 8c3e06132b5b44fb8eb4fc451662d466 (127.5.240.194) added. New cstate: current_term: 3 leader_uuid: "406b32afe710485ba1ca267c6629542e" committed_config { opid_index: 1005 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } attrs { replace: true } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false } } peers { permanent_uuid: "86055af88ab04036878f3822fe571c07" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42232 } attrs { promote: false } } peers { permanent_uuid: "8c3e06132b5b44fb8eb4fc451662d466" member_type: NON_VOTER last_known_addr { host: "127.5.240.194" port: 57030 } attrs { promote: true } } } I0703 01:41:23.916646 21330 ts_tablet_manager.cc:669] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466: Initiating tablet copy from peer 406b32afe710485ba1ca267c6629542e (127.5.240.197:43025) I0703 01:41:23.917215 21330 tablet_copy_client.cc:239] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.197:43025 I0703 01:41:23.918066 20901 tablet_copy_service.cc:135] P 406b32afe710485ba1ca267c6629542e: Received BeginTabletCopySession request for tablet 8058ebbaf0444a03a06a5cc261243a60 from peer 8c3e06132b5b44fb8eb4fc451662d466 ({username='jenkins'} at 127.5.240.194:36675) I0703 01:41:23.918321 20901 tablet_copy_service.cc:156] P 406b32afe710485ba1ca267c6629542e: Beginning new tablet copy session on tablet 8058ebbaf0444a03a06a5cc261243a60 from peer 8c3e06132b5b44fb8eb4fc451662d466 at {username='jenkins'} at 127.5.240.194:36675: session id = 8c3e06132b5b44fb8eb4fc451662d466-8058ebbaf0444a03a06a5cc261243a60 I0703 01:41:23.921155 20901 tablet_copy_source_session.cc:202] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:41:23.922530 21330 ts_tablet_manager.cc:1369] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466: Deleting tablet data with delete state TABLET_DATA_COPYING I0703 01:41:23.926021 21330 ts_tablet_manager.cc:1382] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.308 I0703 01:41:23.926230 21330 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 8058ebbaf0444a03a06a5cc261243a60. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:41:23.927608 21330 ts_tablet_manager.cc:1173] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:41:23.927812 21330 tablet_copy_client.cc:566] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466: tablet copy: Starting download of 0 data blocks... I0703 01:41:23.928057 21330 tablet_copy_client.cc:529] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466: tablet copy: Starting download of 1 WAL segments... I0703 01:41:23.943174 21330 tablet_copy_client.cc:405] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:41:23.946192 21330 ts_tablet_manager.cc:1028] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466: Bootstrapping tablet I0703 01:41:23.946336 21330 tablet_bootstrap.cc:427] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466: Bootstrap starting. W0703 01:41:24.415282 20814 consensus_peers.cc:433] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e -> Peer 8c3e06132b5b44fb8eb4fc451662d466 (127.5.240.194:57030): Couldn't send request to peer 8c3e06132b5b44fb8eb4fc451662d466 for tablet 8058ebbaf0444a03a06a5cc261243a60. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 2 times. W0703 01:41:24.885404 20814 consensus_peers.cc:433] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e -> Peer 8c3e06132b5b44fb8eb4fc451662d466 (127.5.240.194:57030): Couldn't send request to peer 8c3e06132b5b44fb8eb4fc451662d466 for tablet 8058ebbaf0444a03a06a5cc261243a60. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 3 times. I0703 01:41:25.018929 21330 tablet_bootstrap.cc:427] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466: Bootstrap replayed 1/1 log segments. Stats: ops{read=1085 overwritten=0 applied=1085 ignored=0} inserts{seen=1075 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:41:25.019378 21330 tablet_bootstrap.cc:427] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466: Bootstrap complete. I0703 01:41:25.019681 21330 ts_tablet_manager.cc:1045] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466: Time spent bootstrapping tablet: real 1.073s user 0.974s sys 0.054s I0703 01:41:25.020575 21330 raft_consensus.cc:325] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 [term 3 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 1005 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } attrs { replace: true } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false } } peers { permanent_uuid: "86055af88ab04036878f3822fe571c07" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42232 } attrs { promote: false } } peers { permanent_uuid: "8c3e06132b5b44fb8eb4fc451662d466" member_type: NON_VOTER last_known_addr { host: "127.5.240.194" port: 57030 } attrs { promote: true } } I0703 01:41:25.020864 21330 raft_consensus.cc:628] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 8c3e06132b5b44fb8eb4fc451662d466, State: Initialized, Role: LEARNER I0703 01:41:25.021028 21330 consensus_queue.cc:227] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1085, Last appended: 3.1085, Last appended by leader: 1085, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 1005 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } attrs { replace: true } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false } } peers { permanent_uuid: "86055af88ab04036878f3822fe571c07" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42232 } attrs { promote: false } } peers { permanent_uuid: "8c3e06132b5b44fb8eb4fc451662d466" member_type: NON_VOTER last_known_addr { host: "127.5.240.194" port: 57030 } attrs { promote: true } } I0703 01:41:25.024351 21330 ts_tablet_manager.cc:1073] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466: Time spent starting tablet: real 0.005s user 0.002s sys 0.002s I0703 01:41:25.025416 20901 tablet_copy_service.cc:325] P 406b32afe710485ba1ca267c6629542e: Request end of tablet copy session 8c3e06132b5b44fb8eb4fc451662d466-8058ebbaf0444a03a06a5cc261243a60 received from {username='jenkins'} at 127.5.240.194:36675 I0703 01:41:25.025548 20901 tablet_copy_service.cc:413] P 406b32afe710485ba1ca267c6629542e: ending tablet copy session 8c3e06132b5b44fb8eb4fc451662d466-8058ebbaf0444a03a06a5cc261243a60 on tablet 8058ebbaf0444a03a06a5cc261243a60 with peer 8c3e06132b5b44fb8eb4fc451662d466 I0703 01:41:25.470212 20515 raft_consensus.cc:1026] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.1004->[3.1005-3.1388] Dedup: 3.1085->[3.1086-3.1388] I0703 01:41:25.731704 21319 raft_consensus.cc:890] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e: attempting to promote NON_VOTER 8c3e06132b5b44fb8eb4fc451662d466 to VOTER I0703 01:41:25.732908 21319 consensus_peers.cc:455] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e -> Peer 86055af88ab04036878f3822fe571c07 (127.5.240.193:42232): Closing peer: 86055af88ab04036878f3822fe571c07 I0703 01:41:25.732995 21319 consensus_peers.cc:455] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e -> Peer 8c3e06132b5b44fb8eb4fc451662d466 (127.5.240.194:57030): Closing peer: 8c3e06132b5b44fb8eb4fc451662d466 I0703 01:41:25.733048 21319 consensus_peers.cc:455] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e -> Peer d7efcfaf2a624349a32682447014e68f (127.5.240.195:41493): Closing peer: d7efcfaf2a624349a32682447014e68f I0703 01:41:25.733265 21319 consensus_queue.cc:204] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1434, Committed index: 1434, Last appended: 3.1435, Last appended by leader: 781, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 1436 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } attrs { replace: true } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false } } peers { permanent_uuid: "86055af88ab04036878f3822fe571c07" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42232 } attrs { promote: false } } peers { permanent_uuid: "8c3e06132b5b44fb8eb4fc451662d466" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 57030 } attrs { promote: false } } I0703 01:41:25.735544 20636 raft_consensus.cc:1081] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f [term 3 FOLLOWER]: Refusing update from remote peer 406b32afe710485ba1ca267c6629542e: Log matching property violated. Preceding OpId in replica: term: 3 index: 1434. Preceding OpId from leader: term: 3 index: 1436. (index mismatch) I0703 01:41:25.736124 21263 consensus_queue.cc:962] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1436, Last known committed idx: 1434, Time since last communication: 0.000s I0703 01:41:25.737942 20391 raft_consensus.cc:1081] T 8058ebbaf0444a03a06a5cc261243a60 P 86055af88ab04036878f3822fe571c07 [term 3 FOLLOWER]: Refusing update from remote peer 406b32afe710485ba1ca267c6629542e: Log matching property violated. Preceding OpId in replica: term: 3 index: 1435. Preceding OpId from leader: term: 3 index: 1436. (index mismatch) I0703 01:41:25.738587 21254 consensus_queue.cc:962] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [LEADER]: Connected to new peer: Peer: permanent_uuid: "86055af88ab04036878f3822fe571c07" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42232 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1436, Last known committed idx: 1434, Time since last communication: 0.000s I0703 01:41:25.741566 21263 raft_consensus.cc:2684] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [term 3 LEADER]: Committing config change with OpId 3.1436: config changed from index 1005 to 1436, 8c3e06132b5b44fb8eb4fc451662d466 (127.5.240.194) changed from NON_VOTER to VOTER. New config: { opid_index: 1436 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } attrs { replace: true } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false } } peers { permanent_uuid: "86055af88ab04036878f3822fe571c07" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42232 } attrs { promote: false } } peers { permanent_uuid: "8c3e06132b5b44fb8eb4fc451662d466" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 57030 } attrs { promote: false } } } I0703 01:41:25.743127 20391 raft_consensus.cc:2684] T 8058ebbaf0444a03a06a5cc261243a60 P 86055af88ab04036878f3822fe571c07 [term 3 FOLLOWER]: Committing config change with OpId 3.1436: config changed from index 1005 to 1436, 8c3e06132b5b44fb8eb4fc451662d466 (127.5.240.194) changed from NON_VOTER to VOTER. New config: { opid_index: 1436 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } attrs { replace: true } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false } } peers { permanent_uuid: "86055af88ab04036878f3822fe571c07" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42232 } attrs { promote: false } } peers { permanent_uuid: "8c3e06132b5b44fb8eb4fc451662d466" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 57030 } attrs { promote: false } } } I0703 01:41:25.745424 20514 raft_consensus.cc:1081] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 [term 3 LEARNER]: Refusing update from remote peer 406b32afe710485ba1ca267c6629542e: Log matching property violated. Preceding OpId in replica: term: 3 index: 1435. Preceding OpId from leader: term: 3 index: 1436. (index mismatch) I0703 01:41:25.746323 20268 catalog_manager.cc:3798] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e reported cstate change: config changed from index 1005 to 1436, 8c3e06132b5b44fb8eb4fc451662d466 (127.5.240.194) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "406b32afe710485ba1ca267c6629542e" committed_config { opid_index: 1436 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "86055af88ab04036878f3822fe571c07" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42232 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c3e06132b5b44fb8eb4fc451662d466" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 57030 } attrs { promote: false } health_report { overall_health: UNKNOWN } } } I0703 01:41:25.746321 20636 raft_consensus.cc:2684] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f [term 3 FOLLOWER]: Committing config change with OpId 3.1436: config changed from index 1005 to 1436, 8c3e06132b5b44fb8eb4fc451662d466 (127.5.240.194) changed from NON_VOTER to VOTER. New config: { opid_index: 1436 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } attrs { replace: true } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false } } peers { permanent_uuid: "86055af88ab04036878f3822fe571c07" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42232 } attrs { promote: false } } peers { permanent_uuid: "8c3e06132b5b44fb8eb4fc451662d466" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 57030 } attrs { promote: false } } } I0703 01:41:25.749223 21263 consensus_queue.cc:962] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c3e06132b5b44fb8eb4fc451662d466" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 57030 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1436, Last known committed idx: 1434, Time since last communication: 0.000s I0703 01:41:25.751776 20515 raft_consensus.cc:2684] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 [term 3 FOLLOWER]: Committing config change with OpId 3.1436: config changed from index 1005 to 1436, 8c3e06132b5b44fb8eb4fc451662d466 (127.5.240.194) changed from NON_VOTER to VOTER. New config: { opid_index: 1436 OBSOLETE_local: false peers { permanent_uuid: "d7efcfaf2a624349a32682447014e68f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 41493 } attrs { replace: true } } peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false } } peers { permanent_uuid: "86055af88ab04036878f3822fe571c07" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42232 } attrs { promote: false } } peers { permanent_uuid: "8c3e06132b5b44fb8eb4fc451662d466" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 57030 } attrs { promote: false } } } I0703 01:41:25.766587 20271 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 8058ebbaf0444a03a06a5cc261243a60 (attempt 1) I0703 01:41:25.767504 20881 consensus_peers.cc:455] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e -> Peer 86055af88ab04036878f3822fe571c07 (127.5.240.193:42232): Closing peer: 86055af88ab04036878f3822fe571c07 I0703 01:41:25.767596 20881 consensus_peers.cc:455] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e -> Peer 8c3e06132b5b44fb8eb4fc451662d466 (127.5.240.194:57030): Closing peer: 8c3e06132b5b44fb8eb4fc451662d466 I0703 01:41:25.767650 20881 consensus_peers.cc:455] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e -> Peer d7efcfaf2a624349a32682447014e68f (127.5.240.195:41493): Closing peer: d7efcfaf2a624349a32682447014e68f I0703 01:41:25.767869 20881 consensus_queue.cc:204] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1438, Committed index: 1438, Last appended: 3.1439, Last appended by leader: 781, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 1440 OBSOLETE_local: false peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false } } peers { permanent_uuid: "86055af88ab04036878f3822fe571c07" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42232 } attrs { promote: false } } peers { permanent_uuid: "8c3e06132b5b44fb8eb4fc451662d466" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 57030 } attrs { promote: false } } I0703 01:41:25.769407 20391 raft_consensus.cc:1081] T 8058ebbaf0444a03a06a5cc261243a60 P 86055af88ab04036878f3822fe571c07 [term 3 FOLLOWER]: Refusing update from remote peer 406b32afe710485ba1ca267c6629542e: Log matching property violated. Preceding OpId in replica: term: 3 index: 1439. Preceding OpId from leader: term: 3 index: 1440. (index mismatch) I0703 01:41:25.769649 20515 raft_consensus.cc:1081] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 [term 3 FOLLOWER]: Refusing update from remote peer 406b32afe710485ba1ca267c6629542e: Log matching property violated. Preceding OpId in replica: term: 3 index: 1439. Preceding OpId from leader: term: 3 index: 1440. (index mismatch) I0703 01:41:25.769984 21263 consensus_queue.cc:962] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [LEADER]: Connected to new peer: Peer: permanent_uuid: "86055af88ab04036878f3822fe571c07" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42232 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1440, Last known committed idx: 1438, Time since last communication: 0.000s I0703 01:41:25.770541 21263 consensus_queue.cc:962] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c3e06132b5b44fb8eb4fc451662d466" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 57030 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1440, Last known committed idx: 1438, Time since last communication: 0.000s I0703 01:41:25.772176 21254 raft_consensus.cc:2684] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e [term 3 LEADER]: Committing config change with OpId 3.1440: config changed from index 1436 to 1440, VOTER d7efcfaf2a624349a32682447014e68f (127.5.240.195) evicted. New config: { opid_index: 1440 OBSOLETE_local: false peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false } } peers { permanent_uuid: "86055af88ab04036878f3822fe571c07" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42232 } attrs { promote: false } } peers { permanent_uuid: "8c3e06132b5b44fb8eb4fc451662d466" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 57030 } attrs { promote: false } } } I0703 01:41:25.773407 20391 raft_consensus.cc:2684] T 8058ebbaf0444a03a06a5cc261243a60 P 86055af88ab04036878f3822fe571c07 [term 3 FOLLOWER]: Committing config change with OpId 3.1440: config changed from index 1436 to 1440, VOTER d7efcfaf2a624349a32682447014e68f (127.5.240.195) evicted. New config: { opid_index: 1440 OBSOLETE_local: false peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false } } peers { permanent_uuid: "86055af88ab04036878f3822fe571c07" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42232 } attrs { promote: false } } peers { permanent_uuid: "8c3e06132b5b44fb8eb4fc451662d466" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 57030 } attrs { promote: false } } } I0703 01:41:25.775496 20255 catalog_manager.cc:3352] ChangeConfig:REMOVE_PEER RPC for tablet 8058ebbaf0444a03a06a5cc261243a60 with cas_config_opid_index 1436: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0703 01:41:25.777007 20515 raft_consensus.cc:2684] T 8058ebbaf0444a03a06a5cc261243a60 P 8c3e06132b5b44fb8eb4fc451662d466 [term 3 FOLLOWER]: Committing config change with OpId 3.1440: config changed from index 1436 to 1440, VOTER d7efcfaf2a624349a32682447014e68f (127.5.240.195) evicted. New config: { opid_index: 1440 OBSOLETE_local: false peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false } } peers { permanent_uuid: "86055af88ab04036878f3822fe571c07" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42232 } attrs { promote: false } } peers { permanent_uuid: "8c3e06132b5b44fb8eb4fc451662d466" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 57030 } attrs { promote: false } } } I0703 01:41:25.777185 20271 catalog_manager.cc:3798] T 8058ebbaf0444a03a06a5cc261243a60 P 406b32afe710485ba1ca267c6629542e reported cstate change: config changed from index 1436 to 1440, VOTER d7efcfaf2a624349a32682447014e68f (127.5.240.195) evicted. New cstate: current_term: 3 leader_uuid: "406b32afe710485ba1ca267c6629542e" committed_config { opid_index: 1440 OBSOLETE_local: false peers { permanent_uuid: "406b32afe710485ba1ca267c6629542e" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 43025 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "86055af88ab04036878f3822fe571c07" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 42232 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c3e06132b5b44fb8eb4fc451662d466" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 57030 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I0703 01:41:25.779652 20271 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 8058ebbaf0444a03a06a5cc261243a60 on d7efcfaf2a624349a32682447014e68f (127.5.240.195:41493) (TS d7efcfaf2a624349a32682447014e68f not found in new config with opid_index 1440) I0703 01:41:25.780226 20617 tablet_service.cc:800] Processing DeleteTablet for tablet 8058ebbaf0444a03a06a5cc261243a60 with delete_type TABLET_DATA_TOMBSTONED (TS d7efcfaf2a624349a32682447014e68f not found in new config with opid_index 1440) from {username='jenkins'} at 127.5.240.195:35402 I0703 01:41:25.783509 21369 tablet_replica.cc:261] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f: stopping tablet replica I0703 01:41:25.783746 21369 maintenance_manager.cc:237] P d7efcfaf2a624349a32682447014e68f: Unregistered op CompactRowSetsOp(8058ebbaf0444a03a06a5cc261243a60) I0703 01:41:25.783865 21369 maintenance_manager.cc:237] P d7efcfaf2a624349a32682447014e68f: Unregistered op MinorDeltaCompactionOp(8058ebbaf0444a03a06a5cc261243a60) I0703 01:41:25.784198 21369 maintenance_manager.cc:237] P d7efcfaf2a624349a32682447014e68f: Unregistered op MajorDeltaCompactionOp(8058ebbaf0444a03a06a5cc261243a60) I0703 01:41:25.784297 21369 maintenance_manager.cc:237] P d7efcfaf2a624349a32682447014e68f: Unregistered op UndoDeltaBlockGCOp(8058ebbaf0444a03a06a5cc261243a60) I0703 01:41:25.784422 21369 maintenance_manager.cc:237] P d7efcfaf2a624349a32682447014e68f: Unregistered op FlushMRSOp(8058ebbaf0444a03a06a5cc261243a60) I0703 01:41:25.784503 21369 maintenance_manager.cc:237] P d7efcfaf2a624349a32682447014e68f: Unregistered op FlushDeltaMemStoresOp(8058ebbaf0444a03a06a5cc261243a60) I0703 01:41:25.784601 21369 maintenance_manager.cc:237] P d7efcfaf2a624349a32682447014e68f: Unregistered op LogGCOp(8058ebbaf0444a03a06a5cc261243a60) I0703 01:41:25.784710 21369 raft_consensus.cc:2040] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f [term 3 FOLLOWER]: Raft consensus shutting down. I0703 01:41:25.785192 21369 pending_rounds.cc:63] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f: Trying to abort 1 pending transactions. I0703 01:41:25.785270 21369 pending_rounds.cc:68] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f: Aborting transaction as it isn't in flight: id { term: 3 index: 1439 } timestamp: 6269367442487017472 op_type: WRITE_OP write_request { tablet_id: "8058ebbaf0444a03a06a5cc261243a60" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true } } row_operations { rows: "\001\007\000\034XW4\200\317z~\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "3521b0806d957d80" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 6269367442462068736 } request_id { client_id: "2499667f6bcc4c699e4f6bf5bd2dde19" seq_no: 1427 first_incomplete_seq_no: 1427 attempt_no: 0 } I0703 01:41:25.786442 21369 raft_consensus.cc:2067] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f [term 3 FOLLOWER]: Raft consensus is shut down! I0703 01:41:25.787731 21369 ts_tablet_manager.cc:1369] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:41:25.792516 21369 ts_tablet_manager.cc:1382] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.1439 I0703 01:41:25.792632 21369 log.cc:986] T 8058ebbaf0444a03a06a5cc261243a60 P d7efcfaf2a624349a32682447014e68f: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal/wals/8058ebbaf0444a03a06a5cc261243a60 I0703 01:41:25.793308 20255 catalog_manager.cc:3103] TS d7efcfaf2a624349a32682447014e68f (127.5.240.195:41493): tablet 8058ebbaf0444a03a06a5cc261243a60 (table TestTable [id=cca11239bdb84b6ca083a7a4f8892b07]) successfully deleted Master Summary UUID | Address | Status ----------------------------------+---------------------+--------- cab481b3b8e34cbbbbfec8ab4b0e9607 | 127.5.240.254:45994 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status ----------------------------------+---------------------+--------- 406b32afe710485ba1ca267c6629542e | 127.5.240.197:43025 | HEALTHY 86055af88ab04036878f3822fe571c07 | 127.5.240.193:42232 | HEALTHY 8c3e06132b5b44fb8eb4fc451662d466 | 127.5.240.194:57030 | HEALTHY a1e2e6b86a1747a4ae340383599dcff4 | 127.5.240.196:36708 | HEALTHY d7efcfaf2a624349a32682447014e68f | 127.5.240.195:41493 | HEALTHY Flag | Value | Tags | Tablet Server ------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_server_key_size | 1024 | experimental | all 5 server(s) checked local_ip_for_outbound_sockets | 127.5.240.193 | experimental | 127.5.240.193:42232 local_ip_for_outbound_sockets | 127.5.240.194 | experimental | 127.5.240.194:57030 local_ip_for_outbound_sockets | 127.5.240.195 | experimental | 127.5.240.195:41493 local_ip_for_outbound_sockets | 127.5.240.196 | experimental | 127.5.240.196:36708 local_ip_for_outbound_sockets | 127.5.240.197 | experimental | 127.5.240.197:43025 never_fsync | true | unsafe,advanced | all 5 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 5 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 5 server(s) checked server_dump_info_format | pb | hidden | all 5 server(s) checked server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.5.240.193:42232 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.240.194:57030 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.240.195:41493 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.5.240.196:36708 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.240.197:43025 Version Summary Version | Servers ----------------+------------------------- 1.8.0-SNAPSHOT | all 6 server(s) checked Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable -----------+----+---------+---------------+---------+------------+------------------+------------- TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0 | Total Count ----------------+------------- Masters | 1 Tablet Servers | 5 Tables | 1 Tablets | 1 Replicas | 3 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set OK I0703 01:41:26.389032 6083 log_verifier.cc:121] Checking tablet 8058ebbaf0444a03a06a5cc261243a60 I0703 01:41:27.283890 6083 log_verifier.cc:172] Verified matching terms for 1558 ops in tablet 8058ebbaf0444a03a06a5cc261243a60 I0703 01:41:27.288820 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 20306 I0703 01:41:27.291908 20432 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:41:27.385397 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 20306 W0703 01:41:27.399286 20691 connection.cc:512] client connection to 127.5.240.193:42232 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:42232): Connection reset by peer (error 104) W0703 01:41:27.399289 20253 connection.cc:512] server connection from 127.5.240.193:44264 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:44264): Connection reset by peer (error 104) W0703 01:41:27.399402 20569 connection.cc:512] client connection to 127.5.240.193:42232 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:42232): Connection reset by peer (error 104) W0703 01:41:27.399372 20571 connection.cc:512] server connection from 127.5.240.193:56936 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:56936): Connection reset by peer (error 104) W0703 01:41:27.399803 20693 connection.cc:512] server connection from 127.5.240.193:45823 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:45823): Connection reset by peer (error 104) W0703 01:41:27.399956 20812 connection.cc:512] server connection from 127.5.240.193:43463 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:43463): Connection reset by peer (error 104) W0703 01:41:27.399957 20813 connection.cc:512] client connection to 127.5.240.193:42232 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:42232): Connection reset by peer (error 104) I0703 01:41:27.400126 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 20436 I0703 01:41:27.400938 20555 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:41:27.499454 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 20436 W0703 01:41:27.513375 20253 connection.cc:512] server connection from 127.5.240.194:59409 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:59409): Connection reset by peer (error 104) W0703 01:41:27.513381 20691 connection.cc:512] server connection from 127.5.240.194:59611 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:59611): Connection reset by peer (error 104) W0703 01:41:27.513516 20571 connection.cc:512] server connection from 127.5.240.194:36890 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:36890): Connection reset by peer (error 104) W0703 01:41:27.513798 20570 connection.cc:512] client connection to 127.5.240.194:57030 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:57030): Connection reset by peer (error 104) W0703 01:41:27.513820 20692 connection.cc:512] client connection to 127.5.240.194:57030 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:57030): Connection reset by peer (error 104) W0703 01:41:27.513948 20814 connection.cc:512] client connection to 127.5.240.194:57030 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:57030): Connection reset by peer (error 104) W0703 01:41:27.513948 20815 connection.cc:512] server connection from 127.5.240.194:36675 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:36675): Connection reset by peer (error 104) I0703 01:41:27.514246 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 20561 I0703 01:41:27.514921 20677 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:41:27.623620 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 20561 W0703 01:41:27.640702 20256 connection.cc:512] server connection from 127.5.240.195:42571 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.195:42571): Connection reset by peer (error 104) W0703 01:41:27.640872 20692 connection.cc:512] client connection to 127.5.240.195:41493 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.195:41493): Connection reset by peer (error 104) W0703 01:41:27.640898 20814 connection.cc:512] client connection to 127.5.240.195:41493 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.195:41493): Connection reset by peer (error 104) W0703 01:41:27.641248 20692 connection.cc:512] server connection from 127.5.240.195:38096 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.195:38096): Connection reset by peer (error 104) W0703 01:41:27.641284 20814 connection.cc:512] server connection from 127.5.240.195:40528 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.195:40528): Connection reset by peer (error 104) I0703 01:41:27.641342 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 20683 I0703 01:41:27.642058 20799 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:41:27.740245 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 20683 W0703 01:41:27.755321 20812 connection.cc:512] client connection to 127.5.240.196:36708 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.196:36708): Connection reset by peer (error 104) W0703 01:41:27.755327 20254 connection.cc:512] server connection from 127.5.240.196:49233 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.196:49233): Connection reset by peer (error 104) W0703 01:41:27.755724 20814 connection.cc:512] server connection from 127.5.240.196:42903 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.196:42903): Connection reset by peer (error 104) I0703 01:41:27.755769 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 20804 I0703 01:41:27.756450 20921 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:41:27.887183 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 20804 W0703 01:41:27.906141 20253 connection.cc:512] server connection from 127.5.240.197:39644 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.197:39644): Connection reset by peer (error 104) I0703 01:41:27.906395 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master pid 20246 I0703 01:41:27.907122 20302 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:41:27.976760 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master with pid 20246 [ OK ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/3 (15943 ms) [ RUN ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4 I0703 01:41:28.003124 6083 test_util.cc:208] Using random seed: 1202792998 I0703 01:41:28.004967 6083 ts_itest-base.cc:113] Starting cluster with: I0703 01:41:28.005048 6083 ts_itest-base.cc:114] -------------- I0703 01:41:28.005081 6083 ts_itest-base.cc:115] 5 tablet servers I0703 01:41:28.005106 6083 ts_itest-base.cc:116] 3 replicas per TS I0703 01:41:28.005136 6083 ts_itest-base.cc:117] -------------- I0703 01:41:28.006247 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:36705 --webserver_interface=127.5.240.254 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:41:28.190754 21418 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:41:28.190980 21418 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:41:28.193189 21418 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0703 01:41:28.193251 21418 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0703 01:41:28.193296 21418 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:41:28.193337 21418 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0703 01:41:28.193389 21418 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0703 01:41:28.201454 21418 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:36705 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.5.240.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:41:28.207556 21418 master_main.cc:80] Initializing master server... I0703 01:41:28.208274 21418 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 197708us I0703 01:41:28.209362 21418 fs_manager.cc:261] Metadata directory not provided I0703 01:41:28.209430 21418 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:41:28.209697 21418 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:41:28.209769 21418 server_base.cc:433] Attempting to create new FS layout instead I0703 01:41:28.213920 21418 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/instance: uuid: "bbcc1b1575db4565820c9db0db6825cb" format_stamp: "Formatted at 2018-07-03 08:41:28 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:28.214434 21418 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "bbcc1b1575db4565820c9db0db6825cb" format_stamp: "Formatted at 2018-07-03 08:41:28 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:28.222697 21418 fs_manager.cc:501] Time spent creating directory manager: real 0.008s user 0.003s sys 0.004s I0703 01:41:28.223958 21418 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:41:28.224369 21418 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:41:28.228082 21418 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0703 01:41:28.228137 21418 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal uuid: "bbcc1b1575db4565820c9db0db6825cb" format_stamp: "Formatted at 2018-07-03 08:41:28 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:28.228260 21418 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:41:28.275940 21418 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:41:28.277374 21418 master_main.cc:83] Starting Master server... I0703 01:41:28.314198 21418 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.254:36705 I0703 01:41:28.314673 21418 webserver.cc:175] Starting webserver on 127.5.240.254:0 I0703 01:41:28.314713 21418 webserver.cc:186] Document root disabled I0703 01:41:28.315413 21418 webserver.cc:313] Webserver started. Bound to: http://127.5.240.254:50852/ I0703 01:41:28.315999 21418 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb I0703 01:41:28.319653 21477 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:41:28.320745 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master as pid 21418 I0703 01:41:28.321349 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.5.240.193:0 --local_ip_for_outbound_sockets=127.5.240.193 --webserver_interface=127.5.240.193 --webserver_port=0 --tserver_master_addrs=127.5.240.254:36705 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0703 01:41:28.331141 21477 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P bbcc1b1575db4565820c9db0db6825cb: Bootstrap starting. I0703 01:41:28.333185 21477 tablet_bootstrap.cc:578] T 00000000000000000000000000000000 P bbcc1b1575db4565820c9db0db6825cb: No blocks or log segments found. Creating new log. I0703 01:41:28.333945 21477 log.cc:525] T 00000000000000000000000000000000 P bbcc1b1575db4565820c9db0db6825cb: Log is configured to *not* fsync() on all Append() calls I0703 01:41:28.335795 21477 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P bbcc1b1575db4565820c9db0db6825cb: No bootstrap required, opened a new log I0703 01:41:28.342177 21477 raft_consensus.cc:325] T 00000000000000000000000000000000 P bbcc1b1575db4565820c9db0db6825cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbcc1b1575db4565820c9db0db6825cb" member_type: VOTER } I0703 01:41:28.342373 21477 raft_consensus.cc:351] T 00000000000000000000000000000000 P bbcc1b1575db4565820c9db0db6825cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:41:28.342434 21477 raft_consensus.cc:628] T 00000000000000000000000000000000 P bbcc1b1575db4565820c9db0db6825cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbcc1b1575db4565820c9db0db6825cb, State: Initialized, Role: FOLLOWER I0703 01:41:28.342906 21477 consensus_queue.cc:227] T 00000000000000000000000000000000 P bbcc1b1575db4565820c9db0db6825cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbcc1b1575db4565820c9db0db6825cb" member_type: VOTER } I0703 01:41:28.343050 21477 raft_consensus.cc:365] T 00000000000000000000000000000000 P bbcc1b1575db4565820c9db0db6825cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:41:28.343139 21477 raft_consensus.cc:457] T 00000000000000000000000000000000 P bbcc1b1575db4565820c9db0db6825cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:41:28.343226 21477 raft_consensus.cc:2730] T 00000000000000000000000000000000 P bbcc1b1575db4565820c9db0db6825cb: Snoozing failure detection for 1.510s (starting election) I0703 01:41:28.343317 21477 raft_consensus.cc:2782] T 00000000000000000000000000000000 P bbcc1b1575db4565820c9db0db6825cb [term 0 FOLLOWER]: Advancing to term 1 I0703 01:41:28.344677 21477 raft_consensus.cc:479] T 00000000000000000000000000000000 P bbcc1b1575db4565820c9db0db6825cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbcc1b1575db4565820c9db0db6825cb" member_type: VOTER } I0703 01:41:28.345042 21477 leader_election.cc:270] T 00000000000000000000000000000000 P bbcc1b1575db4565820c9db0db6825cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:41:28.346500 21480 raft_consensus.cc:2730] T 00000000000000000000000000000000 P bbcc1b1575db4565820c9db0db6825cb: Snoozing failure detection for 2.151s (election complete) I0703 01:41:28.346781 21480 raft_consensus.cc:2537] T 00000000000000000000000000000000 P bbcc1b1575db4565820c9db0db6825cb [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:41:28.348279 21480 raft_consensus.cc:593] T 00000000000000000000000000000000 P bbcc1b1575db4565820c9db0db6825cb [term 1 LEADER]: Becoming Leader. State: Replica: bbcc1b1575db4565820c9db0db6825cb, State: Running, Role: LEADER I0703 01:41:28.348667 21480 consensus_queue.cc:204] T 00000000000000000000000000000000 P bbcc1b1575db4565820c9db0db6825cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbcc1b1575db4565820c9db0db6825cb" member_type: VOTER } I0703 01:41:28.351429 21477 sys_catalog.cc:337] T 00000000000000000000000000000000 P bbcc1b1575db4565820c9db0db6825cb [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "bbcc1b1575db4565820c9db0db6825cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbcc1b1575db4565820c9db0db6825cb" member_type: VOTER } } I0703 01:41:28.351915 21477 sys_catalog.cc:340] T 00000000000000000000000000000000 P bbcc1b1575db4565820c9db0db6825cb [sys.catalog]: This master's current role is: LEADER I0703 01:41:28.351655 21482 sys_catalog.cc:337] T 00000000000000000000000000000000 P bbcc1b1575db4565820c9db0db6825cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbcc1b1575db4565820c9db0db6825cb. Latest consensus state: current_term: 1 leader_uuid: "bbcc1b1575db4565820c9db0db6825cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbcc1b1575db4565820c9db0db6825cb" member_type: VOTER } } I0703 01:41:28.351657 21481 sys_catalog.cc:337] T 00000000000000000000000000000000 P bbcc1b1575db4565820c9db0db6825cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bbcc1b1575db4565820c9db0db6825cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbcc1b1575db4565820c9db0db6825cb" member_type: VOTER } } I0703 01:41:28.352954 21481 sys_catalog.cc:340] T 00000000000000000000000000000000 P bbcc1b1575db4565820c9db0db6825cb [sys.catalog]: This master's current role is: LEADER I0703 01:41:28.353163 21482 sys_catalog.cc:340] T 00000000000000000000000000000000 P bbcc1b1575db4565820c9db0db6825cb [sys.catalog]: This master's current role is: LEADER I0703 01:41:28.355566 21477 sys_catalog.cc:424] T 00000000000000000000000000000000 P bbcc1b1575db4565820c9db0db6825cb [sys.catalog]: configured and running, proceeding with master startup. I0703 01:41:28.355917 21484 catalog_manager.cc:1006] Loading table and tablet metadata into memory... I0703 01:41:28.361155 21485 catalog_manager.cc:505] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting I0703 01:41:28.361636 21418 master_main.cc:86] Master server successfully started. I0703 01:41:28.366704 21484 catalog_manager.cc:1017] Initializing Kudu internal certificate authority... I0703 01:41:28.449748 21484 catalog_manager.cc:904] Generated new certificate authority record I0703 01:41:28.451740 21484 catalog_manager.cc:1026] Loading token signing keys... I0703 01:41:28.463362 21484 catalog_manager.cc:4102] T 00000000000000000000000000000000 P bbcc1b1575db4565820c9db0db6825cb: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:41:28.507211 21478 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:41:28.507423 21478 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:41:28.509419 21478 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0703 01:41:28.509481 21478 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:41:28.509609 21478 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.193 I0703 01:41:28.516603 21478 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.5.240.193 --webserver_port=0 --tserver_master_addrs=127.5.240.254:36705 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.193 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:41:28.523149 21478 tablet_server_main.cc:85] Initializing tablet server... I0703 01:41:28.524714 21478 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 197708us I0703 01:41:28.525279 21478 fs_manager.cc:261] Metadata directory not provided I0703 01:41:28.525348 21478 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:41:28.525601 21478 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:41:28.525673 21478 server_base.cc:433] Attempting to create new FS layout instead I0703 01:41:28.529801 21478 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "51800c5e0fd34699836c1611adecaf5b" format_stamp: "Formatted at 2018-07-03 08:41:28 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:28.530299 21478 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "51800c5e0fd34699836c1611adecaf5b" format_stamp: "Formatted at 2018-07-03 08:41:28 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:28.538420 21478 fs_manager.cc:501] Time spent creating directory manager: real 0.008s user 0.004s sys 0.001s I0703 01:41:28.539664 21478 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:41:28.540073 21478 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:41:28.543658 21478 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s I0703 01:41:28.543709 21478 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal uuid: "51800c5e0fd34699836c1611adecaf5b" format_stamp: "Formatted at 2018-07-03 08:41:28 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:28.543843 21478 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:41:28.573981 21478 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:41:28.575944 21478 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:41:28.576294 21478 tablet_server_main.cc:90] Starting tablet server... I0703 01:41:28.683136 21478 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.193:42165 I0703 01:41:28.683624 21478 webserver.cc:175] Starting webserver on 127.5.240.193:0 I0703 01:41:28.683670 21478 webserver.cc:186] Document root disabled I0703 01:41:28.685000 21478 webserver.cc:313] Webserver started. Bound to: http://127.5.240.193:48616/ I0703 01:41:28.685534 21478 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb I0703 01:41:28.687381 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 21478 I0703 01:41:28.688252 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.5.240.194:0 --local_ip_for_outbound_sockets=127.5.240.194 --webserver_interface=127.5.240.194 --webserver_port=0 --tserver_master_addrs=127.5.240.254:36705 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0703 01:41:28.698442 21478 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:41:28.721120 21607 heartbeater.cc:346] Connected to a master server at 127.5.240.254:36705 I0703 01:41:28.721276 21607 heartbeater.cc:426] Registering TS with master... I0703 01:41:28.722081 21607 heartbeater.cc:475] Master 127.5.240.254:36705 requested a full tablet report, sending... I0703 01:41:28.723649 21444 ts_manager.cc:98] Registered new tserver with Master: 51800c5e0fd34699836c1611adecaf5b (127.5.240.193:42165) I0703 01:41:28.725436 21444 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.193:49006 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:41:28.862407 21609 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:41:28.862637 21609 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:41:28.864610 21609 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0703 01:41:28.864666 21609 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:41:28.864753 21609 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.194 I0703 01:41:28.871166 21609 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.194:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.5.240.194 --webserver_port=0 --tserver_master_addrs=127.5.240.254:36705 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.194 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:41:28.877725 21609 tablet_server_main.cc:85] Initializing tablet server... I0703 01:41:28.879231 21609 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 197708us I0703 01:41:28.879807 21609 fs_manager.cc:261] Metadata directory not provided I0703 01:41:28.879868 21609 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:41:28.880076 21609 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:41:28.880137 21609 server_base.cc:433] Attempting to create new FS layout instead I0703 01:41:28.884213 21609 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "4289d8404d9e416db70b1b9069cf7231" format_stamp: "Formatted at 2018-07-03 08:41:28 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:28.884775 21609 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "4289d8404d9e416db70b1b9069cf7231" format_stamp: "Formatted at 2018-07-03 08:41:28 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:28.891970 21609 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.003s sys 0.003s I0703 01:41:28.893137 21609 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:41:28.893491 21609 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:41:28.897080 21609 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s I0703 01:41:28.897127 21609 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal uuid: "4289d8404d9e416db70b1b9069cf7231" format_stamp: "Formatted at 2018-07-03 08:41:28 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:28.897243 21609 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:41:28.966379 21609 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:41:28.968736 21609 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:41:28.969074 21609 tablet_server_main.cc:90] Starting tablet server... I0703 01:41:29.053663 21609 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.194:33446 I0703 01:41:29.054086 21609 webserver.cc:175] Starting webserver on 127.5.240.194:0 I0703 01:41:29.054131 21609 webserver.cc:186] Document root disabled I0703 01:41:29.054810 21609 webserver.cc:313] Webserver started. Bound to: http://127.5.240.194:54055/ I0703 01:41:29.055335 21609 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb I0703 01:41:29.057649 21609 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:41:29.063805 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 21609 I0703 01:41:29.064405 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.5.240.195:0 --local_ip_for_outbound_sockets=127.5.240.195 --webserver_interface=127.5.240.195 --webserver_port=0 --tserver_master_addrs=127.5.240.254:36705 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0703 01:41:29.069414 21730 heartbeater.cc:346] Connected to a master server at 127.5.240.254:36705 I0703 01:41:29.069594 21730 heartbeater.cc:426] Registering TS with master... I0703 01:41:29.070515 21730 heartbeater.cc:475] Master 127.5.240.254:36705 requested a full tablet report, sending... I0703 01:41:29.071575 21444 ts_manager.cc:98] Registered new tserver with Master: 4289d8404d9e416db70b1b9069cf7231 (127.5.240.194:33446) I0703 01:41:29.072414 21444 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.194:48000 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:41:29.227562 21733 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:41:29.227783 21733 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:41:29.229811 21733 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0703 01:41:29.229859 21733 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:41:29.229941 21733 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.195 I0703 01:41:29.236480 21733 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.195:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.5.240.195 --webserver_port=0 --tserver_master_addrs=127.5.240.254:36705 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.195 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:41:29.243091 21733 tablet_server_main.cc:85] Initializing tablet server... I0703 01:41:29.244619 21733 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 198208us I0703 01:41:29.245175 21733 fs_manager.cc:261] Metadata directory not provided I0703 01:41:29.245236 21733 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:41:29.245859 21733 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:41:29.245924 21733 server_base.cc:433] Attempting to create new FS layout instead I0703 01:41:29.250030 21733 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "eac7487faa4344c1b3edaac9ab871ac7" format_stamp: "Formatted at 2018-07-03 08:41:29 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:29.250576 21733 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "eac7487faa4344c1b3edaac9ab871ac7" format_stamp: "Formatted at 2018-07-03 08:41:29 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:29.258572 21733 fs_manager.cc:501] Time spent creating directory manager: real 0.008s user 0.003s sys 0.004s I0703 01:41:29.259788 21733 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:41:29.260200 21733 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:41:29.263314 21733 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0703 01:41:29.263358 21733 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal uuid: "eac7487faa4344c1b3edaac9ab871ac7" format_stamp: "Formatted at 2018-07-03 08:41:29 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:29.263480 21733 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:41:29.301841 21733 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:41:29.304091 21733 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:41:29.304420 21733 tablet_server_main.cc:90] Starting tablet server... I0703 01:41:29.388885 21733 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.195:60517 I0703 01:41:29.389297 21733 webserver.cc:175] Starting webserver on 127.5.240.195:0 I0703 01:41:29.389335 21733 webserver.cc:186] Document root disabled I0703 01:41:29.390038 21733 webserver.cc:313] Webserver started. Bound to: http://127.5.240.195:49929/ I0703 01:41:29.390579 21733 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb I0703 01:41:29.392593 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 21733 I0703 01:41:29.393445 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.5.240.196:0 --local_ip_for_outbound_sockets=127.5.240.196 --webserver_interface=127.5.240.196 --webserver_port=0 --tserver_master_addrs=127.5.240.254:36705 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0703 01:41:29.403312 21733 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:41:29.415733 21852 heartbeater.cc:346] Connected to a master server at 127.5.240.254:36705 I0703 01:41:29.415882 21852 heartbeater.cc:426] Registering TS with master... I0703 01:41:29.416685 21852 heartbeater.cc:475] Master 127.5.240.254:36705 requested a full tablet report, sending... I0703 01:41:29.417696 21444 ts_manager.cc:98] Registered new tserver with Master: eac7487faa4344c1b3edaac9ab871ac7 (127.5.240.195:60517) I0703 01:41:29.418515 21444 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.195:58285 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:41:29.561609 21854 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:41:29.561861 21854 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:41:29.563868 21854 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0703 01:41:29.563918 21854 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:41:29.564007 21854 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.196 I0703 01:41:29.570564 21854 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.196:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb --webserver_interface=127.5.240.196 --webserver_port=0 --tserver_master_addrs=127.5.240.254:36705 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.196 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:41:29.577255 21854 tablet_server_main.cc:85] Initializing tablet server... I0703 01:41:29.578768 21854 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 198208us I0703 01:41:29.579321 21854 fs_manager.cc:261] Metadata directory not provided I0703 01:41:29.579380 21854 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:41:29.579587 21854 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:41:29.579672 21854 server_base.cc:433] Attempting to create new FS layout instead I0703 01:41:29.583717 21854 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/instance: uuid: "0668a9f7c6514ec296857579ea1e0c21" format_stamp: "Formatted at 2018-07-03 08:41:29 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:29.584239 21854 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal/instance: uuid: "0668a9f7c6514ec296857579ea1e0c21" format_stamp: "Formatted at 2018-07-03 08:41:29 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:29.591552 21854 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.004s sys 0.002s I0703 01:41:29.592772 21854 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:41:29.593140 21854 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:41:29.596735 21854 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s I0703 01:41:29.596779 21854 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal uuid: "0668a9f7c6514ec296857579ea1e0c21" format_stamp: "Formatted at 2018-07-03 08:41:29 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:29.596901 21854 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:41:29.630316 21854 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:41:29.632652 21854 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:41:29.632982 21854 tablet_server_main.cc:90] Starting tablet server... I0703 01:41:29.717962 21854 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.196:53006 I0703 01:41:29.718387 21854 webserver.cc:175] Starting webserver on 127.5.240.196:0 I0703 01:41:29.718432 21854 webserver.cc:186] Document root disabled I0703 01:41:29.719640 21854 webserver.cc:313] Webserver started. Bound to: http://127.5.240.196:51215/ I0703 01:41:29.720175 21854 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb I0703 01:41:29.721993 21854 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:41:29.725301 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 21854 I0703 01:41:29.726254 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data --block_manager=log --rpc_bind_addresses=127.5.240.197:0 --local_ip_for_outbound_sockets=127.5.240.197 --webserver_interface=127.5.240.197 --webserver_port=0 --tserver_master_addrs=127.5.240.254:36705 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0703 01:41:29.727525 21607 heartbeater.cc:467] Master 127.5.240.254:36705 was elected leader, sending a full tablet report... I0703 01:41:29.736145 21974 heartbeater.cc:346] Connected to a master server at 127.5.240.254:36705 I0703 01:41:29.736308 21974 heartbeater.cc:426] Registering TS with master... I0703 01:41:29.737216 21974 heartbeater.cc:475] Master 127.5.240.254:36705 requested a full tablet report, sending... I0703 01:41:29.738351 21444 ts_manager.cc:98] Registered new tserver with Master: 0668a9f7c6514ec296857579ea1e0c21 (127.5.240.196:53006) I0703 01:41:29.739480 21444 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.196:51319 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:41:29.893705 21977 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:41:29.893910 21977 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:41:29.895895 21977 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0703 01:41:29.895943 21977 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:41:29.896028 21977 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.197 I0703 01:41:29.902458 21977 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.197:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb --webserver_interface=127.5.240.197 --webserver_port=0 --tserver_master_addrs=127.5.240.254:36705 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.197 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:41:29.909081 21977 tablet_server_main.cc:85] Initializing tablet server... I0703 01:41:29.910605 21977 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 198208us I0703 01:41:29.911149 21977 fs_manager.cc:261] Metadata directory not provided I0703 01:41:29.911204 21977 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:41:29.911806 21977 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:41:29.911867 21977 server_base.cc:433] Attempting to create new FS layout instead I0703 01:41:29.915947 21977 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/instance: uuid: "820457af06944ac7966bf2fab634de94" format_stamp: "Formatted at 2018-07-03 08:41:29 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:29.916465 21977 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal/instance: uuid: "820457af06944ac7966bf2fab634de94" format_stamp: "Formatted at 2018-07-03 08:41:29 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:29.924233 21977 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.004s sys 0.002s I0703 01:41:29.925424 21977 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:41:29.925818 21977 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:41:29.928892 21977 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0703 01:41:29.928943 21977 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal uuid: "820457af06944ac7966bf2fab634de94" format_stamp: "Formatted at 2018-07-03 08:41:29 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:29.929065 21977 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:41:29.963686 21977 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:41:29.966114 21977 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:41:29.966516 21977 tablet_server_main.cc:90] Starting tablet server... I0703 01:41:30.055196 21977 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.197:44086 I0703 01:41:30.055604 21977 webserver.cc:175] Starting webserver on 127.5.240.197:0 I0703 01:41:30.055649 21977 webserver.cc:186] Document root disabled I0703 01:41:30.056350 21977 webserver.cc:313] Webserver started. Bound to: http://127.5.240.197:51770/ I0703 01:41:30.056895 21977 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb I0703 01:41:30.058667 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 21977 I0703 01:41:30.061134 21977 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:41:30.073778 22096 heartbeater.cc:346] Connected to a master server at 127.5.240.254:36705 I0703 01:41:30.073920 22096 heartbeater.cc:426] Registering TS with master... I0703 01:41:30.074592 21730 heartbeater.cc:467] Master 127.5.240.254:36705 was elected leader, sending a full tablet report... I0703 01:41:30.074765 22096 heartbeater.cc:475] Master 127.5.240.254:36705 requested a full tablet report, sending... I0703 01:41:30.075883 21443 ts_manager.cc:98] Registered new tserver with Master: 820457af06944ac7966bf2fab634de94 (127.5.240.197:44086) I0703 01:41:30.076915 21443 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.197:58236 I0703 01:41:30.077102 6083 external_mini_cluster.cc:490] 5 TS(s) registered with all masters I0703 01:41:30.086462 21443 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.5.240.254:54380: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0703 01:41:30.109949 21911 tablet_service.cc:759] Processing CreateTablet for tablet cdf4190556a244ef8e614a243281e9a3 (table=TestTable [id=80f8c4eec3b043ccb6b202fd7dcfb127]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:41:30.110821 21911 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet cdf4190556a244ef8e614a243281e9a3. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:41:30.116143 22033 tablet_service.cc:759] Processing CreateTablet for tablet cdf4190556a244ef8e614a243281e9a3 (table=TestTable [id=80f8c4eec3b043ccb6b202fd7dcfb127]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:41:30.116813 22033 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet cdf4190556a244ef8e614a243281e9a3. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:41:30.117295 21667 tablet_service.cc:759] Processing CreateTablet for tablet cdf4190556a244ef8e614a243281e9a3 (table=TestTable [id=80f8c4eec3b043ccb6b202fd7dcfb127]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:41:30.118046 21667 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet cdf4190556a244ef8e614a243281e9a3. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:41:30.124933 21911 ts_tablet_manager.cc:1173] T cdf4190556a244ef8e614a243281e9a3 P 0668a9f7c6514ec296857579ea1e0c21: Registered tablet (data state: TABLET_DATA_READY) I0703 01:41:30.126425 22115 ts_tablet_manager.cc:1028] T cdf4190556a244ef8e614a243281e9a3 P 0668a9f7c6514ec296857579ea1e0c21: Bootstrapping tablet I0703 01:41:30.126687 22033 ts_tablet_manager.cc:1173] T cdf4190556a244ef8e614a243281e9a3 P 820457af06944ac7966bf2fab634de94: Registered tablet (data state: TABLET_DATA_READY) I0703 01:41:30.127423 22115 tablet_bootstrap.cc:427] T cdf4190556a244ef8e614a243281e9a3 P 0668a9f7c6514ec296857579ea1e0c21: Bootstrap starting. I0703 01:41:30.128136 22116 ts_tablet_manager.cc:1028] T cdf4190556a244ef8e614a243281e9a3 P 820457af06944ac7966bf2fab634de94: Bootstrapping tablet I0703 01:41:30.128504 22116 tablet_bootstrap.cc:427] T cdf4190556a244ef8e614a243281e9a3 P 820457af06944ac7966bf2fab634de94: Bootstrap starting. I0703 01:41:30.130661 22115 tablet_bootstrap.cc:578] T cdf4190556a244ef8e614a243281e9a3 P 0668a9f7c6514ec296857579ea1e0c21: No blocks or log segments found. Creating new log. I0703 01:41:30.131403 22115 log.cc:525] T cdf4190556a244ef8e614a243281e9a3 P 0668a9f7c6514ec296857579ea1e0c21: Log is configured to *not* fsync() on all Append() calls I0703 01:41:30.131542 22116 tablet_bootstrap.cc:578] T cdf4190556a244ef8e614a243281e9a3 P 820457af06944ac7966bf2fab634de94: No blocks or log segments found. Creating new log. I0703 01:41:30.132454 22116 log.cc:525] T cdf4190556a244ef8e614a243281e9a3 P 820457af06944ac7966bf2fab634de94: Log is configured to *not* fsync() on all Append() calls I0703 01:41:30.132892 21667 ts_tablet_manager.cc:1173] T cdf4190556a244ef8e614a243281e9a3 P 4289d8404d9e416db70b1b9069cf7231: Registered tablet (data state: TABLET_DATA_READY) I0703 01:41:30.133647 22119 ts_tablet_manager.cc:1028] T cdf4190556a244ef8e614a243281e9a3 P 4289d8404d9e416db70b1b9069cf7231: Bootstrapping tablet I0703 01:41:30.133901 22119 tablet_bootstrap.cc:427] T cdf4190556a244ef8e614a243281e9a3 P 4289d8404d9e416db70b1b9069cf7231: Bootstrap starting. I0703 01:41:30.136337 22119 tablet_bootstrap.cc:578] T cdf4190556a244ef8e614a243281e9a3 P 4289d8404d9e416db70b1b9069cf7231: No blocks or log segments found. Creating new log. I0703 01:41:30.136520 22116 tablet_bootstrap.cc:427] T cdf4190556a244ef8e614a243281e9a3 P 820457af06944ac7966bf2fab634de94: No bootstrap required, opened a new log I0703 01:41:30.136783 22116 ts_tablet_manager.cc:1045] T cdf4190556a244ef8e614a243281e9a3 P 820457af06944ac7966bf2fab634de94: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s I0703 01:41:30.137169 22115 tablet_bootstrap.cc:427] T cdf4190556a244ef8e614a243281e9a3 P 0668a9f7c6514ec296857579ea1e0c21: No bootstrap required, opened a new log I0703 01:41:30.137301 22119 log.cc:525] T cdf4190556a244ef8e614a243281e9a3 P 4289d8404d9e416db70b1b9069cf7231: Log is configured to *not* fsync() on all Append() calls I0703 01:41:30.137373 22115 ts_tablet_manager.cc:1045] T cdf4190556a244ef8e614a243281e9a3 P 0668a9f7c6514ec296857579ea1e0c21: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.006s I0703 01:41:30.139082 22119 tablet_bootstrap.cc:427] T cdf4190556a244ef8e614a243281e9a3 P 4289d8404d9e416db70b1b9069cf7231: No bootstrap required, opened a new log I0703 01:41:30.139269 22119 ts_tablet_manager.cc:1045] T cdf4190556a244ef8e614a243281e9a3 P 4289d8404d9e416db70b1b9069cf7231: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s I0703 01:41:30.143399 22116 raft_consensus.cc:325] T cdf4190556a244ef8e614a243281e9a3 P 820457af06944ac7966bf2fab634de94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0668a9f7c6514ec296857579ea1e0c21" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 53006 } } peers { permanent_uuid: "4289d8404d9e416db70b1b9069cf7231" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 33446 } } peers { permanent_uuid: "820457af06944ac7966bf2fab634de94" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 44086 } } I0703 01:41:30.143736 22116 raft_consensus.cc:351] T cdf4190556a244ef8e614a243281e9a3 P 820457af06944ac7966bf2fab634de94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:41:30.143821 22116 raft_consensus.cc:628] T cdf4190556a244ef8e614a243281e9a3 P 820457af06944ac7966bf2fab634de94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 820457af06944ac7966bf2fab634de94, State: Initialized, Role: FOLLOWER I0703 01:41:30.144284 22116 consensus_queue.cc:227] T cdf4190556a244ef8e614a243281e9a3 P 820457af06944ac7966bf2fab634de94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0668a9f7c6514ec296857579ea1e0c21" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 53006 } } peers { permanent_uuid: "4289d8404d9e416db70b1b9069cf7231" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 33446 } } peers { permanent_uuid: "820457af06944ac7966bf2fab634de94" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 44086 } } I0703 01:41:30.145700 22096 heartbeater.cc:467] Master 127.5.240.254:36705 was elected leader, sending a full tablet report... I0703 01:41:30.146324 22116 ts_tablet_manager.cc:1073] T cdf4190556a244ef8e614a243281e9a3 P 820457af06944ac7966bf2fab634de94: Time spent starting tablet: real 0.009s user 0.007s sys 0.002s I0703 01:41:30.146320 22115 raft_consensus.cc:325] T cdf4190556a244ef8e614a243281e9a3 P 0668a9f7c6514ec296857579ea1e0c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0668a9f7c6514ec296857579ea1e0c21" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 53006 } } peers { permanent_uuid: "4289d8404d9e416db70b1b9069cf7231" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 33446 } } peers { permanent_uuid: "820457af06944ac7966bf2fab634de94" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 44086 } } I0703 01:41:30.146730 22115 raft_consensus.cc:351] T cdf4190556a244ef8e614a243281e9a3 P 0668a9f7c6514ec296857579ea1e0c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:41:30.146806 22115 raft_consensus.cc:628] T cdf4190556a244ef8e614a243281e9a3 P 0668a9f7c6514ec296857579ea1e0c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0668a9f7c6514ec296857579ea1e0c21, State: Initialized, Role: FOLLOWER I0703 01:41:30.147207 22115 consensus_queue.cc:227] T cdf4190556a244ef8e614a243281e9a3 P 0668a9f7c6514ec296857579ea1e0c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0668a9f7c6514ec296857579ea1e0c21" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 53006 } } peers { permanent_uuid: "4289d8404d9e416db70b1b9069cf7231" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 33446 } } peers { permanent_uuid: "820457af06944ac7966bf2fab634de94" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 44086 } } I0703 01:41:30.148346 22119 raft_consensus.cc:325] T cdf4190556a244ef8e614a243281e9a3 P 4289d8404d9e416db70b1b9069cf7231 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0668a9f7c6514ec296857579ea1e0c21" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 53006 } } peers { permanent_uuid: "4289d8404d9e416db70b1b9069cf7231" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 33446 } } peers { permanent_uuid: "820457af06944ac7966bf2fab634de94" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 44086 } } I0703 01:41:30.148707 22119 raft_consensus.cc:351] T cdf4190556a244ef8e614a243281e9a3 P 4289d8404d9e416db70b1b9069cf7231 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:41:30.148766 22119 raft_consensus.cc:628] T cdf4190556a244ef8e614a243281e9a3 P 4289d8404d9e416db70b1b9069cf7231 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4289d8404d9e416db70b1b9069cf7231, State: Initialized, Role: FOLLOWER I0703 01:41:30.149179 22119 consensus_queue.cc:227] T cdf4190556a244ef8e614a243281e9a3 P 4289d8404d9e416db70b1b9069cf7231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0668a9f7c6514ec296857579ea1e0c21" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 53006 } } peers { permanent_uuid: "4289d8404d9e416db70b1b9069cf7231" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 33446 } } peers { permanent_uuid: "820457af06944ac7966bf2fab634de94" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 44086 } } I0703 01:41:30.150522 21974 heartbeater.cc:467] Master 127.5.240.254:36705 was elected leader, sending a full tablet report... I0703 01:41:30.151422 22115 ts_tablet_manager.cc:1073] T cdf4190556a244ef8e614a243281e9a3 P 0668a9f7c6514ec296857579ea1e0c21: Time spent starting tablet: real 0.014s user 0.008s sys 0.005s I0703 01:41:30.153044 22119 ts_tablet_manager.cc:1073] T cdf4190556a244ef8e614a243281e9a3 P 4289d8404d9e416db70b1b9069cf7231: Time spent starting tablet: real 0.014s user 0.009s sys 0.005s W0703 01:41:30.222579 21975 tablet.cc:1678] T cdf4190556a244ef8e614a243281e9a3 P 0668a9f7c6514ec296857579ea1e0c21: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:41:30.244943 22122 raft_consensus.cc:457] T cdf4190556a244ef8e614a243281e9a3 P 0668a9f7c6514ec296857579ea1e0c21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0703 01:41:30.245036 22122 raft_consensus.cc:2730] T cdf4190556a244ef8e614a243281e9a3 P 0668a9f7c6514ec296857579ea1e0c21: Snoozing failure detection for 2.232s (starting election) I0703 01:41:30.245157 22122 raft_consensus.cc:479] T cdf4190556a244ef8e614a243281e9a3 P 0668a9f7c6514ec296857579ea1e0c21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0668a9f7c6514ec296857579ea1e0c21" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 53006 } } peers { permanent_uuid: "4289d8404d9e416db70b1b9069cf7231" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 33446 } } peers { permanent_uuid: "820457af06944ac7966bf2fab634de94" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 44086 } } I0703 01:41:30.245921 22122 leader_election.cc:243] T cdf4190556a244ef8e614a243281e9a3 P 0668a9f7c6514ec296857579ea1e0c21 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 4289d8404d9e416db70b1b9069cf7231 (127.5.240.194:33446) I0703 01:41:30.246239 22122 leader_election.cc:243] T cdf4190556a244ef8e614a243281e9a3 P 0668a9f7c6514ec296857579ea1e0c21 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 820457af06944ac7966bf2fab634de94 (127.5.240.197:44086) I0703 01:41:30.254364 21687 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "cdf4190556a244ef8e614a243281e9a3" candidate_uuid: "0668a9f7c6514ec296857579ea1e0c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4289d8404d9e416db70b1b9069cf7231" is_pre_election: true I0703 01:41:30.255015 21687 raft_consensus.cc:2730] T cdf4190556a244ef8e614a243281e9a3 P 4289d8404d9e416db70b1b9069cf7231: Snoozing failure detection for 1.777s (vote granted) I0703 01:41:30.254609 22053 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "cdf4190556a244ef8e614a243281e9a3" candidate_uuid: "0668a9f7c6514ec296857579ea1e0c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "820457af06944ac7966bf2fab634de94" is_pre_election: true I0703 01:41:30.255098 21687 raft_consensus.cc:2243] T cdf4190556a244ef8e614a243281e9a3 P 4289d8404d9e416db70b1b9069cf7231 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0668a9f7c6514ec296857579ea1e0c21 in term 0. I0703 01:41:30.255250 22053 raft_consensus.cc:2730] T cdf4190556a244ef8e614a243281e9a3 P 820457af06944ac7966bf2fab634de94: Snoozing failure detection for 1.594s (vote granted) I0703 01:41:30.255339 22053 raft_consensus.cc:2243] T cdf4190556a244ef8e614a243281e9a3 P 820457af06944ac7966bf2fab634de94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0668a9f7c6514ec296857579ea1e0c21 in term 0. I0703 01:41:30.255676 21862 leader_election.cc:379] T cdf4190556a244ef8e614a243281e9a3 P 0668a9f7c6514ec296857579ea1e0c21 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 4289d8404d9e416db70b1b9069cf7231 (127.5.240.194:33446) I0703 01:41:30.255739 21862 leader_election.cc:270] T cdf4190556a244ef8e614a243281e9a3 P 0668a9f7c6514ec296857579ea1e0c21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0703 01:41:30.255990 22122 raft_consensus.cc:2730] T cdf4190556a244ef8e614a243281e9a3 P 0668a9f7c6514ec296857579ea1e0c21: Snoozing failure detection for 1.990s (election complete) I0703 01:41:30.256132 21862 leader_election.cc:379] T cdf4190556a244ef8e614a243281e9a3 P 0668a9f7c6514ec296857579ea1e0c21 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 820457af06944ac7966bf2fab634de94 (127.5.240.197:44086) I0703 01:41:30.256129 22122 raft_consensus.cc:2537] T cdf4190556a244ef8e614a243281e9a3 P 0668a9f7c6514ec296857579ea1e0c21 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0703 01:41:30.256217 22122 raft_consensus.cc:457] T cdf4190556a244ef8e614a243281e9a3 P 0668a9f7c6514ec296857579ea1e0c21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0703 01:41:30.256295 22122 raft_consensus.cc:2730] T cdf4190556a244ef8e614a243281e9a3 P 0668a9f7c6514ec296857579ea1e0c21: Snoozing failure detection for 1.625s (starting election) I0703 01:41:30.256376 22122 raft_consensus.cc:2782] T cdf4190556a244ef8e614a243281e9a3 P 0668a9f7c6514ec296857579ea1e0c21 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:41:30.257805 22122 raft_consensus.cc:479] T cdf4190556a244ef8e614a243281e9a3 P 0668a9f7c6514ec296857579ea1e0c21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0668a9f7c6514ec296857579ea1e0c21" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 53006 } } peers { permanent_uuid: "4289d8404d9e416db70b1b9069cf7231" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 33446 } } peers { permanent_uuid: "820457af06944ac7966bf2fab634de94" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 44086 } } I0703 01:41:30.258133 22122 leader_election.cc:243] T cdf4190556a244ef8e614a243281e9a3 P 0668a9f7c6514ec296857579ea1e0c21 [CANDIDATE]: Term 1 election: Requesting vote from peer 4289d8404d9e416db70b1b9069cf7231 (127.5.240.194:33446) I0703 01:41:30.258285 22122 leader_election.cc:243] T cdf4190556a244ef8e614a243281e9a3 P 0668a9f7c6514ec296857579ea1e0c21 [CANDIDATE]: Term 1 election: Requesting vote from peer 820457af06944ac7966bf2fab634de94 (127.5.240.197:44086) I0703 01:41:30.258818 21687 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "cdf4190556a244ef8e614a243281e9a3" candidate_uuid: "0668a9f7c6514ec296857579ea1e0c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4289d8404d9e416db70b1b9069cf7231" I0703 01:41:30.258988 21687 raft_consensus.cc:2782] T cdf4190556a244ef8e614a243281e9a3 P 4289d8404d9e416db70b1b9069cf7231 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:41:30.259091 21687 raft_consensus.cc:2730] T cdf4190556a244ef8e614a243281e9a3 P 4289d8404d9e416db70b1b9069cf7231: Snoozing failure detection for 2.064s (vote granted) I0703 01:41:30.259415 22053 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "cdf4190556a244ef8e614a243281e9a3" candidate_uuid: "0668a9f7c6514ec296857579ea1e0c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "820457af06944ac7966bf2fab634de94" I0703 01:41:30.259544 22053 raft_consensus.cc:2782] T cdf4190556a244ef8e614a243281e9a3 P 820457af06944ac7966bf2fab634de94 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:41:30.259672 22053 raft_consensus.cc:2730] T cdf4190556a244ef8e614a243281e9a3 P 820457af06944ac7966bf2fab634de94: Snoozing failure detection for 1.630s (vote granted) I0703 01:41:30.260391 21687 raft_consensus.cc:2243] T cdf4190556a244ef8e614a243281e9a3 P 4289d8404d9e416db70b1b9069cf7231 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0668a9f7c6514ec296857579ea1e0c21 in term 1. I0703 01:41:30.260808 21862 leader_election.cc:379] T cdf4190556a244ef8e614a243281e9a3 P 0668a9f7c6514ec296857579ea1e0c21 [CANDIDATE]: Term 1 election: Vote granted by peer 4289d8404d9e416db70b1b9069cf7231 (127.5.240.194:33446) I0703 01:41:30.260856 21862 leader_election.cc:270] T cdf4190556a244ef8e614a243281e9a3 P 0668a9f7c6514ec296857579ea1e0c21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:41:30.261000 22122 raft_consensus.cc:2730] T cdf4190556a244ef8e614a243281e9a3 P 0668a9f7c6514ec296857579ea1e0c21: Snoozing failure detection for 2.010s (election complete) I0703 01:41:30.261077 22122 raft_consensus.cc:2537] T cdf4190556a244ef8e614a243281e9a3 P 0668a9f7c6514ec296857579ea1e0c21 [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:41:30.261420 22053 raft_consensus.cc:2243] T cdf4190556a244ef8e614a243281e9a3 P 820457af06944ac7966bf2fab634de94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0668a9f7c6514ec296857579ea1e0c21 in term 1. I0703 01:41:30.262182 21862 leader_election.cc:379] T cdf4190556a244ef8e614a243281e9a3 P 0668a9f7c6514ec296857579ea1e0c21 [CANDIDATE]: Term 1 election: Vote granted by peer 820457af06944ac7966bf2fab634de94 (127.5.240.197:44086) I0703 01:41:30.262068 22122 raft_consensus.cc:593] T cdf4190556a244ef8e614a243281e9a3 P 0668a9f7c6514ec296857579ea1e0c21 [term 1 LEADER]: Becoming Leader. State: Replica: 0668a9f7c6514ec296857579ea1e0c21, State: Running, Role: LEADER I0703 01:41:30.262626 22122 consensus_queue.cc:204] T cdf4190556a244ef8e614a243281e9a3 P 0668a9f7c6514ec296857579ea1e0c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0668a9f7c6514ec296857579ea1e0c21" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 53006 } } peers { permanent_uuid: "4289d8404d9e416db70b1b9069cf7231" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 33446 } } peers { permanent_uuid: "820457af06944ac7966bf2fab634de94" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 44086 } } I0703 01:41:30.266697 21444 catalog_manager.cc:3798] T cdf4190556a244ef8e614a243281e9a3 P 0668a9f7c6514ec296857579ea1e0c21 reported cstate change: term changed from 0 to 1, leader changed from to 0668a9f7c6514ec296857579ea1e0c21 (127.5.240.196). New cstate: current_term: 1 leader_uuid: "0668a9f7c6514ec296857579ea1e0c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0668a9f7c6514ec296857579ea1e0c21" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 53006 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4289d8404d9e416db70b1b9069cf7231" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 33446 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "820457af06944ac7966bf2fab634de94" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 44086 } health_report { overall_health: UNKNOWN } } } I0703 01:41:30.299291 6083 external_mini_cluster.cc:490] 5 TS(s) registered with all masters I0703 01:41:30.300427 6083 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 51800c5e0fd34699836c1611adecaf5b to finish bootstrapping I0703 01:41:30.307829 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 4289d8404d9e416db70b1b9069cf7231 to finish bootstrapping W0703 01:41:30.308743 21731 tablet.cc:1678] T cdf4190556a244ef8e614a243281e9a3 P 4289d8404d9e416db70b1b9069cf7231: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:41:30.312675 6083 ts_itest-base.cc:233] Waiting for 0 tablets on tserver eac7487faa4344c1b3edaac9ab871ac7 to finish bootstrapping W0703 01:41:30.313685 22097 tablet.cc:1678] T cdf4190556a244ef8e614a243281e9a3 P 820457af06944ac7966bf2fab634de94: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:41:30.318769 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 0668a9f7c6514ec296857579ea1e0c21 to finish bootstrapping I0703 01:41:30.323307 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 820457af06944ac7966bf2fab634de94 to finish bootstrapping I0703 01:41:30.328105 6083 test_util.cc:208] Using random seed: 1205117976 I0703 01:41:30.336313 6083 test_workload.cc:306] TestWorkload: Skipping table creation because table TestTable already exists I0703 01:41:30.336745 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 21977 I0703 01:41:30.337473 22093 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:41:30.350944 21687 raft_consensus.cc:1081] T cdf4190556a244ef8e614a243281e9a3 P 4289d8404d9e416db70b1b9069cf7231 [term 1 FOLLOWER]: Refusing update from remote peer 0668a9f7c6514ec296857579ea1e0c21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0703 01:41:30.352169 22126 consensus_queue.cc:962] T cdf4190556a244ef8e614a243281e9a3 P 0668a9f7c6514ec296857579ea1e0c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4289d8404d9e416db70b1b9069cf7231" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 33446 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0703 01:41:30.388525 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 21977 W0703 01:41:30.396889 21428 connection.cc:512] server connection from 127.5.240.197:58236 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.197:58236): Connection reset by peer (error 104) W0703 01:41:30.397295 21862 connection.cc:512] client connection to 127.5.240.197:44086 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.197:44086): Connection reset by peer (error 104) W0703 01:41:30.397441 21862 consensus_peers.cc:433] T cdf4190556a244ef8e614a243281e9a3 P 0668a9f7c6514ec296857579ea1e0c21 -> Peer 820457af06944ac7966bf2fab634de94 (127.5.240.197:44086): Couldn't send request to peer 820457af06944ac7966bf2fab634de94 for tablet cdf4190556a244ef8e614a243281e9a3. Status: Network error: failed to read from TLS socket (remote: 127.5.240.197:44086): Connection reset by peer (error 104). Retrying in the next heartbeat period. Already tried 1 times. I0703 01:41:30.420437 21852 heartbeater.cc:467] Master 127.5.240.254:36705 was elected leader, sending a full tablet report... W0703 01:41:30.648028 22144 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:41:30.804368 21862 consensus_peers.cc:433] T cdf4190556a244ef8e614a243281e9a3 P 0668a9f7c6514ec296857579ea1e0c21 -> Peer 820457af06944ac7966bf2fab634de94 (127.5.240.197:44086): Couldn't send request to peer 820457af06944ac7966bf2fab634de94 for tablet cdf4190556a244ef8e614a243281e9a3. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.197:44086: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. W0703 01:41:31.226044 21862 consensus_peers.cc:433] T cdf4190556a244ef8e614a243281e9a3 P 0668a9f7c6514ec296857579ea1e0c21 -> Peer 820457af06944ac7966bf2fab634de94 (127.5.240.197:44086): Couldn't send request to peer 820457af06944ac7966bf2fab634de94 for tablet cdf4190556a244ef8e614a243281e9a3. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.197:44086: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times. W0703 01:41:31.777709 21862 consensus_peers.cc:433] T cdf4190556a244ef8e614a243281e9a3 P 0668a9f7c6514ec296857579ea1e0c21 -> Peer 820457af06944ac7966bf2fab634de94 (127.5.240.197:44086): Couldn't send request to peer 820457af06944ac7966bf2fab634de94 for tablet cdf4190556a244ef8e614a243281e9a3. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.197:44086: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 4 times. W0703 01:41:32.311400 21862 consensus_peers.cc:433] T cdf4190556a244ef8e614a243281e9a3 P 0668a9f7c6514ec296857579ea1e0c21 -> Peer 820457af06944ac7966bf2fab634de94 (127.5.240.197:44086): Couldn't send request to peer 820457af06944ac7966bf2fab634de94 for tablet cdf4190556a244ef8e614a243281e9a3. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.197:44086: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 5 times. W0703 01:41:32.701216 21862 consensus_peers.cc:433] T cdf4190556a244ef8e614a243281e9a3 P 0668a9f7c6514ec296857579ea1e0c21 -> Peer 820457af06944ac7966bf2fab634de94 (127.5.240.197:44086): Couldn't send request to peer 820457af06944ac7966bf2fab634de94 for tablet cdf4190556a244ef8e614a243281e9a3. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.197:44086: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 6 times. W0703 01:41:33.257133 21862 consensus_peers.cc:433] T cdf4190556a244ef8e614a243281e9a3 P 0668a9f7c6514ec296857579ea1e0c21 -> Peer 820457af06944ac7966bf2fab634de94 (127.5.240.197:44086): Couldn't send request to peer 820457af06944ac7966bf2fab634de94 for tablet cdf4190556a244ef8e614a243281e9a3. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.197:44086: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 7 times. W0703 01:41:33.802043 21862 consensus_peers.cc:433] T cdf4190556a244ef8e614a243281e9a3 P 0668a9f7c6514ec296857579ea1e0c21 -> Peer 820457af06944ac7966bf2fab634de94 (127.5.240.197:44086): Couldn't send request to peer 820457af06944ac7966bf2fab634de94 for tablet cdf4190556a244ef8e614a243281e9a3. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.197:44086: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 8 times. W0703 01:41:34.191515 21862 consensus_peers.cc:433] T cdf4190556a244ef8e614a243281e9a3 P 0668a9f7c6514ec296857579ea1e0c21 -> Peer 820457af06944ac7966bf2fab634de94 (127.5.240.197:44086): Couldn't send request to peer 820457af06944ac7966bf2fab634de94 for tablet cdf4190556a244ef8e614a243281e9a3. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.197:44086: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 9 times. W0703 01:41:34.703974 21862 consensus_peers.cc:433] T cdf4190556a244ef8e614a243281e9a3 P 0668a9f7c6514ec296857579ea1e0c21 -> Peer 820457af06944ac7966bf2fab634de94 (127.5.240.197:44086): Couldn't send request to peer 820457af06944ac7966bf2fab634de94 for tablet cdf4190556a244ef8e614a243281e9a3. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.197:44086: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 10 times. W0703 01:41:35.111415 21862 consensus_peers.cc:433] T cdf4190556a244ef8e614a243281e9a3 P 0668a9f7c6514ec296857579ea1e0c21 -> Peer 820457af06944ac7966bf2fab634de94 (127.5.240.197:44086): Couldn't send request to peer 820457af06944ac7966bf2fab634de94 for tablet cdf4190556a244ef8e614a243281e9a3. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.197:44086: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 11 times. W0703 01:41:35.650990 21862 consensus_peers.cc:433] T cdf4190556a244ef8e614a243281e9a3 P 0668a9f7c6514ec296857579ea1e0c21 -> Peer 820457af06944ac7966bf2fab634de94 (127.5.240.197:44086): Couldn't send request to peer 820457af06944ac7966bf2fab634de94 for tablet cdf4190556a244ef8e614a243281e9a3. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.197:44086: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 12 times. W0703 01:41:36.143306 21862 consensus_peers.cc:433] T cdf4190556a244ef8e614a243281e9a3 P 0668a9f7c6514ec296857579ea1e0c21 -> Peer 820457af06944ac7966bf2fab634de94 (127.5.240.197:44086): Couldn't send request to peer 820457af06944ac7966bf2fab634de94 for tablet cdf4190556a244ef8e614a243281e9a3. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.197:44086: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 13 times. Corruption: ksck pre-move health check failed: timed out with ksck errors remaining: last error: 1 out of 1 table(s) are not healthy I0703 01:41:36.217283 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 21478 I0703 01:41:36.220180 21604 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:41:36.260283 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 21478 W0703 01:41:36.268671 21426 connection.cc:512] server connection from 127.5.240.193:49006 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:49006): Connection reset by peer (error 104) I0703 01:41:36.268856 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 21609 I0703 01:41:36.269594 21727 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:41:36.388149 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 21609 W0703 01:41:36.403681 21862 connection.cc:512] client connection to 127.5.240.194:33446 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:33446): Connection reset by peer (error 104) W0703 01:41:36.403797 21425 connection.cc:512] server connection from 127.5.240.194:48000 recv error: Network error: failed to read from TLS socket (remote: 0.0.0.0:0): Connection reset by peer (error 104) I0703 01:41:36.403877 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 21733 W0703 01:41:36.403970 21425 connection.cc:185] Error closing socket: Network error: TlsSocket::Close: Broken pipe I0703 01:41:36.404613 21849 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:41:36.444485 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 21733 W0703 01:41:36.452730 21425 connection.cc:512] server connection from 127.5.240.195:58285 recv error: Network error: failed to read from TLS socket (remote: 0.0.0.0:0): Connection reset by peer (error 104) I0703 01:41:36.452837 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 21854 W0703 01:41:36.452889 21425 connection.cc:185] Error closing socket: Network error: TlsSocket::Close: Broken pipe I0703 01:41:36.453516 21971 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:41:36.572855 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 21854 W0703 01:41:36.589087 21425 connection.cc:512] server connection from 127.5.240.196:51319 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.196:51319): Connection reset by peer (error 104) I0703 01:41:36.589262 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master pid 21418 I0703 01:41:36.589962 21474 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:41:36.655467 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master with pid 21418 [ OK ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4 (8675 ms) [ RUN ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/5 I0703 01:41:36.678607 6083 test_util.cc:208] Using random seed: 1211468476 I0703 01:41:36.680500 6083 ts_itest-base.cc:113] Starting cluster with: I0703 01:41:36.680577 6083 ts_itest-base.cc:114] -------------- I0703 01:41:36.680598 6083 ts_itest-base.cc:115] 5 tablet servers I0703 01:41:36.680615 6083 ts_itest-base.cc:116] 3 replicas per TS I0703 01:41:36.680639 6083 ts_itest-base.cc:117] -------------- I0703 01:41:36.681732 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:51317 --webserver_interface=127.5.240.254 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:41:36.866497 22312 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:41:36.866722 22312 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:41:36.868901 22312 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0703 01:41:36.868952 22312 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0703 01:41:36.868985 22312 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:41:36.869009 22312 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0703 01:41:36.869036 22312 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0703 01:41:36.877080 22312 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:51317 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.5.240.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:41:36.883076 22312 master_main.cc:80] Initializing master server... I0703 01:41:36.883751 22312 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 201708us I0703 01:41:36.884791 22312 fs_manager.cc:261] Metadata directory not provided I0703 01:41:36.884851 22312 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:41:36.885071 22312 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:41:36.885134 22312 server_base.cc:433] Attempting to create new FS layout instead I0703 01:41:36.889276 22312 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/instance: uuid: "fd87525fbee744be802e40c074502554" format_stamp: "Formatted at 2018-07-03 08:41:36 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:36.889807 22312 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "fd87525fbee744be802e40c074502554" format_stamp: "Formatted at 2018-07-03 08:41:36 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:36.897816 22312 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.004s sys 0.003s I0703 01:41:36.899021 22312 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:41:36.899387 22312 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:41:36.902540 22312 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0703 01:41:36.902590 22312 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal uuid: "fd87525fbee744be802e40c074502554" format_stamp: "Formatted at 2018-07-03 08:41:36 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:36.902709 22312 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:41:36.964727 22312 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:41:36.965695 22312 master_main.cc:83] Starting Master server... I0703 01:41:37.002816 22312 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.254:51317 I0703 01:41:37.003296 22312 webserver.cc:175] Starting webserver on 127.5.240.254:0 I0703 01:41:37.003331 22312 webserver.cc:186] Document root disabled I0703 01:41:37.004514 22312 webserver.cc:313] Webserver started. Bound to: http://127.5.240.254:57589/ I0703 01:41:37.005091 22312 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb I0703 01:41:37.006834 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master as pid 22312 I0703 01:41:37.007458 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.5.240.193:0 --local_ip_for_outbound_sockets=127.5.240.193 --webserver_interface=127.5.240.193 --webserver_port=0 --tserver_master_addrs=127.5.240.254:51317 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0703 01:41:37.008293 22371 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:41:37.020033 22371 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P fd87525fbee744be802e40c074502554: Bootstrap starting. I0703 01:41:37.022075 22371 tablet_bootstrap.cc:578] T 00000000000000000000000000000000 P fd87525fbee744be802e40c074502554: No blocks or log segments found. Creating new log. I0703 01:41:37.022830 22371 log.cc:525] T 00000000000000000000000000000000 P fd87525fbee744be802e40c074502554: Log is configured to *not* fsync() on all Append() calls I0703 01:41:37.025107 22371 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P fd87525fbee744be802e40c074502554: No bootstrap required, opened a new log I0703 01:41:37.031185 22371 raft_consensus.cc:325] T 00000000000000000000000000000000 P fd87525fbee744be802e40c074502554 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd87525fbee744be802e40c074502554" member_type: VOTER } I0703 01:41:37.031378 22371 raft_consensus.cc:351] T 00000000000000000000000000000000 P fd87525fbee744be802e40c074502554 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:41:37.031445 22371 raft_consensus.cc:628] T 00000000000000000000000000000000 P fd87525fbee744be802e40c074502554 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd87525fbee744be802e40c074502554, State: Initialized, Role: FOLLOWER I0703 01:41:37.031911 22371 consensus_queue.cc:227] T 00000000000000000000000000000000 P fd87525fbee744be802e40c074502554 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd87525fbee744be802e40c074502554" member_type: VOTER } I0703 01:41:37.032059 22371 raft_consensus.cc:365] T 00000000000000000000000000000000 P fd87525fbee744be802e40c074502554 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:41:37.032143 22371 raft_consensus.cc:457] T 00000000000000000000000000000000 P fd87525fbee744be802e40c074502554 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:41:37.032229 22371 raft_consensus.cc:2730] T 00000000000000000000000000000000 P fd87525fbee744be802e40c074502554: Snoozing failure detection for 1.662s (starting election) I0703 01:41:37.032317 22371 raft_consensus.cc:2782] T 00000000000000000000000000000000 P fd87525fbee744be802e40c074502554 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:41:37.033651 22371 raft_consensus.cc:479] T 00000000000000000000000000000000 P fd87525fbee744be802e40c074502554 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd87525fbee744be802e40c074502554" member_type: VOTER } I0703 01:41:37.034016 22371 leader_election.cc:270] T 00000000000000000000000000000000 P fd87525fbee744be802e40c074502554 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:41:37.035425 22374 raft_consensus.cc:2730] T 00000000000000000000000000000000 P fd87525fbee744be802e40c074502554: Snoozing failure detection for 1.804s (election complete) I0703 01:41:37.035881 22374 raft_consensus.cc:2537] T 00000000000000000000000000000000 P fd87525fbee744be802e40c074502554 [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:41:37.037318 22374 raft_consensus.cc:593] T 00000000000000000000000000000000 P fd87525fbee744be802e40c074502554 [term 1 LEADER]: Becoming Leader. State: Replica: fd87525fbee744be802e40c074502554, State: Running, Role: LEADER I0703 01:41:37.037675 22374 consensus_queue.cc:204] T 00000000000000000000000000000000 P fd87525fbee744be802e40c074502554 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd87525fbee744be802e40c074502554" member_type: VOTER } I0703 01:41:37.039881 22371 sys_catalog.cc:337] T 00000000000000000000000000000000 P fd87525fbee744be802e40c074502554 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "fd87525fbee744be802e40c074502554" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd87525fbee744be802e40c074502554" member_type: VOTER } } I0703 01:41:37.040108 22371 sys_catalog.cc:340] T 00000000000000000000000000000000 P fd87525fbee744be802e40c074502554 [sys.catalog]: This master's current role is: LEADER I0703 01:41:37.040105 22375 sys_catalog.cc:337] T 00000000000000000000000000000000 P fd87525fbee744be802e40c074502554 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd87525fbee744be802e40c074502554" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd87525fbee744be802e40c074502554" member_type: VOTER } } I0703 01:41:37.042585 22375 sys_catalog.cc:340] T 00000000000000000000000000000000 P fd87525fbee744be802e40c074502554 [sys.catalog]: This master's current role is: LEADER I0703 01:41:37.042479 22376 sys_catalog.cc:337] T 00000000000000000000000000000000 P fd87525fbee744be802e40c074502554 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd87525fbee744be802e40c074502554. Latest consensus state: current_term: 1 leader_uuid: "fd87525fbee744be802e40c074502554" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd87525fbee744be802e40c074502554" member_type: VOTER } } I0703 01:41:37.043222 22376 sys_catalog.cc:340] T 00000000000000000000000000000000 P fd87525fbee744be802e40c074502554 [sys.catalog]: This master's current role is: LEADER I0703 01:41:37.043886 22371 sys_catalog.cc:424] T 00000000000000000000000000000000 P fd87525fbee744be802e40c074502554 [sys.catalog]: configured and running, proceeding with master startup. I0703 01:41:37.045473 22312 master_main.cc:86] Master server successfully started. W0703 01:41:37.050618 22379 catalog_manager.cc:1064] T 00000000000000000000000000000000 P fd87525fbee744be802e40c074502554: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0703 01:41:37.050726 22379 catalog_manager.cc:563] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0703 01:41:37.050829 22378 catalog_manager.cc:1006] Loading table and tablet metadata into memory... I0703 01:41:37.052860 22378 catalog_manager.cc:1017] Initializing Kudu internal certificate authority... I0703 01:41:37.165455 22378 catalog_manager.cc:904] Generated new certificate authority record I0703 01:41:37.167480 22378 catalog_manager.cc:1026] Loading token signing keys... WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:41:37.174711 22372 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:41:37.174932 22372 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:41:37.176992 22372 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0703 01:41:37.177083 22372 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:41:37.177250 22372 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.193 I0703 01:41:37.178747 22378 catalog_manager.cc:4102] T 00000000000000000000000000000000 P fd87525fbee744be802e40c074502554: Generated new TSK 0 I0703 01:41:37.184489 22372 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.5.240.193 --webserver_port=0 --tserver_master_addrs=127.5.240.254:51317 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.193 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:41:37.191062 22372 tablet_server_main.cc:85] Initializing tablet server... I0703 01:41:37.192580 22372 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 202208us I0703 01:41:37.193173 22372 fs_manager.cc:261] Metadata directory not provided I0703 01:41:37.193238 22372 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:41:37.193464 22372 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:41:37.193532 22372 server_base.cc:433] Attempting to create new FS layout instead I0703 01:41:37.197643 22372 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "6af80a37db6048b4951d8408b5cf7552" format_stamp: "Formatted at 2018-07-03 08:41:37 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:37.198130 22372 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "6af80a37db6048b4951d8408b5cf7552" format_stamp: "Formatted at 2018-07-03 08:41:37 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:37.205750 22372 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.004s sys 0.003s I0703 01:41:37.206943 22372 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:41:37.207314 22372 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:41:37.210873 22372 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0703 01:41:37.210925 22372 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal uuid: "6af80a37db6048b4951d8408b5cf7552" format_stamp: "Formatted at 2018-07-03 08:41:37 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:37.211078 22372 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:41:37.248276 22372 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:41:37.250246 22372 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:41:37.250604 22372 tablet_server_main.cc:90] Starting tablet server... I0703 01:41:37.346566 22372 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.193:58187 I0703 01:41:37.347054 22372 webserver.cc:175] Starting webserver on 127.5.240.193:0 I0703 01:41:37.347100 22372 webserver.cc:186] Document root disabled I0703 01:41:37.348438 22372 webserver.cc:313] Webserver started. Bound to: http://127.5.240.193:53525/ I0703 01:41:37.348989 22372 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb I0703 01:41:37.351029 22372 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:41:37.353495 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 22372 I0703 01:41:37.354377 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.5.240.194:0 --local_ip_for_outbound_sockets=127.5.240.194 --webserver_interface=127.5.240.194 --webserver_port=0 --tserver_master_addrs=127.5.240.254:51317 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0703 01:41:37.375813 22502 heartbeater.cc:346] Connected to a master server at 127.5.240.254:51317 I0703 01:41:37.375993 22502 heartbeater.cc:426] Registering TS with master... I0703 01:41:37.376930 22502 heartbeater.cc:475] Master 127.5.240.254:51317 requested a full tablet report, sending... I0703 01:41:37.379175 22338 ts_manager.cc:98] Registered new tserver with Master: 6af80a37db6048b4951d8408b5cf7552 (127.5.240.193:58187) I0703 01:41:37.381961 22338 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.193:38479 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:41:37.522869 22506 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:41:37.523106 22506 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:41:37.525104 22506 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0703 01:41:37.525172 22506 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:41:37.525297 22506 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.194 I0703 01:41:37.532232 22506 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.194:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.5.240.194 --webserver_port=0 --tserver_master_addrs=127.5.240.254:51317 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.194 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:41:37.538882 22506 tablet_server_main.cc:85] Initializing tablet server... I0703 01:41:37.540431 22506 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 202208us I0703 01:41:37.541046 22506 fs_manager.cc:261] Metadata directory not provided I0703 01:41:37.541115 22506 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:41:37.541344 22506 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:41:37.541415 22506 server_base.cc:433] Attempting to create new FS layout instead I0703 01:41:37.545537 22506 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "a3572ff046ff4c58b4987006387804e1" format_stamp: "Formatted at 2018-07-03 08:41:37 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:37.546111 22506 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "a3572ff046ff4c58b4987006387804e1" format_stamp: "Formatted at 2018-07-03 08:41:37 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:37.553448 22506 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.001s sys 0.005s I0703 01:41:37.554661 22506 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:41:37.555032 22506 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:41:37.558611 22506 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s I0703 01:41:37.558662 22506 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal uuid: "a3572ff046ff4c58b4987006387804e1" format_stamp: "Formatted at 2018-07-03 08:41:37 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:37.558789 22506 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:41:37.615190 22506 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:41:37.617460 22506 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:41:37.617812 22506 tablet_server_main.cc:90] Starting tablet server... I0703 01:41:37.702183 22506 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.194:39288 I0703 01:41:37.702608 22506 webserver.cc:175] Starting webserver on 127.5.240.194:0 I0703 01:41:37.702668 22506 webserver.cc:186] Document root disabled I0703 01:41:37.703294 22506 webserver.cc:313] Webserver started. Bound to: http://127.5.240.194:33892/ I0703 01:41:37.703851 22506 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb I0703 01:41:37.706120 22506 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:41:37.710274 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 22506 I0703 01:41:37.711097 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.5.240.195:0 --local_ip_for_outbound_sockets=127.5.240.195 --webserver_interface=127.5.240.195 --webserver_port=0 --tserver_master_addrs=127.5.240.254:51317 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0703 01:41:37.718010 22625 heartbeater.cc:346] Connected to a master server at 127.5.240.254:51317 I0703 01:41:37.718185 22625 heartbeater.cc:426] Registering TS with master... I0703 01:41:37.719077 22625 heartbeater.cc:475] Master 127.5.240.254:51317 requested a full tablet report, sending... I0703 01:41:37.720058 22338 ts_manager.cc:98] Registered new tserver with Master: a3572ff046ff4c58b4987006387804e1 (127.5.240.194:39288) I0703 01:41:37.720908 22338 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.194:46617 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:41:37.871232 22628 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:41:37.871464 22628 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:41:37.873440 22628 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0703 01:41:37.873502 22628 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:41:37.873644 22628 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.195 I0703 01:41:37.880266 22628 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.195:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.5.240.195 --webserver_port=0 --tserver_master_addrs=127.5.240.254:51317 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.195 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:41:37.886721 22628 tablet_server_main.cc:85] Initializing tablet server... I0703 01:41:37.888211 22628 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 202208us I0703 01:41:37.888787 22628 fs_manager.cc:261] Metadata directory not provided I0703 01:41:37.888854 22628 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:41:37.889456 22628 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:41:37.889518 22628 server_base.cc:433] Attempting to create new FS layout instead I0703 01:41:37.893504 22628 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "b7b4950f9efa4a829572d08be938050f" format_stamp: "Formatted at 2018-07-03 08:41:37 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:37.894069 22628 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "b7b4950f9efa4a829572d08be938050f" format_stamp: "Formatted at 2018-07-03 08:41:37 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:37.901729 22628 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.003s sys 0.004s I0703 01:41:37.902905 22628 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:41:37.903264 22628 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:41:37.906319 22628 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0703 01:41:37.906368 22628 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal uuid: "b7b4950f9efa4a829572d08be938050f" format_stamp: "Formatted at 2018-07-03 08:41:37 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:37.906488 22628 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:41:37.973201 22628 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:41:37.975458 22628 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:41:37.975800 22628 tablet_server_main.cc:90] Starting tablet server... I0703 01:41:38.060837 22628 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.195:53950 I0703 01:41:38.061254 22628 webserver.cc:175] Starting webserver on 127.5.240.195:0 I0703 01:41:38.061300 22628 webserver.cc:186] Document root disabled I0703 01:41:38.062008 22628 webserver.cc:313] Webserver started. Bound to: http://127.5.240.195:52696/ I0703 01:41:38.062546 22628 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb I0703 01:41:38.065073 22628 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:41:38.068627 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 22628 I0703 01:41:38.069478 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.5.240.196:0 --local_ip_for_outbound_sockets=127.5.240.196 --webserver_interface=127.5.240.196 --webserver_port=0 --tserver_master_addrs=127.5.240.254:51317 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0703 01:41:38.077597 22747 heartbeater.cc:346] Connected to a master server at 127.5.240.254:51317 I0703 01:41:38.077754 22747 heartbeater.cc:426] Registering TS with master... I0703 01:41:38.078552 22747 heartbeater.cc:475] Master 127.5.240.254:51317 requested a full tablet report, sending... I0703 01:41:38.079627 22338 ts_manager.cc:98] Registered new tserver with Master: b7b4950f9efa4a829572d08be938050f (127.5.240.195:53950) I0703 01:41:38.080451 22338 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.195:50812 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:41:38.235255 22750 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:41:38.235468 22750 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:41:38.237529 22750 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0703 01:41:38.237579 22750 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:41:38.237679 22750 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.196 I0703 01:41:38.244586 22750 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.196:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb --webserver_interface=127.5.240.196 --webserver_port=0 --tserver_master_addrs=127.5.240.254:51317 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.196 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:41:38.251278 22750 tablet_server_main.cc:85] Initializing tablet server... I0703 01:41:38.252823 22750 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 202708us I0703 01:41:38.253371 22750 fs_manager.cc:261] Metadata directory not provided I0703 01:41:38.253433 22750 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:41:38.253670 22750 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:41:38.253736 22750 server_base.cc:433] Attempting to create new FS layout instead I0703 01:41:38.257751 22750 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/instance: uuid: "5f5927fad870472d8e084aaf82f2323b" format_stamp: "Formatted at 2018-07-03 08:41:38 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:38.258271 22750 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal/instance: uuid: "5f5927fad870472d8e084aaf82f2323b" format_stamp: "Formatted at 2018-07-03 08:41:38 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:38.265640 22750 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.004s sys 0.002s I0703 01:41:38.266904 22750 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:41:38.267300 22750 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:41:38.270925 22750 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s I0703 01:41:38.270977 22750 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal uuid: "5f5927fad870472d8e084aaf82f2323b" format_stamp: "Formatted at 2018-07-03 08:41:38 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:38.271111 22750 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:41:38.297679 22750 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:41:38.299964 22750 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:41:38.300371 22750 tablet_server_main.cc:90] Starting tablet server... I0703 01:41:38.384807 22502 heartbeater.cc:467] Master 127.5.240.254:51317 was elected leader, sending a full tablet report... I0703 01:41:38.386839 22750 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.196:52229 I0703 01:41:38.387249 22750 webserver.cc:175] Starting webserver on 127.5.240.196:0 I0703 01:41:38.387295 22750 webserver.cc:186] Document root disabled I0703 01:41:38.388463 22750 webserver.cc:313] Webserver started. Bound to: http://127.5.240.196:57601/ I0703 01:41:38.389048 22750 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb I0703 01:41:38.390799 22750 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:41:38.399175 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 22750 I0703 01:41:38.399835 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data --block_manager=log --rpc_bind_addresses=127.5.240.197:0 --local_ip_for_outbound_sockets=127.5.240.197 --webserver_interface=127.5.240.197 --webserver_port=0 --tserver_master_addrs=127.5.240.254:51317 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0703 01:41:38.404047 22869 heartbeater.cc:346] Connected to a master server at 127.5.240.254:51317 I0703 01:41:38.404183 22869 heartbeater.cc:426] Registering TS with master... I0703 01:41:38.404973 22869 heartbeater.cc:475] Master 127.5.240.254:51317 requested a full tablet report, sending... I0703 01:41:38.405961 22338 ts_manager.cc:98] Registered new tserver with Master: 5f5927fad870472d8e084aaf82f2323b (127.5.240.196:52229) I0703 01:41:38.407058 22338 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.196:41695 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:41:38.561955 22872 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:41:38.562180 22872 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:41:38.564242 22872 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0703 01:41:38.564306 22872 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:41:38.564409 22872 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.197 I0703 01:41:38.571527 22872 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.197:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb --webserver_interface=127.5.240.197 --webserver_port=0 --tserver_master_addrs=127.5.240.254:51317 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.197 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:41:38.578197 22872 tablet_server_main.cc:85] Initializing tablet server... I0703 01:41:38.579777 22872 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 202708us I0703 01:41:38.580368 22872 fs_manager.cc:261] Metadata directory not provided I0703 01:41:38.580438 22872 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:41:38.580695 22872 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:41:38.580765 22872 server_base.cc:433] Attempting to create new FS layout instead I0703 01:41:38.584913 22872 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/instance: uuid: "7dc1b3f7123e460891db0377178f5634" format_stamp: "Formatted at 2018-07-03 08:41:38 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:38.585455 22872 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal/instance: uuid: "7dc1b3f7123e460891db0377178f5634" format_stamp: "Formatted at 2018-07-03 08:41:38 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:38.592873 22872 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.004s sys 0.002s I0703 01:41:38.594079 22872 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:41:38.594455 22872 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:41:38.598078 22872 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0703 01:41:38.598129 22872 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal uuid: "7dc1b3f7123e460891db0377178f5634" format_stamp: "Formatted at 2018-07-03 08:41:38 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:38.598261 22872 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:41:38.622579 22872 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:41:38.624868 22872 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:41:38.625211 22872 tablet_server_main.cc:90] Starting tablet server... I0703 01:41:38.709254 22872 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.197:38959 I0703 01:41:38.709703 22872 webserver.cc:175] Starting webserver on 127.5.240.197:0 I0703 01:41:38.709745 22872 webserver.cc:186] Document root disabled I0703 01:41:38.710911 22872 webserver.cc:313] Webserver started. Bound to: http://127.5.240.197:41890/ I0703 01:41:38.711432 22872 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb I0703 01:41:38.713320 22872 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:41:38.718854 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 22872 I0703 01:41:38.722720 22625 heartbeater.cc:467] Master 127.5.240.254:51317 was elected leader, sending a full tablet report... I0703 01:41:38.728209 22991 heartbeater.cc:346] Connected to a master server at 127.5.240.254:51317 I0703 01:41:38.728354 22991 heartbeater.cc:426] Registering TS with master... I0703 01:41:38.729131 22991 heartbeater.cc:475] Master 127.5.240.254:51317 requested a full tablet report, sending... I0703 01:41:38.730310 22337 ts_manager.cc:98] Registered new tserver with Master: 7dc1b3f7123e460891db0377178f5634 (127.5.240.197:38959) I0703 01:41:38.731164 22337 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.197:52525 I0703 01:41:38.732471 6083 external_mini_cluster.cc:490] 5 TS(s) registered with all masters I0703 01:41:38.742149 22337 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.5.240.254:60238: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0703 01:41:38.766826 22684 tablet_service.cc:759] Processing CreateTablet for tablet 5c9976bd26bc43c1a026a2148f7079ed (table=TestTable [id=0541c3d083f84ed7997411acb96335cb]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:41:38.767534 22684 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 5c9976bd26bc43c1a026a2148f7079ed. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:41:38.767163 22806 tablet_service.cc:759] Processing CreateTablet for tablet 5c9976bd26bc43c1a026a2148f7079ed (table=TestTable [id=0541c3d083f84ed7997411acb96335cb]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:41:38.767987 22806 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 5c9976bd26bc43c1a026a2148f7079ed. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:41:38.775005 22439 tablet_service.cc:759] Processing CreateTablet for tablet 5c9976bd26bc43c1a026a2148f7079ed (table=TestTable [id=0541c3d083f84ed7997411acb96335cb]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:41:38.775877 22439 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 5c9976bd26bc43c1a026a2148f7079ed. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:41:38.778009 22806 ts_tablet_manager.cc:1173] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b: Registered tablet (data state: TABLET_DATA_READY) I0703 01:41:38.778816 23011 ts_tablet_manager.cc:1028] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b: Bootstrapping tablet I0703 01:41:38.779372 23011 tablet_bootstrap.cc:427] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b: Bootstrap starting. I0703 01:41:38.781872 23011 tablet_bootstrap.cc:578] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b: No blocks or log segments found. Creating new log. I0703 01:41:38.782627 23011 log.cc:525] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b: Log is configured to *not* fsync() on all Append() calls I0703 01:41:38.784189 23011 tablet_bootstrap.cc:427] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b: No bootstrap required, opened a new log I0703 01:41:38.784363 23011 ts_tablet_manager.cc:1045] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s I0703 01:41:38.784850 22684 ts_tablet_manager.cc:1173] T 5c9976bd26bc43c1a026a2148f7079ed P b7b4950f9efa4a829572d08be938050f: Registered tablet (data state: TABLET_DATA_READY) I0703 01:41:38.785648 23013 ts_tablet_manager.cc:1028] T 5c9976bd26bc43c1a026a2148f7079ed P b7b4950f9efa4a829572d08be938050f: Bootstrapping tablet I0703 01:41:38.785987 23013 tablet_bootstrap.cc:427] T 5c9976bd26bc43c1a026a2148f7079ed P b7b4950f9efa4a829572d08be938050f: Bootstrap starting. I0703 01:41:38.788692 22439 ts_tablet_manager.cc:1173] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552: Registered tablet (data state: TABLET_DATA_READY) I0703 01:41:38.789187 23013 tablet_bootstrap.cc:578] T 5c9976bd26bc43c1a026a2148f7079ed P b7b4950f9efa4a829572d08be938050f: No blocks or log segments found. Creating new log. I0703 01:41:38.789410 23014 ts_tablet_manager.cc:1028] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552: Bootstrapping tablet I0703 01:41:38.789957 23014 tablet_bootstrap.cc:427] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552: Bootstrap starting. I0703 01:41:38.790112 23013 log.cc:525] T 5c9976bd26bc43c1a026a2148f7079ed P b7b4950f9efa4a829572d08be938050f: Log is configured to *not* fsync() on all Append() calls I0703 01:41:38.792114 23014 tablet_bootstrap.cc:578] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552: No blocks or log segments found. Creating new log. I0703 01:41:38.792804 23014 log.cc:525] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552: Log is configured to *not* fsync() on all Append() calls I0703 01:41:38.792621 23011 raft_consensus.cc:325] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7b4950f9efa4a829572d08be938050f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 53950 } } peers { permanent_uuid: "6af80a37db6048b4951d8408b5cf7552" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 58187 } } peers { permanent_uuid: "5f5927fad870472d8e084aaf82f2323b" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 52229 } } I0703 01:41:38.793057 23011 raft_consensus.cc:351] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:41:38.793150 23011 raft_consensus.cc:628] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f5927fad870472d8e084aaf82f2323b, State: Initialized, Role: FOLLOWER I0703 01:41:38.793267 23013 tablet_bootstrap.cc:427] T 5c9976bd26bc43c1a026a2148f7079ed P b7b4950f9efa4a829572d08be938050f: No bootstrap required, opened a new log I0703 01:41:38.793646 23013 ts_tablet_manager.cc:1045] T 5c9976bd26bc43c1a026a2148f7079ed P b7b4950f9efa4a829572d08be938050f: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.002s I0703 01:41:38.793658 23011 consensus_queue.cc:227] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7b4950f9efa4a829572d08be938050f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 53950 } } peers { permanent_uuid: "6af80a37db6048b4951d8408b5cf7552" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 58187 } } peers { permanent_uuid: "5f5927fad870472d8e084aaf82f2323b" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 52229 } } I0703 01:41:38.794350 23014 tablet_bootstrap.cc:427] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552: No bootstrap required, opened a new log I0703 01:41:38.794579 23014 ts_tablet_manager.cc:1045] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s I0703 01:41:38.795158 22869 heartbeater.cc:467] Master 127.5.240.254:51317 was elected leader, sending a full tablet report... I0703 01:41:38.796234 23011 ts_tablet_manager.cc:1073] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b: Time spent starting tablet: real 0.012s user 0.006s sys 0.005s I0703 01:41:38.800650 23013 raft_consensus.cc:325] T 5c9976bd26bc43c1a026a2148f7079ed P b7b4950f9efa4a829572d08be938050f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7b4950f9efa4a829572d08be938050f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 53950 } } peers { permanent_uuid: "6af80a37db6048b4951d8408b5cf7552" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 58187 } } peers { permanent_uuid: "5f5927fad870472d8e084aaf82f2323b" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 52229 } } I0703 01:41:38.800920 23013 raft_consensus.cc:351] T 5c9976bd26bc43c1a026a2148f7079ed P b7b4950f9efa4a829572d08be938050f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:41:38.800985 23013 raft_consensus.cc:628] T 5c9976bd26bc43c1a026a2148f7079ed P b7b4950f9efa4a829572d08be938050f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7b4950f9efa4a829572d08be938050f, State: Initialized, Role: FOLLOWER I0703 01:41:38.801381 23013 consensus_queue.cc:227] T 5c9976bd26bc43c1a026a2148f7079ed P b7b4950f9efa4a829572d08be938050f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7b4950f9efa4a829572d08be938050f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 53950 } } peers { permanent_uuid: "6af80a37db6048b4951d8408b5cf7552" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 58187 } } peers { permanent_uuid: "5f5927fad870472d8e084aaf82f2323b" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 52229 } } I0703 01:41:38.801795 23014 raft_consensus.cc:325] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7b4950f9efa4a829572d08be938050f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 53950 } } peers { permanent_uuid: "6af80a37db6048b4951d8408b5cf7552" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 58187 } } peers { permanent_uuid: "5f5927fad870472d8e084aaf82f2323b" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 52229 } } I0703 01:41:38.802062 23014 raft_consensus.cc:351] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:41:38.802129 23014 raft_consensus.cc:628] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6af80a37db6048b4951d8408b5cf7552, State: Initialized, Role: FOLLOWER I0703 01:41:38.802506 23014 consensus_queue.cc:227] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7b4950f9efa4a829572d08be938050f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 53950 } } peers { permanent_uuid: "6af80a37db6048b4951d8408b5cf7552" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 58187 } } peers { permanent_uuid: "5f5927fad870472d8e084aaf82f2323b" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 52229 } } I0703 01:41:38.803225 22747 heartbeater.cc:467] Master 127.5.240.254:51317 was elected leader, sending a full tablet report... I0703 01:41:38.804126 23013 ts_tablet_manager.cc:1073] T 5c9976bd26bc43c1a026a2148f7079ed P b7b4950f9efa4a829572d08be938050f: Time spent starting tablet: real 0.010s user 0.006s sys 0.003s I0703 01:41:38.804435 23014 ts_tablet_manager.cc:1073] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552: Time spent starting tablet: real 0.010s user 0.007s sys 0.002s W0703 01:41:38.815505 22748 tablet.cc:1678] T 5c9976bd26bc43c1a026a2148f7079ed P b7b4950f9efa4a829572d08be938050f: Can't schedule compaction. Clean time has not been advanced past its initial value. W0703 01:41:38.852223 22503 tablet.cc:1678] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552: Can't schedule compaction. Clean time has not been advanced past its initial value. W0703 01:41:38.891203 22870 tablet.cc:1678] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:41:39.159766 23017 raft_consensus.cc:457] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0703 01:41:39.159890 23017 raft_consensus.cc:2730] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b: Snoozing failure detection for 1.857s (starting election) I0703 01:41:39.160001 23017 raft_consensus.cc:479] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7b4950f9efa4a829572d08be938050f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 53950 } } peers { permanent_uuid: "6af80a37db6048b4951d8408b5cf7552" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 58187 } } peers { permanent_uuid: "5f5927fad870472d8e084aaf82f2323b" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 52229 } } I0703 01:41:39.160778 23017 leader_election.cc:243] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer b7b4950f9efa4a829572d08be938050f (127.5.240.195:53950) I0703 01:41:39.161104 23017 leader_election.cc:243] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 6af80a37db6048b4951d8408b5cf7552 (127.5.240.193:58187) I0703 01:41:39.164178 23019 raft_consensus.cc:457] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0703 01:41:39.164408 23019 raft_consensus.cc:2730] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552: Snoozing failure detection for 1.721s (starting election) I0703 01:41:39.164608 23019 raft_consensus.cc:479] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7b4950f9efa4a829572d08be938050f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 53950 } } peers { permanent_uuid: "6af80a37db6048b4951d8408b5cf7552" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 58187 } } peers { permanent_uuid: "5f5927fad870472d8e084aaf82f2323b" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 52229 } } I0703 01:41:39.165863 23019 leader_election.cc:243] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer b7b4950f9efa4a829572d08be938050f (127.5.240.195:53950) I0703 01:41:39.171450 23019 leader_election.cc:243] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 5f5927fad870472d8e084aaf82f2323b (127.5.240.196:52229) I0703 01:41:39.172073 22459 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "5c9976bd26bc43c1a026a2148f7079ed" candidate_uuid: "5f5927fad870472d8e084aaf82f2323b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af80a37db6048b4951d8408b5cf7552" is_pre_election: true I0703 01:41:39.172032 22704 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "5c9976bd26bc43c1a026a2148f7079ed" candidate_uuid: "5f5927fad870472d8e084aaf82f2323b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7b4950f9efa4a829572d08be938050f" is_pre_election: true I0703 01:41:39.172758 22459 raft_consensus.cc:2730] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552: Snoozing failure detection for 1.567s (vote granted) I0703 01:41:39.172904 22704 raft_consensus.cc:2730] T 5c9976bd26bc43c1a026a2148f7079ed P b7b4950f9efa4a829572d08be938050f: Snoozing failure detection for 1.692s (vote granted) I0703 01:41:39.172935 22459 raft_consensus.cc:2243] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f5927fad870472d8e084aaf82f2323b in term 0. I0703 01:41:39.173029 22704 raft_consensus.cc:2243] T 5c9976bd26bc43c1a026a2148f7079ed P b7b4950f9efa4a829572d08be938050f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f5927fad870472d8e084aaf82f2323b in term 0. I0703 01:41:39.173784 22760 leader_election.cc:379] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b [CANDIDATE]: Term 1 pre-election: Vote granted by peer 6af80a37db6048b4951d8408b5cf7552 (127.5.240.193:58187) I0703 01:41:39.173872 22760 leader_election.cc:270] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0703 01:41:39.174233 22758 leader_election.cc:379] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b [CANDIDATE]: Term 1 pre-election: Vote granted by peer b7b4950f9efa4a829572d08be938050f (127.5.240.195:53950) I0703 01:41:39.174237 23017 raft_consensus.cc:2730] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b: Snoozing failure detection for 1.856s (election complete) I0703 01:41:39.174355 23017 raft_consensus.cc:2537] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b [term 0 FOLLOWER]: Leader pre-election won for term 1 I0703 01:41:39.174456 23017 raft_consensus.cc:457] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0703 01:41:39.174530 23017 raft_consensus.cc:2730] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b: Snoozing failure detection for 1.987s (starting election) I0703 01:41:39.174616 23017 raft_consensus.cc:2782] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b [term 0 FOLLOWER]: Advancing to term 1 I0703 01:41:39.176576 23017 raft_consensus.cc:479] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7b4950f9efa4a829572d08be938050f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 53950 } } peers { permanent_uuid: "6af80a37db6048b4951d8408b5cf7552" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 58187 } } peers { permanent_uuid: "5f5927fad870472d8e084aaf82f2323b" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 52229 } } I0703 01:41:39.177021 23017 leader_election.cc:243] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b [CANDIDATE]: Term 1 election: Requesting vote from peer b7b4950f9efa4a829572d08be938050f (127.5.240.195:53950) I0703 01:41:39.177232 23017 leader_election.cc:243] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b [CANDIDATE]: Term 1 election: Requesting vote from peer 6af80a37db6048b4951d8408b5cf7552 (127.5.240.193:58187) I0703 01:41:39.177752 22704 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "5c9976bd26bc43c1a026a2148f7079ed" candidate_uuid: "5f5927fad870472d8e084aaf82f2323b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7b4950f9efa4a829572d08be938050f" I0703 01:41:39.177894 22704 raft_consensus.cc:2782] T 5c9976bd26bc43c1a026a2148f7079ed P b7b4950f9efa4a829572d08be938050f [term 0 FOLLOWER]: Advancing to term 1 I0703 01:41:39.177932 22459 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "5c9976bd26bc43c1a026a2148f7079ed" candidate_uuid: "5f5927fad870472d8e084aaf82f2323b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6af80a37db6048b4951d8408b5cf7552" I0703 01:41:39.177989 22704 raft_consensus.cc:2730] T 5c9976bd26bc43c1a026a2148f7079ed P b7b4950f9efa4a829572d08be938050f: Snoozing failure detection for 1.970s (vote granted) I0703 01:41:39.178086 22459 raft_consensus.cc:2782] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:41:39.178195 22459 raft_consensus.cc:2730] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552: Snoozing failure detection for 2.038s (vote granted) I0703 01:41:39.179991 22704 raft_consensus.cc:2243] T 5c9976bd26bc43c1a026a2148f7079ed P b7b4950f9efa4a829572d08be938050f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f5927fad870472d8e084aaf82f2323b in term 1. I0703 01:41:39.180389 22758 leader_election.cc:379] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b [CANDIDATE]: Term 1 election: Vote granted by peer b7b4950f9efa4a829572d08be938050f (127.5.240.195:53950) I0703 01:41:39.180479 22758 leader_election.cc:270] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:41:39.180712 23017 raft_consensus.cc:2730] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b: Snoozing failure detection for 2.081s (election complete) I0703 01:41:39.180814 23017 raft_consensus.cc:2537] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:41:39.180872 22459 raft_consensus.cc:2243] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f5927fad870472d8e084aaf82f2323b in term 1. I0703 01:41:39.181674 23017 raft_consensus.cc:593] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b [term 1 LEADER]: Becoming Leader. State: Replica: 5f5927fad870472d8e084aaf82f2323b, State: Running, Role: LEADER I0703 01:41:39.182191 23017 consensus_queue.cc:204] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7b4950f9efa4a829572d08be938050f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 53950 } } peers { permanent_uuid: "6af80a37db6048b4951d8408b5cf7552" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 58187 } } peers { permanent_uuid: "5f5927fad870472d8e084aaf82f2323b" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 52229 } } I0703 01:41:39.182900 22760 leader_election.cc:379] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b [CANDIDATE]: Term 1 election: Vote granted by peer 6af80a37db6048b4951d8408b5cf7552 (127.5.240.193:58187) I0703 01:41:39.183457 22826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "5c9976bd26bc43c1a026a2148f7079ed" candidate_uuid: "6af80a37db6048b4951d8408b5cf7552" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f5927fad870472d8e084aaf82f2323b" is_pre_election: true I0703 01:41:39.184667 22704 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "5c9976bd26bc43c1a026a2148f7079ed" candidate_uuid: "6af80a37db6048b4951d8408b5cf7552" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7b4950f9efa4a829572d08be938050f" is_pre_election: true I0703 01:41:39.184847 22704 raft_consensus.cc:2173] T 5c9976bd26bc43c1a026a2148f7079ed P b7b4950f9efa4a829572d08be938050f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6af80a37db6048b4951d8408b5cf7552 in current term 1: Already voted for candidate 5f5927fad870472d8e084aaf82f2323b in this term. I0703 01:41:39.185886 22826 raft_consensus.cc:2204] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 6af80a37db6048b4951d8408b5cf7552 for term 1 because replica is either leader or believes a valid leader to be alive. I0703 01:41:39.186357 22391 leader_election.cc:393] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552 [CANDIDATE]: Term 1 pre-election: Vote denied by peer b7b4950f9efa4a829572d08be938050f (127.5.240.195:53950). Message: Invalid argument: T 5c9976bd26bc43c1a026a2148f7079ed P b7b4950f9efa4a829572d08be938050f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6af80a37db6048b4951d8408b5cf7552 in current term 1: Already voted for candidate 5f5927fad870472d8e084aaf82f2323b in this term. I0703 01:41:39.187687 22390 leader_election.cc:393] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552 [CANDIDATE]: Term 1 pre-election: Vote denied by peer 5f5927fad870472d8e084aaf82f2323b (127.5.240.196:52229). Message: Invalid argument: T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 6af80a37db6048b4951d8408b5cf7552 for term 1 because replica is either leader or believes a valid leader to be alive. I0703 01:41:39.187763 22390 leader_election.cc:270] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. I0703 01:41:39.188036 23019 raft_consensus.cc:2730] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552: Snoozing failure detection for 2.204s (election complete) I0703 01:41:39.188115 23019 raft_consensus.cc:2485] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority I0703 01:41:39.188360 22337 catalog_manager.cc:3798] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b reported cstate change: term changed from 0 to 1, leader changed from to 5f5927fad870472d8e084aaf82f2323b (127.5.240.196). New cstate: current_term: 1 leader_uuid: "5f5927fad870472d8e084aaf82f2323b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7b4950f9efa4a829572d08be938050f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 53950 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6af80a37db6048b4951d8408b5cf7552" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 58187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f5927fad870472d8e084aaf82f2323b" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 52229 } health_report { overall_health: HEALTHY } } } I0703 01:41:39.203650 6083 external_mini_cluster.cc:490] 5 TS(s) registered with all masters I0703 01:41:39.204692 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 6af80a37db6048b4951d8408b5cf7552 to finish bootstrapping I0703 01:41:39.209659 6083 ts_itest-base.cc:233] Waiting for 0 tablets on tserver a3572ff046ff4c58b4987006387804e1 to finish bootstrapping I0703 01:41:39.215996 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver b7b4950f9efa4a829572d08be938050f to finish bootstrapping I0703 01:41:39.220610 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 5f5927fad870472d8e084aaf82f2323b to finish bootstrapping I0703 01:41:39.224787 6083 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 7dc1b3f7123e460891db0377178f5634 to finish bootstrapping I0703 01:41:39.230713 6083 test_util.cc:208] Using random seed: 1214020587 I0703 01:41:39.239475 6083 test_workload.cc:306] TestWorkload: Skipping table creation because table TestTable already exists I0703 01:41:39.239974 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 22506 I0703 01:41:39.240911 22622 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:41:39.258303 22704 raft_consensus.cc:1081] T 5c9976bd26bc43c1a026a2148f7079ed P b7b4950f9efa4a829572d08be938050f [term 1 FOLLOWER]: Refusing update from remote peer 5f5927fad870472d8e084aaf82f2323b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0703 01:41:39.258428 22459 raft_consensus.cc:1081] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552 [term 1 FOLLOWER]: Refusing update from remote peer 5f5927fad870472d8e084aaf82f2323b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0703 01:41:39.259598 23017 consensus_queue.cc:962] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7b4950f9efa4a829572d08be938050f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 53950 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0703 01:41:39.259829 23025 consensus_queue.cc:962] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6af80a37db6048b4951d8408b5cf7552" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 58187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0703 01:41:39.295835 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 22506 W0703 01:41:39.307158 22321 connection.cc:512] server connection from 127.5.240.194:46617 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:46617): Connection reset by peer (error 104) W0703 01:41:39.591970 23046 flags.cc:406] Enabled unsafe flag: --never_fsync=true I0703 01:41:39.733206 22991 heartbeater.cc:467] Master 127.5.240.254:51317 was elected leader, sending a full tablet report... I0703 01:41:39.838055 22826 consensus_peers.cc:455] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b -> Peer 6af80a37db6048b4951d8408b5cf7552 (127.5.240.193:58187): Closing peer: 6af80a37db6048b4951d8408b5cf7552 I0703 01:41:39.838181 22826 consensus_peers.cc:455] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b -> Peer b7b4950f9efa4a829572d08be938050f (127.5.240.195:53950): Closing peer: b7b4950f9efa4a829572d08be938050f I0703 01:41:39.838377 22826 consensus_queue.cc:204] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 104, Committed index: 104, Last appended: 1.104, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "b7b4950f9efa4a829572d08be938050f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 53950 } } peers { permanent_uuid: "6af80a37db6048b4951d8408b5cf7552" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 58187 } } peers { permanent_uuid: "5f5927fad870472d8e084aaf82f2323b" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 52229 } attrs { replace: true } } peers { permanent_uuid: "7dc1b3f7123e460891db0377178f5634" member_type: NON_VOTER last_known_addr { host: "127.5.240.197" port: 38959 } attrs { promote: true } } I0703 01:41:39.840725 22458 raft_consensus.cc:1081] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552 [term 1 FOLLOWER]: Refusing update from remote peer 5f5927fad870472d8e084aaf82f2323b: Log matching property violated. Preceding OpId in replica: term: 1 index: 104. Preceding OpId from leader: term: 1 index: 105. (index mismatch) I0703 01:41:39.841284 22704 raft_consensus.cc:1081] T 5c9976bd26bc43c1a026a2148f7079ed P b7b4950f9efa4a829572d08be938050f [term 1 FOLLOWER]: Refusing update from remote peer 5f5927fad870472d8e084aaf82f2323b: Log matching property violated. Preceding OpId in replica: term: 1 index: 104. Preceding OpId from leader: term: 1 index: 105. (index mismatch) I0703 01:41:39.841429 23017 consensus_queue.cc:962] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6af80a37db6048b4951d8408b5cf7552" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 58187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 105, Last known committed idx: 103, Time since last communication: 0.000s I0703 01:41:39.842074 23017 consensus_queue.cc:962] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7b4950f9efa4a829572d08be938050f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 53950 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 105, Last known committed idx: 104, Time since last communication: 0.000s I0703 01:41:39.845271 23047 raft_consensus.cc:2684] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b [term 1 LEADER]: Committing config change with OpId 1.105: config changed from index -1 to 105, NON_VOTER 7dc1b3f7123e460891db0377178f5634 (127.5.240.197) added. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "b7b4950f9efa4a829572d08be938050f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 53950 } } peers { permanent_uuid: "6af80a37db6048b4951d8408b5cf7552" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 58187 } } peers { permanent_uuid: "5f5927fad870472d8e084aaf82f2323b" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 52229 } attrs { replace: true } } peers { permanent_uuid: "7dc1b3f7123e460891db0377178f5634" member_type: NON_VOTER last_known_addr { host: "127.5.240.197" port: 38959 } attrs { promote: true } } } I0703 01:41:39.845744 22704 raft_consensus.cc:2684] T 5c9976bd26bc43c1a026a2148f7079ed P b7b4950f9efa4a829572d08be938050f [term 1 FOLLOWER]: Committing config change with OpId 1.105: config changed from index -1 to 105, NON_VOTER 7dc1b3f7123e460891db0377178f5634 (127.5.240.197) added. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "b7b4950f9efa4a829572d08be938050f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 53950 } } peers { permanent_uuid: "6af80a37db6048b4951d8408b5cf7552" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 58187 } } peers { permanent_uuid: "5f5927fad870472d8e084aaf82f2323b" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 52229 } attrs { replace: true } } peers { permanent_uuid: "7dc1b3f7123e460891db0377178f5634" member_type: NON_VOTER last_known_addr { host: "127.5.240.197" port: 38959 } attrs { promote: true } } } I0703 01:41:39.846444 22458 raft_consensus.cc:2684] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552 [term 1 FOLLOWER]: Committing config change with OpId 1.105: config changed from index -1 to 105, NON_VOTER 7dc1b3f7123e460891db0377178f5634 (127.5.240.197) added. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "b7b4950f9efa4a829572d08be938050f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 53950 } } peers { permanent_uuid: "6af80a37db6048b4951d8408b5cf7552" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 58187 } } peers { permanent_uuid: "5f5927fad870472d8e084aaf82f2323b" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 52229 } attrs { replace: true } } peers { permanent_uuid: "7dc1b3f7123e460891db0377178f5634" member_type: NON_VOTER last_known_addr { host: "127.5.240.197" port: 38959 } attrs { promote: true } } } I0703 01:41:39.850797 22337 catalog_manager.cc:3798] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b reported cstate change: config changed from index -1 to 105, NON_VOTER 7dc1b3f7123e460891db0377178f5634 (127.5.240.197) added. New cstate: current_term: 1 leader_uuid: "5f5927fad870472d8e084aaf82f2323b" committed_config { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "b7b4950f9efa4a829572d08be938050f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 53950 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6af80a37db6048b4951d8408b5cf7552" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 58187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f5927fad870472d8e084aaf82f2323b" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 52229 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7dc1b3f7123e460891db0377178f5634" member_type: NON_VOTER last_known_addr { host: "127.5.240.197" port: 38959 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } W0703 01:41:39.853015 22757 consensus_peers.cc:433] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b -> Peer 7dc1b3f7123e460891db0377178f5634 (127.5.240.197:38959): Couldn't send request to peer 7dc1b3f7123e460891db0377178f5634 for tablet 5c9976bd26bc43c1a026a2148f7079ed. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5c9976bd26bc43c1a026a2148f7079ed. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:41:40.276645 23096 ts_tablet_manager.cc:669] T 5c9976bd26bc43c1a026a2148f7079ed P 7dc1b3f7123e460891db0377178f5634: Initiating tablet copy from peer 5f5927fad870472d8e084aaf82f2323b (127.5.240.196:52229) I0703 01:41:40.277076 23096 tablet_copy_client.cc:239] T 5c9976bd26bc43c1a026a2148f7079ed P 7dc1b3f7123e460891db0377178f5634: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.196:52229 I0703 01:41:40.286890 22846 tablet_copy_service.cc:135] P 5f5927fad870472d8e084aaf82f2323b: Received BeginTabletCopySession request for tablet 5c9976bd26bc43c1a026a2148f7079ed from peer 7dc1b3f7123e460891db0377178f5634 ({username='jenkins'} at 127.5.240.197:59500) I0703 01:41:40.287029 22846 tablet_copy_service.cc:156] P 5f5927fad870472d8e084aaf82f2323b: Beginning new tablet copy session on tablet 5c9976bd26bc43c1a026a2148f7079ed from peer 7dc1b3f7123e460891db0377178f5634 at {username='jenkins'} at 127.5.240.197:59500: session id = 7dc1b3f7123e460891db0377178f5634-5c9976bd26bc43c1a026a2148f7079ed I0703 01:41:40.289397 22846 tablet_copy_source_session.cc:202] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:41:40.293047 23096 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 5c9976bd26bc43c1a026a2148f7079ed. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:41:40.307490 23096 ts_tablet_manager.cc:1173] T 5c9976bd26bc43c1a026a2148f7079ed P 7dc1b3f7123e460891db0377178f5634: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:41:40.308018 23096 tablet_copy_client.cc:566] T 5c9976bd26bc43c1a026a2148f7079ed P 7dc1b3f7123e460891db0377178f5634: tablet copy: Starting download of 0 data blocks... I0703 01:41:40.308233 23096 tablet_copy_client.cc:529] T 5c9976bd26bc43c1a026a2148f7079ed P 7dc1b3f7123e460891db0377178f5634: tablet copy: Starting download of 1 WAL segments... I0703 01:41:40.315059 23096 tablet_copy_client.cc:405] T 5c9976bd26bc43c1a026a2148f7079ed P 7dc1b3f7123e460891db0377178f5634: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:41:40.318540 23096 ts_tablet_manager.cc:1028] T 5c9976bd26bc43c1a026a2148f7079ed P 7dc1b3f7123e460891db0377178f5634: Bootstrapping tablet I0703 01:41:40.318886 23096 tablet_bootstrap.cc:427] T 5c9976bd26bc43c1a026a2148f7079ed P 7dc1b3f7123e460891db0377178f5634: Bootstrap starting. I0703 01:41:40.361307 23096 log.cc:525] T 5c9976bd26bc43c1a026a2148f7079ed P 7dc1b3f7123e460891db0377178f5634: Log is configured to *not* fsync() on all Append() calls I0703 01:41:40.560900 23096 tablet_bootstrap.cc:427] T 5c9976bd26bc43c1a026a2148f7079ed P 7dc1b3f7123e460891db0377178f5634: Bootstrap replayed 1/1 log segments. Stats: ops{read=197 overwritten=0 applied=196 ignored=0} inserts{seen=194 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates I0703 01:41:40.561481 23096 tablet_bootstrap.cc:427] T 5c9976bd26bc43c1a026a2148f7079ed P 7dc1b3f7123e460891db0377178f5634: Bootstrap complete. I0703 01:41:40.561950 23096 ts_tablet_manager.cc:1045] T 5c9976bd26bc43c1a026a2148f7079ed P 7dc1b3f7123e460891db0377178f5634: Time spent bootstrapping tablet: real 0.243s user 0.192s sys 0.008s I0703 01:41:40.566728 23096 raft_consensus.cc:325] T 5c9976bd26bc43c1a026a2148f7079ed P 7dc1b3f7123e460891db0377178f5634 [term 1 LEARNER]: Replica starting. Triggering 1 pending transactions. Active config: opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "b7b4950f9efa4a829572d08be938050f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 53950 } } peers { permanent_uuid: "6af80a37db6048b4951d8408b5cf7552" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 58187 } } peers { permanent_uuid: "5f5927fad870472d8e084aaf82f2323b" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 52229 } attrs { replace: true } } peers { permanent_uuid: "7dc1b3f7123e460891db0377178f5634" member_type: NON_VOTER last_known_addr { host: "127.5.240.197" port: 38959 } attrs { promote: true } } I0703 01:41:40.571858 23096 raft_consensus.cc:628] T 5c9976bd26bc43c1a026a2148f7079ed P 7dc1b3f7123e460891db0377178f5634 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7dc1b3f7123e460891db0377178f5634, State: Initialized, Role: LEARNER I0703 01:41:40.572301 23096 consensus_queue.cc:227] T 5c9976bd26bc43c1a026a2148f7079ed P 7dc1b3f7123e460891db0377178f5634 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 196, Last appended: 1.197, Last appended by leader: 197, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "b7b4950f9efa4a829572d08be938050f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 53950 } } peers { permanent_uuid: "6af80a37db6048b4951d8408b5cf7552" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 58187 } } peers { permanent_uuid: "5f5927fad870472d8e084aaf82f2323b" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 52229 } attrs { replace: true } } peers { permanent_uuid: "7dc1b3f7123e460891db0377178f5634" member_type: NON_VOTER last_known_addr { host: "127.5.240.197" port: 38959 } attrs { promote: true } } I0703 01:41:40.574640 23096 ts_tablet_manager.cc:1073] T 5c9976bd26bc43c1a026a2148f7079ed P 7dc1b3f7123e460891db0377178f5634: Time spent starting tablet: real 0.013s user 0.009s sys 0.002s I0703 01:41:40.576395 22846 tablet_copy_service.cc:325] P 5f5927fad870472d8e084aaf82f2323b: Request end of tablet copy session 7dc1b3f7123e460891db0377178f5634-5c9976bd26bc43c1a026a2148f7079ed received from {username='jenkins'} at 127.5.240.197:59500 I0703 01:41:40.576494 22846 tablet_copy_service.cc:413] P 5f5927fad870472d8e084aaf82f2323b: ending tablet copy session 7dc1b3f7123e460891db0377178f5634-5c9976bd26bc43c1a026a2148f7079ed on tablet 5c9976bd26bc43c1a026a2148f7079ed with peer 7dc1b3f7123e460891db0377178f5634 I0703 01:41:40.752916 22948 raft_consensus.cc:1026] T 5c9976bd26bc43c1a026a2148f7079ed P 7dc1b3f7123e460891db0377178f5634 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.104->[1.105-1.286] Dedup: 1.197->[1.198-1.286] I0703 01:41:40.838599 23025 raft_consensus.cc:890] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b: attempting to promote NON_VOTER 7dc1b3f7123e460891db0377178f5634 to VOTER I0703 01:41:40.839100 23025 consensus_peers.cc:455] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b -> Peer 7dc1b3f7123e460891db0377178f5634 (127.5.240.197:38959): Closing peer: 7dc1b3f7123e460891db0377178f5634 I0703 01:41:40.839202 23025 consensus_peers.cc:455] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b -> Peer 6af80a37db6048b4951d8408b5cf7552 (127.5.240.193:58187): Closing peer: 6af80a37db6048b4951d8408b5cf7552 I0703 01:41:40.839531 23025 consensus_peers.cc:455] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b -> Peer b7b4950f9efa4a829572d08be938050f (127.5.240.195:53950): Closing peer: b7b4950f9efa4a829572d08be938050f I0703 01:41:40.839843 23025 consensus_queue.cc:204] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 302, Committed index: 302, Last appended: 1.303, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "b7b4950f9efa4a829572d08be938050f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 53950 } } peers { permanent_uuid: "6af80a37db6048b4951d8408b5cf7552" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 58187 } } peers { permanent_uuid: "5f5927fad870472d8e084aaf82f2323b" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 52229 } attrs { replace: true } } peers { permanent_uuid: "7dc1b3f7123e460891db0377178f5634" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 38959 } attrs { promote: false } } I0703 01:41:40.844679 22459 raft_consensus.cc:1081] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552 [term 1 FOLLOWER]: Refusing update from remote peer 5f5927fad870472d8e084aaf82f2323b: Log matching property violated. Preceding OpId in replica: term: 1 index: 303. Preceding OpId from leader: term: 1 index: 304. (index mismatch) I0703 01:41:40.844764 22948 raft_consensus.cc:1081] T 5c9976bd26bc43c1a026a2148f7079ed P 7dc1b3f7123e460891db0377178f5634 [term 1 LEARNER]: Refusing update from remote peer 5f5927fad870472d8e084aaf82f2323b: Log matching property violated. Preceding OpId in replica: term: 1 index: 303. Preceding OpId from leader: term: 1 index: 304. (index mismatch) I0703 01:41:40.845837 23091 consensus_queue.cc:962] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6af80a37db6048b4951d8408b5cf7552" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 58187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 304, Last known committed idx: 302, Time since last communication: 0.000s I0703 01:41:40.846359 23091 consensus_queue.cc:962] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7dc1b3f7123e460891db0377178f5634" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 38959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 304, Last known committed idx: 302, Time since last communication: 0.000s I0703 01:41:40.848388 22703 raft_consensus.cc:1081] T 5c9976bd26bc43c1a026a2148f7079ed P b7b4950f9efa4a829572d08be938050f [term 1 FOLLOWER]: Refusing update from remote peer 5f5927fad870472d8e084aaf82f2323b: Log matching property violated. Preceding OpId in replica: term: 1 index: 303. Preceding OpId from leader: term: 1 index: 304. (index mismatch) I0703 01:41:40.849280 23017 consensus_queue.cc:962] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7b4950f9efa4a829572d08be938050f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 53950 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 304, Last known committed idx: 303, Time since last communication: 0.000s I0703 01:41:40.851552 22459 raft_consensus.cc:2684] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552 [term 1 FOLLOWER]: Committing config change with OpId 1.304: config changed from index 105 to 304, 7dc1b3f7123e460891db0377178f5634 (127.5.240.197) changed from NON_VOTER to VOTER. New config: { opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "b7b4950f9efa4a829572d08be938050f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 53950 } } peers { permanent_uuid: "6af80a37db6048b4951d8408b5cf7552" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 58187 } } peers { permanent_uuid: "5f5927fad870472d8e084aaf82f2323b" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 52229 } attrs { replace: true } } peers { permanent_uuid: "7dc1b3f7123e460891db0377178f5634" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 38959 } attrs { promote: false } } } I0703 01:41:40.849752 23025 raft_consensus.cc:2684] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b [term 1 LEADER]: Committing config change with OpId 1.304: config changed from index 105 to 304, 7dc1b3f7123e460891db0377178f5634 (127.5.240.197) changed from NON_VOTER to VOTER. New config: { opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "b7b4950f9efa4a829572d08be938050f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 53950 } } peers { permanent_uuid: "6af80a37db6048b4951d8408b5cf7552" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 58187 } } peers { permanent_uuid: "5f5927fad870472d8e084aaf82f2323b" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 52229 } attrs { replace: true } } peers { permanent_uuid: "7dc1b3f7123e460891db0377178f5634" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 38959 } attrs { promote: false } } } I0703 01:41:40.852056 22948 raft_consensus.cc:2684] T 5c9976bd26bc43c1a026a2148f7079ed P 7dc1b3f7123e460891db0377178f5634 [term 1 FOLLOWER]: Committing config change with OpId 1.304: config changed from index 105 to 304, 7dc1b3f7123e460891db0377178f5634 (127.5.240.197) changed from NON_VOTER to VOTER. New config: { opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "b7b4950f9efa4a829572d08be938050f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 53950 } } peers { permanent_uuid: "6af80a37db6048b4951d8408b5cf7552" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 58187 } } peers { permanent_uuid: "5f5927fad870472d8e084aaf82f2323b" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 52229 } attrs { replace: true } } peers { permanent_uuid: "7dc1b3f7123e460891db0377178f5634" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 38959 } attrs { promote: false } } } I0703 01:41:40.853380 22703 raft_consensus.cc:2684] T 5c9976bd26bc43c1a026a2148f7079ed P b7b4950f9efa4a829572d08be938050f [term 1 FOLLOWER]: Committing config change with OpId 1.304: config changed from index 105 to 304, 7dc1b3f7123e460891db0377178f5634 (127.5.240.197) changed from NON_VOTER to VOTER. New config: { opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "b7b4950f9efa4a829572d08be938050f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 53950 } } peers { permanent_uuid: "6af80a37db6048b4951d8408b5cf7552" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 58187 } } peers { permanent_uuid: "5f5927fad870472d8e084aaf82f2323b" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 52229 } attrs { replace: true } } peers { permanent_uuid: "7dc1b3f7123e460891db0377178f5634" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 38959 } attrs { promote: false } } } I0703 01:41:40.856425 22336 catalog_manager.cc:3798] T 5c9976bd26bc43c1a026a2148f7079ed P 7dc1b3f7123e460891db0377178f5634 reported cstate change: config changed from index 105 to 304, 7dc1b3f7123e460891db0377178f5634 (127.5.240.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5f5927fad870472d8e084aaf82f2323b" committed_config { opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "b7b4950f9efa4a829572d08be938050f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 53950 } } peers { permanent_uuid: "6af80a37db6048b4951d8408b5cf7552" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 58187 } } peers { permanent_uuid: "5f5927fad870472d8e084aaf82f2323b" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 52229 } attrs { replace: true } } peers { permanent_uuid: "7dc1b3f7123e460891db0377178f5634" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 38959 } attrs { promote: false } } } I0703 01:41:40.944900 22826 tablet_service.cc:1140] Received LeaderStepDown RPC: tablet_id: "5c9976bd26bc43c1a026a2148f7079ed" dest_uuid: "5f5927fad870472d8e084aaf82f2323b" from {username='jenkins'} at 127.5.240.196:40899 I0703 01:41:40.945086 22826 raft_consensus.cc:552] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b [term 1 LEADER]: Received request to step down I0703 01:41:40.945176 22826 raft_consensus.cc:628] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5f5927fad870472d8e084aaf82f2323b, State: Running, Role: LEADER I0703 01:41:40.945544 22826 consensus_queue.cc:227] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 318, Committed index: 318, Last appended: 1.319, Last appended by leader: 319, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "b7b4950f9efa4a829572d08be938050f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 53950 } } peers { permanent_uuid: "6af80a37db6048b4951d8408b5cf7552" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 58187 } } peers { permanent_uuid: "5f5927fad870472d8e084aaf82f2323b" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 52229 } attrs { replace: true } } peers { permanent_uuid: "7dc1b3f7123e460891db0377178f5634" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 38959 } attrs { promote: false } } I0703 01:41:40.945797 22826 consensus_peers.cc:455] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b -> Peer 7dc1b3f7123e460891db0377178f5634 (127.5.240.197:38959): Closing peer: 7dc1b3f7123e460891db0377178f5634 I0703 01:41:40.945864 22826 consensus_peers.cc:455] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b -> Peer 6af80a37db6048b4951d8408b5cf7552 (127.5.240.193:58187): Closing peer: 6af80a37db6048b4951d8408b5cf7552 I0703 01:41:40.945914 22826 consensus_peers.cc:455] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b -> Peer b7b4950f9efa4a829572d08be938050f (127.5.240.195:53950): Closing peer: b7b4950f9efa4a829572d08be938050f I0703 01:41:40.945981 22826 raft_consensus.cc:2730] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b: Snoozing failure detection for 3.000s (explicit stepdown request) W0703 01:41:40.946116 23091 consensus_queue.cc:1041] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "7dc1b3f7123e460891db0377178f5634" responder_term: 1 status { last_received { term: 1 index: 319 } last_committed_idx: 318 last_received_current_leader { term: 1 index: 319 } } I0703 01:41:42.444803 23141 raft_consensus.cc:457] T 5c9976bd26bc43c1a026a2148f7079ed P b7b4950f9efa4a829572d08be938050f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f5927fad870472d8e084aaf82f2323b) I0703 01:41:42.444871 23142 raft_consensus.cc:457] T 5c9976bd26bc43c1a026a2148f7079ed P 7dc1b3f7123e460891db0377178f5634 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f5927fad870472d8e084aaf82f2323b) I0703 01:41:42.444905 23141 raft_consensus.cc:2730] T 5c9976bd26bc43c1a026a2148f7079ed P b7b4950f9efa4a829572d08be938050f: Snoozing failure detection for 1.941s (starting election) I0703 01:41:42.444952 23142 raft_consensus.cc:2730] T 5c9976bd26bc43c1a026a2148f7079ed P 7dc1b3f7123e460891db0377178f5634: Snoozing failure detection for 2.201s (starting election) I0703 01:41:42.445019 23141 raft_consensus.cc:479] T 5c9976bd26bc43c1a026a2148f7079ed P b7b4950f9efa4a829572d08be938050f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "b7b4950f9efa4a829572d08be938050f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 53950 } } peers { permanent_uuid: "6af80a37db6048b4951d8408b5cf7552" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 58187 } } peers { permanent_uuid: "5f5927fad870472d8e084aaf82f2323b" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 52229 } attrs { replace: true } } peers { permanent_uuid: "7dc1b3f7123e460891db0377178f5634" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 38959 } attrs { promote: false } } I0703 01:41:42.445045 23142 raft_consensus.cc:479] T 5c9976bd26bc43c1a026a2148f7079ed P 7dc1b3f7123e460891db0377178f5634 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "b7b4950f9efa4a829572d08be938050f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 53950 } } peers { permanent_uuid: "6af80a37db6048b4951d8408b5cf7552" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 58187 } } peers { permanent_uuid: "5f5927fad870472d8e084aaf82f2323b" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 52229 } attrs { replace: true } } peers { permanent_uuid: "7dc1b3f7123e460891db0377178f5634" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 38959 } attrs { promote: false } } I0703 01:41:42.445812 23141 leader_election.cc:243] T 5c9976bd26bc43c1a026a2148f7079ed P b7b4950f9efa4a829572d08be938050f [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 6af80a37db6048b4951d8408b5cf7552 (127.5.240.193:58187) I0703 01:41:42.445818 23142 leader_election.cc:243] T 5c9976bd26bc43c1a026a2148f7079ed P 7dc1b3f7123e460891db0377178f5634 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer b7b4950f9efa4a829572d08be938050f (127.5.240.195:53950) I0703 01:41:42.446089 23142 leader_election.cc:243] T 5c9976bd26bc43c1a026a2148f7079ed P 7dc1b3f7123e460891db0377178f5634 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 6af80a37db6048b4951d8408b5cf7552 (127.5.240.193:58187) I0703 01:41:42.446121 23141 leader_election.cc:243] T 5c9976bd26bc43c1a026a2148f7079ed P b7b4950f9efa4a829572d08be938050f [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 5f5927fad870472d8e084aaf82f2323b (127.5.240.196:52229) I0703 01:41:42.450811 23142 leader_election.cc:243] T 5c9976bd26bc43c1a026a2148f7079ed P 7dc1b3f7123e460891db0377178f5634 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 5f5927fad870472d8e084aaf82f2323b (127.5.240.196:52229) I0703 01:41:42.451828 22826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "5c9976bd26bc43c1a026a2148f7079ed" candidate_uuid: "7dc1b3f7123e460891db0377178f5634" candidate_term: 2 candidate_status { last_received { term: 1 index: 319 } } ignore_live_leader: false dest_uuid: "5f5927fad870472d8e084aaf82f2323b" is_pre_election: true I0703 01:41:42.452019 22826 raft_consensus.cc:2730] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b: Snoozing failure detection for 2.213s (vote granted) I0703 01:41:42.452091 22826 raft_consensus.cc:2243] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7dc1b3f7123e460891db0377178f5634 in term 1. I0703 01:41:42.452744 22879 leader_election.cc:379] T 5c9976bd26bc43c1a026a2148f7079ed P 7dc1b3f7123e460891db0377178f5634 [CANDIDATE]: Term 2 pre-election: Vote granted by peer 5f5927fad870472d8e084aaf82f2323b (127.5.240.196:52229) I0703 01:41:42.457011 23141 leader_election.cc:243] T 5c9976bd26bc43c1a026a2148f7079ed P b7b4950f9efa4a829572d08be938050f [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 7dc1b3f7123e460891db0377178f5634 (127.5.240.197:38959) I0703 01:41:42.458670 23143 raft_consensus.cc:457] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5f5927fad870472d8e084aaf82f2323b) I0703 01:41:42.458937 23143 raft_consensus.cc:2730] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552: Snoozing failure detection for 1.534s (starting election) I0703 01:41:42.459108 23143 raft_consensus.cc:479] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "b7b4950f9efa4a829572d08be938050f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 53950 } } peers { permanent_uuid: "6af80a37db6048b4951d8408b5cf7552" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 58187 } } peers { permanent_uuid: "5f5927fad870472d8e084aaf82f2323b" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 52229 } attrs { replace: true } } peers { permanent_uuid: "7dc1b3f7123e460891db0377178f5634" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 38959 } attrs { promote: false } } I0703 01:41:42.460103 23143 leader_election.cc:243] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer b7b4950f9efa4a829572d08be938050f (127.5.240.195:53950) I0703 01:41:42.460325 23143 leader_election.cc:243] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 5f5927fad870472d8e084aaf82f2323b (127.5.240.196:52229) I0703 01:41:42.462744 22703 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "5c9976bd26bc43c1a026a2148f7079ed" candidate_uuid: "6af80a37db6048b4951d8408b5cf7552" candidate_term: 2 candidate_status { last_received { term: 1 index: 319 } } ignore_live_leader: false dest_uuid: "b7b4950f9efa4a829572d08be938050f" is_pre_election: true I0703 01:41:42.463057 22703 raft_consensus.cc:2730] T 5c9976bd26bc43c1a026a2148f7079ed P b7b4950f9efa4a829572d08be938050f: Snoozing failure detection for 1.733s (vote granted) I0703 01:41:42.463498 22703 raft_consensus.cc:2243] T 5c9976bd26bc43c1a026a2148f7079ed P b7b4950f9efa4a829572d08be938050f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6af80a37db6048b4951d8408b5cf7552 in term 1. I0703 01:41:42.464246 22391 leader_election.cc:379] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552 [CANDIDATE]: Term 2 pre-election: Vote granted by peer b7b4950f9efa4a829572d08be938050f (127.5.240.195:53950) I0703 01:41:42.464834 23143 leader_election.cc:243] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 7dc1b3f7123e460891db0377178f5634 (127.5.240.197:38959) I0703 01:41:42.464922 22826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "5c9976bd26bc43c1a026a2148f7079ed" candidate_uuid: "6af80a37db6048b4951d8408b5cf7552" candidate_term: 2 candidate_status { last_received { term: 1 index: 319 } } ignore_live_leader: false dest_uuid: "5f5927fad870472d8e084aaf82f2323b" is_pre_election: true I0703 01:41:42.465050 22826 raft_consensus.cc:2730] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b: Snoozing failure detection for 1.773s (vote granted) I0703 01:41:42.465102 22826 raft_consensus.cc:2243] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6af80a37db6048b4951d8408b5cf7552 in term 1. I0703 01:41:42.467672 22390 leader_election.cc:379] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552 [CANDIDATE]: Term 2 pre-election: Vote granted by peer 5f5927fad870472d8e084aaf82f2323b (127.5.240.196:52229) I0703 01:41:42.467731 22390 leader_election.cc:270] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. I0703 01:41:42.468420 23143 raft_consensus.cc:2730] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552: Snoozing failure detection for 1.550s (election complete) I0703 01:41:42.468488 23143 raft_consensus.cc:2537] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0703 01:41:42.468542 23143 raft_consensus.cc:457] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5f5927fad870472d8e084aaf82f2323b) I0703 01:41:42.469132 23143 raft_consensus.cc:2730] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552: Snoozing failure detection for 2.088s (starting election) I0703 01:41:42.469352 23143 raft_consensus.cc:2782] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552 [term 1 FOLLOWER]: Advancing to term 2 I0703 01:41:42.469457 22459 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "5c9976bd26bc43c1a026a2148f7079ed" candidate_uuid: "b7b4950f9efa4a829572d08be938050f" candidate_term: 2 candidate_status { last_received { term: 1 index: 319 } } ignore_live_leader: false dest_uuid: "6af80a37db6048b4951d8408b5cf7552" is_pre_election: true I0703 01:41:42.474287 22826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "5c9976bd26bc43c1a026a2148f7079ed" candidate_uuid: "b7b4950f9efa4a829572d08be938050f" candidate_term: 2 candidate_status { last_received { term: 1 index: 319 } } ignore_live_leader: false dest_uuid: "5f5927fad870472d8e084aaf82f2323b" is_pre_election: true I0703 01:41:42.474484 22826 raft_consensus.cc:2730] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b: Snoozing failure detection for 1.707s (vote granted) I0703 01:41:42.474546 22826 raft_consensus.cc:2243] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7b4950f9efa4a829572d08be938050f in term 1. I0703 01:41:42.477042 22635 leader_election.cc:379] T 5c9976bd26bc43c1a026a2148f7079ed P b7b4950f9efa4a829572d08be938050f [CANDIDATE]: Term 2 pre-election: Vote granted by peer 5f5927fad870472d8e084aaf82f2323b (127.5.240.196:52229) I0703 01:41:42.477819 22703 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "5c9976bd26bc43c1a026a2148f7079ed" candidate_uuid: "7dc1b3f7123e460891db0377178f5634" candidate_term: 2 candidate_status { last_received { term: 1 index: 319 } } ignore_live_leader: false dest_uuid: "b7b4950f9efa4a829572d08be938050f" is_pre_election: true I0703 01:41:42.477973 22703 raft_consensus.cc:2730] T 5c9976bd26bc43c1a026a2148f7079ed P b7b4950f9efa4a829572d08be938050f: Snoozing failure detection for 2.196s (vote granted) I0703 01:41:42.478041 22703 raft_consensus.cc:2243] T 5c9976bd26bc43c1a026a2148f7079ed P b7b4950f9efa4a829572d08be938050f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7dc1b3f7123e460891db0377178f5634 in term 1. I0703 01:41:42.478524 22880 leader_election.cc:379] T 5c9976bd26bc43c1a026a2148f7079ed P 7dc1b3f7123e460891db0377178f5634 [CANDIDATE]: Term 2 pre-election: Vote granted by peer b7b4950f9efa4a829572d08be938050f (127.5.240.195:53950) I0703 01:41:42.478617 22880 leader_election.cc:270] T 5c9976bd26bc43c1a026a2148f7079ed P 7dc1b3f7123e460891db0377178f5634 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. I0703 01:41:42.478955 23142 raft_consensus.cc:2730] T 5c9976bd26bc43c1a026a2148f7079ed P 7dc1b3f7123e460891db0377178f5634: Snoozing failure detection for 1.798s (election complete) I0703 01:41:42.479055 23142 raft_consensus.cc:2537] T 5c9976bd26bc43c1a026a2148f7079ed P 7dc1b3f7123e460891db0377178f5634 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0703 01:41:42.479154 23142 raft_consensus.cc:457] T 5c9976bd26bc43c1a026a2148f7079ed P 7dc1b3f7123e460891db0377178f5634 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5f5927fad870472d8e084aaf82f2323b) I0703 01:41:42.479228 23142 raft_consensus.cc:2730] T 5c9976bd26bc43c1a026a2148f7079ed P 7dc1b3f7123e460891db0377178f5634: Snoozing failure detection for 1.549s (starting election) I0703 01:41:42.479323 23142 raft_consensus.cc:2782] T 5c9976bd26bc43c1a026a2148f7079ed P 7dc1b3f7123e460891db0377178f5634 [term 1 FOLLOWER]: Advancing to term 2 I0703 01:41:42.480247 23143 raft_consensus.cc:479] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "b7b4950f9efa4a829572d08be938050f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 53950 } } peers { permanent_uuid: "6af80a37db6048b4951d8408b5cf7552" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 58187 } } peers { permanent_uuid: "5f5927fad870472d8e084aaf82f2323b" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 52229 } attrs { replace: true } } peers { permanent_uuid: "7dc1b3f7123e460891db0377178f5634" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 38959 } attrs { promote: false } } I0703 01:41:42.480943 22459 raft_consensus.cc:2173] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b7b4950f9efa4a829572d08be938050f in current term 2: Already voted for candidate 6af80a37db6048b4951d8408b5cf7552 in this term. I0703 01:41:42.481581 23143 leader_election.cc:243] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552 [CANDIDATE]: Term 2 election: Requesting vote from peer b7b4950f9efa4a829572d08be938050f (127.5.240.195:53950) I0703 01:41:42.482231 23143 leader_election.cc:243] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552 [CANDIDATE]: Term 2 election: Requesting vote from peer 5f5927fad870472d8e084aaf82f2323b (127.5.240.196:52229) I0703 01:41:42.482311 22703 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "5c9976bd26bc43c1a026a2148f7079ed" candidate_uuid: "6af80a37db6048b4951d8408b5cf7552" candidate_term: 2 candidate_status { last_received { term: 1 index: 319 } } ignore_live_leader: false dest_uuid: "b7b4950f9efa4a829572d08be938050f" I0703 01:41:42.482257 23142 raft_consensus.cc:479] T 5c9976bd26bc43c1a026a2148f7079ed P 7dc1b3f7123e460891db0377178f5634 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "b7b4950f9efa4a829572d08be938050f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 53950 } } peers { permanent_uuid: "6af80a37db6048b4951d8408b5cf7552" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 58187 } } peers { permanent_uuid: "5f5927fad870472d8e084aaf82f2323b" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 52229 } attrs { replace: true } } peers { permanent_uuid: "7dc1b3f7123e460891db0377178f5634" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 38959 } attrs { promote: false } } I0703 01:41:42.482434 22703 raft_consensus.cc:2782] T 5c9976bd26bc43c1a026a2148f7079ed P b7b4950f9efa4a829572d08be938050f [term 1 FOLLOWER]: Advancing to term 2 I0703 01:41:42.482527 22703 raft_consensus.cc:2730] T 5c9976bd26bc43c1a026a2148f7079ed P b7b4950f9efa4a829572d08be938050f: Snoozing failure detection for 2.032s (vote granted) I0703 01:41:42.482671 23143 leader_election.cc:243] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552 [CANDIDATE]: Term 2 election: Requesting vote from peer 7dc1b3f7123e460891db0377178f5634 (127.5.240.197:38959) I0703 01:41:42.482839 23142 leader_election.cc:243] T 5c9976bd26bc43c1a026a2148f7079ed P 7dc1b3f7123e460891db0377178f5634 [CANDIDATE]: Term 2 election: Requesting vote from peer b7b4950f9efa4a829572d08be938050f (127.5.240.195:53950) I0703 01:41:42.482973 23142 leader_election.cc:243] T 5c9976bd26bc43c1a026a2148f7079ed P 7dc1b3f7123e460891db0377178f5634 [CANDIDATE]: Term 2 election: Requesting vote from peer 6af80a37db6048b4951d8408b5cf7552 (127.5.240.193:58187) I0703 01:41:42.483011 22826 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "5c9976bd26bc43c1a026a2148f7079ed" candidate_uuid: "6af80a37db6048b4951d8408b5cf7552" candidate_term: 2 candidate_status { last_received { term: 1 index: 319 } } ignore_live_leader: false dest_uuid: "5f5927fad870472d8e084aaf82f2323b" I0703 01:41:42.483096 23142 leader_election.cc:243] T 5c9976bd26bc43c1a026a2148f7079ed P 7dc1b3f7123e460891db0377178f5634 [CANDIDATE]: Term 2 election: Requesting vote from peer 5f5927fad870472d8e084aaf82f2323b (127.5.240.196:52229) I0703 01:41:42.483139 22826 raft_consensus.cc:2782] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b [term 1 FOLLOWER]: Advancing to term 2 I0703 01:41:42.483235 22826 raft_consensus.cc:2730] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b: Snoozing failure detection for 2.110s (vote granted) I0703 01:41:42.483952 22638 leader_election.cc:393] T 5c9976bd26bc43c1a026a2148f7079ed P b7b4950f9efa4a829572d08be938050f [CANDIDATE]: Term 2 pre-election: Vote denied by peer 6af80a37db6048b4951d8408b5cf7552 (127.5.240.193:58187). Message: Invalid argument: T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b7b4950f9efa4a829572d08be938050f in current term 2: Already voted for candidate 6af80a37db6048b4951d8408b5cf7552 in this term. I0703 01:41:42.484216 22704 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "5c9976bd26bc43c1a026a2148f7079ed" candidate_uuid: "7dc1b3f7123e460891db0377178f5634" candidate_term: 2 candidate_status { last_received { term: 1 index: 319 } } ignore_live_leader: false dest_uuid: "b7b4950f9efa4a829572d08be938050f" I0703 01:41:42.484683 22948 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "5c9976bd26bc43c1a026a2148f7079ed" candidate_uuid: "b7b4950f9efa4a829572d08be938050f" candidate_term: 2 candidate_status { last_received { term: 1 index: 319 } } ignore_live_leader: false dest_uuid: "7dc1b3f7123e460891db0377178f5634" is_pre_election: true I0703 01:41:42.484760 22459 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "5c9976bd26bc43c1a026a2148f7079ed" candidate_uuid: "7dc1b3f7123e460891db0377178f5634" candidate_term: 2 candidate_status { last_received { term: 1 index: 319 } } ignore_live_leader: false dest_uuid: "6af80a37db6048b4951d8408b5cf7552" is_pre_election: true I0703 01:41:42.484890 22948 raft_consensus.cc:2173] T 5c9976bd26bc43c1a026a2148f7079ed P 7dc1b3f7123e460891db0377178f5634 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b7b4950f9efa4a829572d08be938050f in current term 2: Already voted for candidate 7dc1b3f7123e460891db0377178f5634 in this term. I0703 01:41:42.484910 22459 raft_consensus.cc:2173] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7dc1b3f7123e460891db0377178f5634 in current term 2: Already voted for candidate 6af80a37db6048b4951d8408b5cf7552 in this term. I0703 01:41:42.485061 22703 raft_consensus.cc:2243] T 5c9976bd26bc43c1a026a2148f7079ed P b7b4950f9efa4a829572d08be938050f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6af80a37db6048b4951d8408b5cf7552 in term 2. I0703 01:41:42.485293 22826 raft_consensus.cc:2243] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6af80a37db6048b4951d8408b5cf7552 in term 2. I0703 01:41:42.485627 22635 leader_election.cc:393] T 5c9976bd26bc43c1a026a2148f7079ed P b7b4950f9efa4a829572d08be938050f [CANDIDATE]: Term 2 pre-election: Vote denied by peer 7dc1b3f7123e460891db0377178f5634 (127.5.240.197:38959). Message: Invalid argument: T 5c9976bd26bc43c1a026a2148f7079ed P 7dc1b3f7123e460891db0377178f5634 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b7b4950f9efa4a829572d08be938050f in current term 2: Already voted for candidate 7dc1b3f7123e460891db0377178f5634 in this term. I0703 01:41:42.485558 22825 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "5c9976bd26bc43c1a026a2148f7079ed" candidate_uuid: "7dc1b3f7123e460891db0377178f5634" candidate_term: 2 candidate_status { last_received { term: 1 index: 319 } } ignore_live_leader: false dest_uuid: "5f5927fad870472d8e084aaf82f2323b" I0703 01:41:42.485741 22635 leader_election.cc:270] T 5c9976bd26bc43c1a026a2148f7079ed P b7b4950f9efa4a829572d08be938050f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0703 01:41:42.485771 22825 raft_consensus.cc:2173] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7dc1b3f7123e460891db0377178f5634 in current term 2: Already voted for candidate 6af80a37db6048b4951d8408b5cf7552 in this term. I0703 01:41:42.485951 22704 raft_consensus.cc:2218] T 5c9976bd26bc43c1a026a2148f7079ed P b7b4950f9efa4a829572d08be938050f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7dc1b3f7123e460891db0377178f5634 for term 2 because replica is already servicing an update from a current leader or another vote. I0703 01:41:42.486254 23141 raft_consensus.cc:2730] T 5c9976bd26bc43c1a026a2148f7079ed P b7b4950f9efa4a829572d08be938050f: Snoozing failure detection for 1.648s (election complete) I0703 01:41:42.486325 23141 raft_consensus.cc:2485] T 5c9976bd26bc43c1a026a2148f7079ed P b7b4950f9efa4a829572d08be938050f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0703 01:41:42.486548 22391 leader_election.cc:379] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552 [CANDIDATE]: Term 2 election: Vote granted by peer b7b4950f9efa4a829572d08be938050f (127.5.240.195:53950) I0703 01:41:42.486730 22390 leader_election.cc:379] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552 [CANDIDATE]: Term 2 election: Vote granted by peer 5f5927fad870472d8e084aaf82f2323b (127.5.240.196:52229) I0703 01:41:42.486788 22390 leader_election.cc:270] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0703 01:41:42.486770 22882 leader_election.cc:393] T 5c9976bd26bc43c1a026a2148f7079ed P 7dc1b3f7123e460891db0377178f5634 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 6af80a37db6048b4951d8408b5cf7552 (127.5.240.193:58187). Message: Invalid argument: T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7dc1b3f7123e460891db0377178f5634 in current term 2: Already voted for candidate 6af80a37db6048b4951d8408b5cf7552 in this term. I0703 01:41:42.486989 23143 raft_consensus.cc:2730] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552: Snoozing failure detection for 1.718s (election complete) I0703 01:41:42.486979 22879 leader_election.cc:393] T 5c9976bd26bc43c1a026a2148f7079ed P 7dc1b3f7123e460891db0377178f5634 [CANDIDATE]: Term 2 election: Vote denied by peer 5f5927fad870472d8e084aaf82f2323b (127.5.240.196:52229). Message: Invalid argument: T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7dc1b3f7123e460891db0377178f5634 in current term 2: Already voted for candidate 6af80a37db6048b4951d8408b5cf7552 in this term. I0703 01:41:42.487061 23143 raft_consensus.cc:2537] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552 [term 2 FOLLOWER]: Leader election won for term 2 I0703 01:41:42.487357 22948 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "5c9976bd26bc43c1a026a2148f7079ed" candidate_uuid: "6af80a37db6048b4951d8408b5cf7552" candidate_term: 2 candidate_status { last_received { term: 1 index: 319 } } ignore_live_leader: false dest_uuid: "7dc1b3f7123e460891db0377178f5634" is_pre_election: true I0703 01:41:42.487515 22948 raft_consensus.cc:2173] T 5c9976bd26bc43c1a026a2148f7079ed P 7dc1b3f7123e460891db0377178f5634 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6af80a37db6048b4951d8408b5cf7552 in current term 2: Already voted for candidate 7dc1b3f7123e460891db0377178f5634 in this term. I0703 01:41:42.487748 22459 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "5c9976bd26bc43c1a026a2148f7079ed" candidate_uuid: "7dc1b3f7123e460891db0377178f5634" candidate_term: 2 candidate_status { last_received { term: 1 index: 319 } } ignore_live_leader: false dest_uuid: "6af80a37db6048b4951d8408b5cf7552" I0703 01:41:42.487879 22880 leader_election.cc:393] T 5c9976bd26bc43c1a026a2148f7079ed P 7dc1b3f7123e460891db0377178f5634 [CANDIDATE]: Term 2 election: Vote denied by peer b7b4950f9efa4a829572d08be938050f (127.5.240.195:53950). Message: Service unavailable: T 5c9976bd26bc43c1a026a2148f7079ed P b7b4950f9efa4a829572d08be938050f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 7dc1b3f7123e460891db0377178f5634 for term 2 because replica is already servicing an update from a current leader or another vote. I0703 01:41:42.487946 22880 leader_election.cc:270] T 5c9976bd26bc43c1a026a2148f7079ed P 7dc1b3f7123e460891db0377178f5634 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. I0703 01:41:42.488099 23142 raft_consensus.cc:2730] T 5c9976bd26bc43c1a026a2148f7079ed P 7dc1b3f7123e460891db0377178f5634: Snoozing failure detection for 1.881s (election complete) I0703 01:41:42.488160 23142 raft_consensus.cc:2485] T 5c9976bd26bc43c1a026a2148f7079ed P 7dc1b3f7123e460891db0377178f5634 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority I0703 01:41:42.488128 23143 raft_consensus.cc:593] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552 [term 2 LEADER]: Becoming Leader. State: Replica: 6af80a37db6048b4951d8408b5cf7552, State: Running, Role: LEADER I0703 01:41:42.488535 22948 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "5c9976bd26bc43c1a026a2148f7079ed" candidate_uuid: "6af80a37db6048b4951d8408b5cf7552" candidate_term: 2 candidate_status { last_received { term: 1 index: 319 } } ignore_live_leader: false dest_uuid: "7dc1b3f7123e460891db0377178f5634" I0703 01:41:42.488723 22948 raft_consensus.cc:2173] T 5c9976bd26bc43c1a026a2148f7079ed P 7dc1b3f7123e460891db0377178f5634 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 6af80a37db6048b4951d8408b5cf7552 in current term 2: Already voted for candidate 7dc1b3f7123e460891db0377178f5634 in this term. I0703 01:41:42.488600 23143 consensus_queue.cc:204] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 318, Committed index: 318, Last appended: 1.319, Last appended by leader: 319, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "b7b4950f9efa4a829572d08be938050f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 53950 } } peers { permanent_uuid: "6af80a37db6048b4951d8408b5cf7552" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 58187 } } peers { permanent_uuid: "5f5927fad870472d8e084aaf82f2323b" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 52229 } attrs { replace: true } } peers { permanent_uuid: "7dc1b3f7123e460891db0377178f5634" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 38959 } attrs { promote: false } } I0703 01:41:42.489202 22390 leader_election.cc:393] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 7dc1b3f7123e460891db0377178f5634 (127.5.240.197:38959). Message: Invalid argument: T 5c9976bd26bc43c1a026a2148f7079ed P 7dc1b3f7123e460891db0377178f5634 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6af80a37db6048b4951d8408b5cf7552 in current term 2: Already voted for candidate 7dc1b3f7123e460891db0377178f5634 in this term. I0703 01:41:42.489699 22390 leader_election.cc:393] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552 [CANDIDATE]: Term 2 election: Vote denied by peer 7dc1b3f7123e460891db0377178f5634 (127.5.240.197:38959). Message: Invalid argument: T 5c9976bd26bc43c1a026a2148f7079ed P 7dc1b3f7123e460891db0377178f5634 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 6af80a37db6048b4951d8408b5cf7552 in current term 2: Already voted for candidate 7dc1b3f7123e460891db0377178f5634 in this term. I0703 01:41:42.490198 22459 raft_consensus.cc:2204] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552 [term 2 LEADER]: Leader election vote request: Denying vote to candidate 7dc1b3f7123e460891db0377178f5634 for term 2 because replica is either leader or believes a valid leader to be alive. I0703 01:41:42.491065 22882 leader_election.cc:393] T 5c9976bd26bc43c1a026a2148f7079ed P 7dc1b3f7123e460891db0377178f5634 [CANDIDATE]: Term 2 election: Vote denied by peer 6af80a37db6048b4951d8408b5cf7552 (127.5.240.193:58187). Message: Invalid argument: T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552 [term 2 LEADER]: Leader election vote request: Denying vote to candidate 7dc1b3f7123e460891db0377178f5634 for term 2 because replica is either leader or believes a valid leader to be alive. I0703 01:41:42.491624 22337 catalog_manager.cc:3798] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552 reported cstate change: term changed from 1 to 2, leader changed from 5f5927fad870472d8e084aaf82f2323b (127.5.240.196) to 6af80a37db6048b4951d8408b5cf7552 (127.5.240.193). New cstate: current_term: 2 leader_uuid: "6af80a37db6048b4951d8408b5cf7552" committed_config { opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "b7b4950f9efa4a829572d08be938050f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 53950 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6af80a37db6048b4951d8408b5cf7552" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 58187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f5927fad870472d8e084aaf82f2323b" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 52229 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7dc1b3f7123e460891db0377178f5634" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 38959 } attrs { promote: false } health_report { overall_health: UNKNOWN } } } I0703 01:41:42.980592 22948 raft_consensus.cc:1081] T 5c9976bd26bc43c1a026a2148f7079ed P 7dc1b3f7123e460891db0377178f5634 [term 2 FOLLOWER]: Refusing update from remote peer 6af80a37db6048b4951d8408b5cf7552: Log matching property violated. Preceding OpId in replica: term: 1 index: 319. Preceding OpId from leader: term: 2 index: 320. (index mismatch) I0703 01:41:42.981187 23143 consensus_queue.cc:962] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7dc1b3f7123e460891db0377178f5634" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 38959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 320, Last known committed idx: 318, Time since last communication: 0.000s I0703 01:41:43.060781 22825 raft_consensus.cc:1081] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b [term 2 FOLLOWER]: Refusing update from remote peer 6af80a37db6048b4951d8408b5cf7552: Log matching property violated. Preceding OpId in replica: term: 1 index: 319. Preceding OpId from leader: term: 2 index: 320. (index mismatch) I0703 01:41:43.061439 23143 consensus_queue.cc:962] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f5927fad870472d8e084aaf82f2323b" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 52229 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 320, Last known committed idx: 318, Time since last communication: 0.000s I0703 01:41:43.066390 23177 rpcz_store.cc:265] Call kudu.tserver.TabletServerService.Write from 127.5.240.196:40884 (ReqId={client: e49cca48a88244bfb17c4ea4457efef5, seq_no=315, attempt_no=0}) took 2124ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":245,"num_ops":1,"prepare.queue_time_us":58,"prepare.run_cpu_time_us":396,"prepare.run_wall_time_us":395,"raft.queue_time_us":65,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":195,"replication_time_us":2122914,"thread_start_us":143,"threads_started":1}]]} I0703 01:41:43.071123 22704 raft_consensus.cc:1081] T 5c9976bd26bc43c1a026a2148f7079ed P b7b4950f9efa4a829572d08be938050f [term 2 FOLLOWER]: Refusing update from remote peer 6af80a37db6048b4951d8408b5cf7552: Log matching property violated. Preceding OpId in replica: term: 1 index: 319. Preceding OpId from leader: term: 2 index: 320. (index mismatch) I0703 01:41:43.071779 23175 consensus_queue.cc:962] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7b4950f9efa4a829572d08be938050f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 53950 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 320, Last known committed idx: 318, Time since last communication: 0.000s I0703 01:41:43.094311 22335 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 5c9976bd26bc43c1a026a2148f7079ed (attempt 1) I0703 01:41:43.096168 22459 consensus_peers.cc:455] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552 -> Peer 7dc1b3f7123e460891db0377178f5634 (127.5.240.197:38959): Closing peer: 7dc1b3f7123e460891db0377178f5634 I0703 01:41:43.096251 22459 consensus_peers.cc:455] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552 -> Peer 5f5927fad870472d8e084aaf82f2323b (127.5.240.196:52229): Closing peer: 5f5927fad870472d8e084aaf82f2323b I0703 01:41:43.096309 22459 consensus_peers.cc:455] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552 -> Peer b7b4950f9efa4a829572d08be938050f (127.5.240.195:53950): Closing peer: b7b4950f9efa4a829572d08be938050f I0703 01:41:43.096505 22459 consensus_queue.cc:204] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 321, Committed index: 321, Last appended: 2.321, Last appended by leader: 319, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "b7b4950f9efa4a829572d08be938050f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 53950 } } peers { permanent_uuid: "6af80a37db6048b4951d8408b5cf7552" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 58187 } } peers { permanent_uuid: "7dc1b3f7123e460891db0377178f5634" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 38959 } attrs { promote: false } } I0703 01:41:43.098987 22704 raft_consensus.cc:1081] T 5c9976bd26bc43c1a026a2148f7079ed P b7b4950f9efa4a829572d08be938050f [term 2 FOLLOWER]: Refusing update from remote peer 6af80a37db6048b4951d8408b5cf7552: Log matching property violated. Preceding OpId in replica: term: 2 index: 321. Preceding OpId from leader: term: 2 index: 322. (index mismatch) I0703 01:41:43.099306 22948 raft_consensus.cc:1081] T 5c9976bd26bc43c1a026a2148f7079ed P 7dc1b3f7123e460891db0377178f5634 [term 2 FOLLOWER]: Refusing update from remote peer 6af80a37db6048b4951d8408b5cf7552: Log matching property violated. Preceding OpId in replica: term: 2 index: 321. Preceding OpId from leader: term: 2 index: 322. (index mismatch) I0703 01:41:43.099534 23143 consensus_queue.cc:962] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7b4950f9efa4a829572d08be938050f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 53950 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 322, Last known committed idx: 321, Time since last communication: 0.000s I0703 01:41:43.099895 23175 consensus_queue.cc:962] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7dc1b3f7123e460891db0377178f5634" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 38959 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 322, Last known committed idx: 321, Time since last communication: 0.000s I0703 01:41:43.102090 23143 raft_consensus.cc:2684] T 5c9976bd26bc43c1a026a2148f7079ed P 6af80a37db6048b4951d8408b5cf7552 [term 2 LEADER]: Committing config change with OpId 2.322: config changed from index 304 to 322, VOTER 5f5927fad870472d8e084aaf82f2323b (127.5.240.196) evicted. New config: { opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "b7b4950f9efa4a829572d08be938050f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 53950 } } peers { permanent_uuid: "6af80a37db6048b4951d8408b5cf7552" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 58187 } } peers { permanent_uuid: "7dc1b3f7123e460891db0377178f5634" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 38959 } attrs { promote: false } } } I0703 01:41:43.102882 22948 raft_consensus.cc:2684] T 5c9976bd26bc43c1a026a2148f7079ed P 7dc1b3f7123e460891db0377178f5634 [term 2 FOLLOWER]: Committing config change with OpId 2.322: config changed from index 304 to 322, VOTER 5f5927fad870472d8e084aaf82f2323b (127.5.240.196) evicted. New config: { opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "b7b4950f9efa4a829572d08be938050f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 53950 } } peers { permanent_uuid: "6af80a37db6048b4951d8408b5cf7552" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 58187 } } peers { permanent_uuid: "7dc1b3f7123e460891db0377178f5634" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 38959 } attrs { promote: false } } } I0703 01:41:43.103289 22704 raft_consensus.cc:2684] T 5c9976bd26bc43c1a026a2148f7079ed P b7b4950f9efa4a829572d08be938050f [term 2 FOLLOWER]: Committing config change with OpId 2.322: config changed from index 304 to 322, VOTER 5f5927fad870472d8e084aaf82f2323b (127.5.240.196) evicted. New config: { opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "b7b4950f9efa4a829572d08be938050f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 53950 } } peers { permanent_uuid: "6af80a37db6048b4951d8408b5cf7552" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 58187 } } peers { permanent_uuid: "7dc1b3f7123e460891db0377178f5634" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 38959 } attrs { promote: false } } } I0703 01:41:43.105434 22322 catalog_manager.cc:3352] ChangeConfig:REMOVE_PEER RPC for tablet 5c9976bd26bc43c1a026a2148f7079ed with cas_config_opid_index 304: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0703 01:41:43.106485 22335 catalog_manager.cc:3798] T 5c9976bd26bc43c1a026a2148f7079ed P 7dc1b3f7123e460891db0377178f5634 reported cstate change: config changed from index 304 to 322, VOTER 5f5927fad870472d8e084aaf82f2323b (127.5.240.196) evicted. New cstate: current_term: 2 leader_uuid: "6af80a37db6048b4951d8408b5cf7552" committed_config { opid_index: 322 OBSOLETE_local: false peers { permanent_uuid: "b7b4950f9efa4a829572d08be938050f" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 53950 } } peers { permanent_uuid: "6af80a37db6048b4951d8408b5cf7552" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 58187 } } peers { permanent_uuid: "7dc1b3f7123e460891db0377178f5634" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 38959 } attrs { promote: false } } } I0703 01:41:43.112376 22335 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 5c9976bd26bc43c1a026a2148f7079ed on 5f5927fad870472d8e084aaf82f2323b (127.5.240.196:52229) (TS 5f5927fad870472d8e084aaf82f2323b not found in new config with opid_index 322) I0703 01:41:43.113348 22806 tablet_service.cc:800] Processing DeleteTablet for tablet 5c9976bd26bc43c1a026a2148f7079ed with delete_type TABLET_DATA_TOMBSTONED (TS 5f5927fad870472d8e084aaf82f2323b not found in new config with opid_index 322) from {username='jenkins'} at 127.5.240.196:40877 I0703 01:41:43.113948 23193 tablet_replica.cc:261] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b: stopping tablet replica I0703 01:41:43.114133 23193 maintenance_manager.cc:237] P 5f5927fad870472d8e084aaf82f2323b: Unregistered op CompactRowSetsOp(5c9976bd26bc43c1a026a2148f7079ed) I0703 01:41:43.114213 23193 maintenance_manager.cc:237] P 5f5927fad870472d8e084aaf82f2323b: Unregistered op MinorDeltaCompactionOp(5c9976bd26bc43c1a026a2148f7079ed) I0703 01:41:43.114260 23193 maintenance_manager.cc:237] P 5f5927fad870472d8e084aaf82f2323b: Unregistered op MajorDeltaCompactionOp(5c9976bd26bc43c1a026a2148f7079ed) I0703 01:41:43.114310 23193 maintenance_manager.cc:237] P 5f5927fad870472d8e084aaf82f2323b: Unregistered op UndoDeltaBlockGCOp(5c9976bd26bc43c1a026a2148f7079ed) I0703 01:41:43.114382 23193 maintenance_manager.cc:237] P 5f5927fad870472d8e084aaf82f2323b: Unregistered op FlushMRSOp(5c9976bd26bc43c1a026a2148f7079ed) I0703 01:41:43.114430 23193 maintenance_manager.cc:237] P 5f5927fad870472d8e084aaf82f2323b: Unregistered op FlushDeltaMemStoresOp(5c9976bd26bc43c1a026a2148f7079ed) I0703 01:41:43.114471 23193 maintenance_manager.cc:237] P 5f5927fad870472d8e084aaf82f2323b: Unregistered op LogGCOp(5c9976bd26bc43c1a026a2148f7079ed) I0703 01:41:43.114596 23193 raft_consensus.cc:2040] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b [term 2 FOLLOWER]: Raft consensus shutting down. I0703 01:41:43.114850 23193 raft_consensus.cc:2067] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b [term 2 FOLLOWER]: Raft consensus is shut down! I0703 01:41:43.116622 23193 ts_tablet_manager.cc:1369] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:41:43.122472 23193 ts_tablet_manager.cc:1382] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.321 I0703 01:41:43.122627 23193 log.cc:986] T 5c9976bd26bc43c1a026a2148f7079ed P 5f5927fad870472d8e084aaf82f2323b: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_5.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal/wals/5c9976bd26bc43c1a026a2148f7079ed I0703 01:41:43.123443 22319 catalog_manager.cc:3103] TS 5f5927fad870472d8e084aaf82f2323b (127.5.240.196:52229): tablet 5c9976bd26bc43c1a026a2148f7079ed (table TestTable [id=0541c3d083f84ed7997411acb96335cb]) successfully deleted I0703 01:41:43.628720 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 22372 I0703 01:41:43.630612 22499 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:41:43.700248 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 22372 W0703 01:41:43.711902 22320 connection.cc:512] server connection from 127.5.240.193:38479 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:38479): Connection reset by peer (error 104) W0703 01:41:43.712378 22759 connection.cc:512] server connection from 127.5.240.193:44035 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:44035): Connection reset by peer (error 104) W0703 01:41:43.712347 22636 connection.cc:512] server connection from 127.5.240.193:54951 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:54951): Connection reset by peer (error 104) W0703 01:41:43.712378 22760 connection.cc:512] client connection to 127.5.240.193:58187 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:58187): Connection reset by peer (error 104) W0703 01:41:43.712782 22882 connection.cc:512] client connection to 127.5.240.193:58187 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:58187): Connection reset by peer (error 104) W0703 01:41:43.713165 22638 connection.cc:512] client connection to 127.5.240.193:58187 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:58187): Connection reset by peer (error 104) W0703 01:41:43.713210 22882 connection.cc:512] server connection from 127.5.240.193:35179 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:35179): Connection reset by peer (error 104) I0703 01:41:43.713255 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 22628 I0703 01:41:43.713986 22744 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:41:43.776083 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 22628 W0703 01:41:43.787299 22758 connection.cc:512] client connection to 127.5.240.195:53950 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.195:53950): Connection reset by peer (error 104) W0703 01:41:43.787324 22319 connection.cc:512] server connection from 127.5.240.195:50812 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.195:50812): Connection reset by peer (error 104) W0703 01:41:43.787333 22880 connection.cc:512] client connection to 127.5.240.195:53950 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.195:53950): Connection reset by peer (error 104) W0703 01:41:43.787695 22757 connection.cc:512] server connection from 127.5.240.195:59933 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.195:59933): Connection reset by peer (error 104) W0703 01:41:43.787735 22882 connection.cc:512] server connection from 127.5.240.195:57228 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.195:57228): Connection reset by peer (error 104) I0703 01:41:43.787796 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 22750 I0703 01:41:43.788507 22866 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:41:43.858229 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 22750 W0703 01:41:43.870085 22320 connection.cc:512] server connection from 127.5.240.196:41695 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.196:41695): Connection reset by peer (error 104) W0703 01:41:43.870091 22879 connection.cc:512] client connection to 127.5.240.196:52229 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.196:52229): Connection reset by peer (error 104) W0703 01:41:43.870486 22880 connection.cc:512] server connection from 127.5.240.196:33420 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.196:33420): Connection reset by peer (error 104) I0703 01:41:43.870568 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 22872 I0703 01:41:43.871302 22988 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:41:43.925431 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 22872 W0703 01:41:43.935045 22321 connection.cc:512] server connection from 127.5.240.197:52525 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.197:52525): Connection reset by peer (error 104) I0703 01:41:43.935210 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master pid 22312 I0703 01:41:43.936013 22368 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:41:44.000267 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master with pid 22312 [ OK ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/5 (7345 ms) [----------] 6 tests from EnableKudu1097AndDownTS/MoveTabletParamTest (62294 ms total) [----------] 2 tests from RebalanceStartCriteriaTest [ RUN ] RebalanceStartCriteriaTest.TabletServerIsDown/0 I0703 01:41:44.023363 6083 test_util.cc:208] Using random seed: 1218813237 I0703 01:41:44.025077 6083 ts_itest-base.cc:113] Starting cluster with: I0703 01:41:44.025107 6083 ts_itest-base.cc:114] -------------- I0703 01:41:44.025149 6083 ts_itest-base.cc:115] 5 tablet servers I0703 01:41:44.025177 6083 ts_itest-base.cc:116] 3 replicas per TS I0703 01:41:44.025223 6083 ts_itest-base.cc:117] -------------- I0703 01:41:44.026347 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:45494 --webserver_interface=127.5.240.254 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:41:44.210180 23212 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:41:44.210410 23212 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:41:44.212673 23212 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0703 01:41:44.212738 23212 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0703 01:41:44.212786 23212 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:41:44.212827 23212 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0703 01:41:44.212882 23212 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0703 01:41:44.220597 23212 master_main.cc:73] Master server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:45494 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.5.240.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:41:44.226233 23212 master_main.cc:80] Initializing master server... I0703 01:41:44.226941 23212 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 205708us I0703 01:41:44.227982 23212 fs_manager.cc:261] Metadata directory not provided I0703 01:41:44.228051 23212 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:41:44.228287 23212 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:41:44.228356 23212 server_base.cc:433] Attempting to create new FS layout instead I0703 01:41:44.232517 23212 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/instance: uuid: "7402ef5507614472b2bf8284d93adaf8" format_stamp: "Formatted at 2018-07-03 08:41:44 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:44.233057 23212 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "7402ef5507614472b2bf8284d93adaf8" format_stamp: "Formatted at 2018-07-03 08:41:44 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:44.241124 23212 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.004s sys 0.003s I0703 01:41:44.242326 23212 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:41:44.243114 23212 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:41:44.246282 23212 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0703 01:41:44.246333 23212 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal uuid: "7402ef5507614472b2bf8284d93adaf8" format_stamp: "Formatted at 2018-07-03 08:41:44 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:44.246448 23212 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:41:44.281059 23212 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:41:44.282019 23212 master_main.cc:83] Starting Master server... I0703 01:41:44.319150 23212 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.254:45494 I0703 01:41:44.319628 23212 webserver.cc:175] Starting webserver on 127.5.240.254:0 I0703 01:41:44.319671 23212 webserver.cc:186] Document root disabled I0703 01:41:44.320420 23212 webserver.cc:313] Webserver started. Bound to: http://127.5.240.254:42901/ I0703 01:41:44.321022 23212 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb I0703 01:41:44.324728 23271 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:41:44.330591 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master as pid 23212 I0703 01:41:44.331429 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.5.240.193:0 --local_ip_for_outbound_sockets=127.5.240.193 --webserver_interface=127.5.240.193 --webserver_port=0 --tserver_master_addrs=127.5.240.254:45494 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0703 01:41:44.337919 23271 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 7402ef5507614472b2bf8284d93adaf8: Bootstrap starting. I0703 01:41:44.340425 23271 tablet_bootstrap.cc:578] T 00000000000000000000000000000000 P 7402ef5507614472b2bf8284d93adaf8: No blocks or log segments found. Creating new log. I0703 01:41:44.341248 23271 log.cc:525] T 00000000000000000000000000000000 P 7402ef5507614472b2bf8284d93adaf8: Log is configured to *not* fsync() on all Append() calls I0703 01:41:44.343482 23271 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 7402ef5507614472b2bf8284d93adaf8: No bootstrap required, opened a new log I0703 01:41:44.349630 23271 raft_consensus.cc:325] T 00000000000000000000000000000000 P 7402ef5507614472b2bf8284d93adaf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7402ef5507614472b2bf8284d93adaf8" member_type: VOTER } I0703 01:41:44.349826 23271 raft_consensus.cc:351] T 00000000000000000000000000000000 P 7402ef5507614472b2bf8284d93adaf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:41:44.349885 23271 raft_consensus.cc:628] T 00000000000000000000000000000000 P 7402ef5507614472b2bf8284d93adaf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7402ef5507614472b2bf8284d93adaf8, State: Initialized, Role: FOLLOWER I0703 01:41:44.350374 23271 consensus_queue.cc:227] T 00000000000000000000000000000000 P 7402ef5507614472b2bf8284d93adaf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7402ef5507614472b2bf8284d93adaf8" member_type: VOTER } I0703 01:41:44.350539 23271 raft_consensus.cc:365] T 00000000000000000000000000000000 P 7402ef5507614472b2bf8284d93adaf8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:41:44.350621 23271 raft_consensus.cc:457] T 00000000000000000000000000000000 P 7402ef5507614472b2bf8284d93adaf8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:41:44.350699 23271 raft_consensus.cc:2730] T 00000000000000000000000000000000 P 7402ef5507614472b2bf8284d93adaf8: Snoozing failure detection for 1.621s (starting election) I0703 01:41:44.350780 23271 raft_consensus.cc:2782] T 00000000000000000000000000000000 P 7402ef5507614472b2bf8284d93adaf8 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:41:44.352113 23271 raft_consensus.cc:479] T 00000000000000000000000000000000 P 7402ef5507614472b2bf8284d93adaf8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7402ef5507614472b2bf8284d93adaf8" member_type: VOTER } I0703 01:41:44.352470 23271 leader_election.cc:270] T 00000000000000000000000000000000 P 7402ef5507614472b2bf8284d93adaf8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:41:44.354019 23274 raft_consensus.cc:2730] T 00000000000000000000000000000000 P 7402ef5507614472b2bf8284d93adaf8: Snoozing failure detection for 1.538s (election complete) I0703 01:41:44.354578 23274 raft_consensus.cc:2537] T 00000000000000000000000000000000 P 7402ef5507614472b2bf8284d93adaf8 [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:41:44.355906 23274 raft_consensus.cc:593] T 00000000000000000000000000000000 P 7402ef5507614472b2bf8284d93adaf8 [term 1 LEADER]: Becoming Leader. State: Replica: 7402ef5507614472b2bf8284d93adaf8, State: Running, Role: LEADER I0703 01:41:44.356315 23274 consensus_queue.cc:204] T 00000000000000000000000000000000 P 7402ef5507614472b2bf8284d93adaf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7402ef5507614472b2bf8284d93adaf8" member_type: VOTER } I0703 01:41:44.359127 23271 sys_catalog.cc:337] T 00000000000000000000000000000000 P 7402ef5507614472b2bf8284d93adaf8 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "7402ef5507614472b2bf8284d93adaf8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7402ef5507614472b2bf8284d93adaf8" member_type: VOTER } } I0703 01:41:44.359562 23271 sys_catalog.cc:340] T 00000000000000000000000000000000 P 7402ef5507614472b2bf8284d93adaf8 [sys.catalog]: This master's current role is: LEADER I0703 01:41:44.359041 23275 sys_catalog.cc:337] T 00000000000000000000000000000000 P 7402ef5507614472b2bf8284d93adaf8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7402ef5507614472b2bf8284d93adaf8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7402ef5507614472b2bf8284d93adaf8" member_type: VOTER } } I0703 01:41:44.359127 23276 sys_catalog.cc:337] T 00000000000000000000000000000000 P 7402ef5507614472b2bf8284d93adaf8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7402ef5507614472b2bf8284d93adaf8. Latest consensus state: current_term: 1 leader_uuid: "7402ef5507614472b2bf8284d93adaf8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7402ef5507614472b2bf8284d93adaf8" member_type: VOTER } } I0703 01:41:44.360327 23276 sys_catalog.cc:340] T 00000000000000000000000000000000 P 7402ef5507614472b2bf8284d93adaf8 [sys.catalog]: This master's current role is: LEADER I0703 01:41:44.360328 23275 sys_catalog.cc:340] T 00000000000000000000000000000000 P 7402ef5507614472b2bf8284d93adaf8 [sys.catalog]: This master's current role is: LEADER I0703 01:41:44.362491 23271 sys_catalog.cc:424] T 00000000000000000000000000000000 P 7402ef5507614472b2bf8284d93adaf8 [sys.catalog]: configured and running, proceeding with master startup. I0703 01:41:44.362861 23278 catalog_manager.cc:1006] Loading table and tablet metadata into memory... I0703 01:41:44.363690 23212 master_main.cc:86] Master server successfully started. I0703 01:41:44.369204 23278 catalog_manager.cc:1017] Initializing Kudu internal certificate authority... I0703 01:41:44.395097 23278 catalog_manager.cc:904] Generated new certificate authority record I0703 01:41:44.397110 23278 catalog_manager.cc:1026] Loading token signing keys... I0703 01:41:44.405045 23278 catalog_manager.cc:4102] T 00000000000000000000000000000000 P 7402ef5507614472b2bf8284d93adaf8: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:41:44.506850 23272 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:41:44.507088 23272 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:41:44.509075 23272 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0703 01:41:44.509140 23272 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:41:44.509255 23272 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.193 I0703 01:41:44.518815 23272 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.5.240.193 --webserver_port=0 --tserver_master_addrs=127.5.240.254:45494 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.193 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:41:44.525429 23272 tablet_server_main.cc:85] Initializing tablet server... I0703 01:41:44.526959 23272 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 205708us I0703 01:41:44.527513 23272 fs_manager.cc:261] Metadata directory not provided I0703 01:41:44.527601 23272 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:41:44.527842 23272 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:41:44.527910 23272 server_base.cc:433] Attempting to create new FS layout instead I0703 01:41:44.531980 23272 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "483d3c608319410d9718f61955b1f929" format_stamp: "Formatted at 2018-07-03 08:41:44 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:44.532440 23272 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "483d3c608319410d9718f61955b1f929" format_stamp: "Formatted at 2018-07-03 08:41:44 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:44.540199 23272 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.002s sys 0.003s I0703 01:41:44.541399 23272 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:41:44.541806 23272 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:41:44.545778 23272 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s I0703 01:41:44.545827 23272 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal uuid: "483d3c608319410d9718f61955b1f929" format_stamp: "Formatted at 2018-07-03 08:41:44 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:44.545953 23272 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:41:44.596029 23272 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:41:44.597990 23272 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:41:44.598311 23272 tablet_server_main.cc:90] Starting tablet server... I0703 01:41:44.694134 23272 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.193:48535 I0703 01:41:44.694622 23272 webserver.cc:175] Starting webserver on 127.5.240.193:0 I0703 01:41:44.694671 23272 webserver.cc:186] Document root disabled I0703 01:41:44.696007 23272 webserver.cc:313] Webserver started. Bound to: http://127.5.240.193:44172/ I0703 01:41:44.696568 23272 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb I0703 01:41:44.698599 23272 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:41:44.702211 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 23272 I0703 01:41:44.702889 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.5.240.194:0 --local_ip_for_outbound_sockets=127.5.240.194 --webserver_interface=127.5.240.194 --webserver_port=0 --tserver_master_addrs=127.5.240.254:45494 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0703 01:41:44.713946 23401 heartbeater.cc:346] Connected to a master server at 127.5.240.254:45494 I0703 01:41:44.714110 23401 heartbeater.cc:426] Registering TS with master... I0703 01:41:44.714967 23401 heartbeater.cc:475] Master 127.5.240.254:45494 requested a full tablet report, sending... I0703 01:41:44.716464 23238 ts_manager.cc:98] Registered new tserver with Master: 483d3c608319410d9718f61955b1f929 (127.5.240.193:48535) I0703 01:41:44.719008 23238 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.193:36549 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:41:44.868602 23405 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:41:44.868825 23405 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:41:44.870798 23405 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0703 01:41:44.870846 23405 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:41:44.870930 23405 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.194 I0703 01:41:44.877851 23405 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.194:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.5.240.194 --webserver_port=0 --tserver_master_addrs=127.5.240.254:45494 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.194 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:41:44.884341 23405 tablet_server_main.cc:85] Initializing tablet server... I0703 01:41:44.885866 23405 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 205708us I0703 01:41:44.886421 23405 fs_manager.cc:261] Metadata directory not provided I0703 01:41:44.886481 23405 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:41:44.886720 23405 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:41:44.886783 23405 server_base.cc:433] Attempting to create new FS layout instead I0703 01:41:44.890825 23405 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "b228ec228f89487b8a2aba0b6b755217" format_stamp: "Formatted at 2018-07-03 08:41:44 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:44.891330 23405 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "b228ec228f89487b8a2aba0b6b755217" format_stamp: "Formatted at 2018-07-03 08:41:44 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:44.898684 23405 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.004s sys 0.003s I0703 01:41:44.899863 23405 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:41:44.900228 23405 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:41:44.903812 23405 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0703 01:41:44.903857 23405 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal uuid: "b228ec228f89487b8a2aba0b6b755217" format_stamp: "Formatted at 2018-07-03 08:41:44 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:44.903975 23405 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:41:44.946017 23405 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:41:44.948248 23405 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:41:44.948590 23405 tablet_server_main.cc:90] Starting tablet server... I0703 01:41:45.032481 23405 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.194:53674 I0703 01:41:45.032924 23405 webserver.cc:175] Starting webserver on 127.5.240.194:0 I0703 01:41:45.032975 23405 webserver.cc:186] Document root disabled I0703 01:41:45.034175 23405 webserver.cc:313] Webserver started. Bound to: http://127.5.240.194:56321/ I0703 01:41:45.034766 23405 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb I0703 01:41:45.036602 23405 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:41:45.041333 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 23405 I0703 01:41:45.042197 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.5.240.195:0 --local_ip_for_outbound_sockets=127.5.240.195 --webserver_interface=127.5.240.195 --webserver_port=0 --tserver_master_addrs=127.5.240.254:45494 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0703 01:41:45.051168 23524 heartbeater.cc:346] Connected to a master server at 127.5.240.254:45494 I0703 01:41:45.051318 23524 heartbeater.cc:426] Registering TS with master... I0703 01:41:45.052083 23524 heartbeater.cc:475] Master 127.5.240.254:45494 requested a full tablet report, sending... I0703 01:41:45.053098 23238 ts_manager.cc:98] Registered new tserver with Master: b228ec228f89487b8a2aba0b6b755217 (127.5.240.194:53674) I0703 01:41:45.054049 23238 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.194:48451 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:41:45.208609 23527 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:41:45.208842 23527 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:41:45.210917 23527 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0703 01:41:45.210978 23527 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:41:45.211081 23527 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.195 I0703 01:41:45.217795 23527 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.195:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.5.240.195 --webserver_port=0 --tserver_master_addrs=127.5.240.254:45494 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.195 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:41:45.224460 23527 tablet_server_main.cc:85] Initializing tablet server... I0703 01:41:45.226022 23527 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 206208us I0703 01:41:45.226619 23527 fs_manager.cc:261] Metadata directory not provided I0703 01:41:45.226688 23527 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:41:45.226922 23527 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:41:45.226985 23527 server_base.cc:433] Attempting to create new FS layout instead I0703 01:41:45.231039 23527 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "890887632c52418080b79444147136fe" format_stamp: "Formatted at 2018-07-03 08:41:45 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:45.231595 23527 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "890887632c52418080b79444147136fe" format_stamp: "Formatted at 2018-07-03 08:41:45 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:45.239035 23527 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.003s sys 0.003s I0703 01:41:45.240219 23527 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:41:45.240633 23527 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:41:45.244221 23527 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s I0703 01:41:45.244272 23527 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal uuid: "890887632c52418080b79444147136fe" format_stamp: "Formatted at 2018-07-03 08:41:45 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:45.244396 23527 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:41:45.274320 23527 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:41:45.276139 23527 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:41:45.276445 23527 tablet_server_main.cc:90] Starting tablet server... I0703 01:41:45.361824 23527 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.195:54776 I0703 01:41:45.362248 23527 webserver.cc:175] Starting webserver on 127.5.240.195:0 I0703 01:41:45.362293 23527 webserver.cc:186] Document root disabled I0703 01:41:45.363462 23527 webserver.cc:313] Webserver started. Bound to: http://127.5.240.195:40283/ I0703 01:41:45.364085 23527 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb I0703 01:41:45.366021 23527 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:41:45.374030 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 23527 I0703 01:41:45.374673 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.5.240.196:0 --local_ip_for_outbound_sockets=127.5.240.196 --webserver_interface=127.5.240.196 --webserver_port=0 --tserver_master_addrs=127.5.240.254:45494 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0703 01:41:45.377972 23646 heartbeater.cc:346] Connected to a master server at 127.5.240.254:45494 I0703 01:41:45.378159 23646 heartbeater.cc:426] Registering TS with master... I0703 01:41:45.379029 23646 heartbeater.cc:475] Master 127.5.240.254:45494 requested a full tablet report, sending... I0703 01:41:45.380057 23238 ts_manager.cc:98] Registered new tserver with Master: 890887632c52418080b79444147136fe (127.5.240.195:54776) I0703 01:41:45.380916 23238 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.195:38057 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:41:45.537268 23649 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:41:45.537506 23649 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:41:45.539535 23649 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0703 01:41:45.539618 23649 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:41:45.539724 23649 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.196 I0703 01:41:45.546200 23649 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.196:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb --webserver_interface=127.5.240.196 --webserver_port=0 --tserver_master_addrs=127.5.240.254:45494 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.196 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:41:45.552772 23649 tablet_server_main.cc:85] Initializing tablet server... I0703 01:41:45.554348 23649 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 206208us I0703 01:41:45.554955 23649 fs_manager.cc:261] Metadata directory not provided I0703 01:41:45.555027 23649 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:41:45.555670 23649 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:41:45.555739 23649 server_base.cc:433] Attempting to create new FS layout instead I0703 01:41:45.559780 23649 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/instance: uuid: "1d8ef998f2044df3a15ce2cee08280db" format_stamp: "Formatted at 2018-07-03 08:41:45 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:45.560317 23649 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal/instance: uuid: "1d8ef998f2044df3a15ce2cee08280db" format_stamp: "Formatted at 2018-07-03 08:41:45 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:45.568073 23649 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s I0703 01:41:45.569257 23649 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:41:45.569664 23649 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:41:45.572746 23649 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0703 01:41:45.572796 23649 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal uuid: "1d8ef998f2044df3a15ce2cee08280db" format_stamp: "Formatted at 2018-07-03 08:41:45 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:45.572923 23649 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:41:45.660948 23649 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:41:45.663259 23649 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:41:45.663588 23649 tablet_server_main.cc:90] Starting tablet server... I0703 01:41:45.721598 23401 heartbeater.cc:467] Master 127.5.240.254:45494 was elected leader, sending a full tablet report... I0703 01:41:45.747973 23649 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.196:33187 I0703 01:41:45.748349 23649 webserver.cc:175] Starting webserver on 127.5.240.196:0 I0703 01:41:45.748387 23649 webserver.cc:186] Document root disabled I0703 01:41:45.749558 23649 webserver.cc:313] Webserver started. Bound to: http://127.5.240.196:45079/ I0703 01:41:45.750074 23649 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb I0703 01:41:45.752513 23649 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:41:45.753451 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 23649 I0703 01:41:45.754335 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data --block_manager=log --rpc_bind_addresses=127.5.240.197:0 --local_ip_for_outbound_sockets=127.5.240.197 --webserver_interface=127.5.240.197 --webserver_port=0 --tserver_master_addrs=127.5.240.254:45494 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0703 01:41:45.763662 23768 heartbeater.cc:346] Connected to a master server at 127.5.240.254:45494 I0703 01:41:45.763821 23768 heartbeater.cc:426] Registering TS with master... I0703 01:41:45.764683 23768 heartbeater.cc:475] Master 127.5.240.254:45494 requested a full tablet report, sending... I0703 01:41:45.765650 23238 ts_manager.cc:98] Registered new tserver with Master: 1d8ef998f2044df3a15ce2cee08280db (127.5.240.196:33187) I0703 01:41:45.766476 23238 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.196:54371 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:41:45.916734 23771 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:41:45.916946 23771 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:41:45.918921 23771 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0703 01:41:45.918968 23771 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:41:45.919052 23771 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.197 I0703 01:41:45.926533 23771 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.197:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb --webserver_interface=127.5.240.197 --webserver_port=0 --tserver_master_addrs=127.5.240.254:45494 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.197 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:41:45.932973 23771 tablet_server_main.cc:85] Initializing tablet server... I0703 01:41:45.934460 23771 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 206208us I0703 01:41:45.935019 23771 fs_manager.cc:261] Metadata directory not provided I0703 01:41:45.935082 23771 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:41:45.935304 23771 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:41:45.935365 23771 server_base.cc:433] Attempting to create new FS layout instead I0703 01:41:45.939353 23771 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/instance: uuid: "ac322d270bf445adaa0038eb0dc24410" format_stamp: "Formatted at 2018-07-03 08:41:45 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:45.939869 23771 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal/instance: uuid: "ac322d270bf445adaa0038eb0dc24410" format_stamp: "Formatted at 2018-07-03 08:41:45 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:45.947610 23771 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.004s sys 0.002s I0703 01:41:45.948820 23771 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:41:45.949203 23771 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:41:45.952391 23771 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0703 01:41:45.952437 23771 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal uuid: "ac322d270bf445adaa0038eb0dc24410" format_stamp: "Formatted at 2018-07-03 08:41:45 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:45.952579 23771 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:41:45.982833 23771 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:41:45.984616 23771 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:41:45.984900 23771 tablet_server_main.cc:90] Starting tablet server... I0703 01:41:46.056006 23524 heartbeater.cc:467] Master 127.5.240.254:45494 was elected leader, sending a full tablet report... I0703 01:41:46.069782 23771 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.197:47608 I0703 01:41:46.070214 23771 webserver.cc:175] Starting webserver on 127.5.240.197:0 I0703 01:41:46.070258 23771 webserver.cc:186] Document root disabled I0703 01:41:46.071450 23771 webserver.cc:313] Webserver started. Bound to: http://127.5.240.197:38319/ I0703 01:41:46.071996 23771 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb I0703 01:41:46.075268 23771 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:41:46.081570 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 23771 I0703 01:41:46.086916 23890 heartbeater.cc:346] Connected to a master server at 127.5.240.254:45494 I0703 01:41:46.087075 23890 heartbeater.cc:426] Registering TS with master... I0703 01:41:46.087831 23890 heartbeater.cc:475] Master 127.5.240.254:45494 requested a full tablet report, sending... I0703 01:41:46.088899 23238 ts_manager.cc:98] Registered new tserver with Master: ac322d270bf445adaa0038eb0dc24410 (127.5.240.197:47608) I0703 01:41:46.089596 6083 external_mini_cluster.cc:490] 5 TS(s) registered with all masters I0703 01:41:46.090209 23238 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.197:46730 I0703 01:41:46.099478 23238 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.5.240.254:58958: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0703 01:41:46.124295 23338 tablet_service.cc:759] Processing CreateTablet for tablet d57e314b004b420f88b334f2cced89a1 (table=TestTable [id=aefec0d53c2e4db399189724b96d9dc2]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:41:46.124295 23583 tablet_service.cc:759] Processing CreateTablet for tablet d57e314b004b420f88b334f2cced89a1 (table=TestTable [id=aefec0d53c2e4db399189724b96d9dc2]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:41:46.125171 23338 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet d57e314b004b420f88b334f2cced89a1. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:41:46.125174 23583 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet d57e314b004b420f88b334f2cced89a1. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:41:46.125290 23705 tablet_service.cc:759] Processing CreateTablet for tablet d57e314b004b420f88b334f2cced89a1 (table=TestTable [id=aefec0d53c2e4db399189724b96d9dc2]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:41:46.125876 23705 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet d57e314b004b420f88b334f2cced89a1. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:41:46.136500 23705 ts_tablet_manager.cc:1173] T d57e314b004b420f88b334f2cced89a1 P 1d8ef998f2044df3a15ce2cee08280db: Registered tablet (data state: TABLET_DATA_READY) I0703 01:41:46.137929 23910 ts_tablet_manager.cc:1028] T d57e314b004b420f88b334f2cced89a1 P 1d8ef998f2044df3a15ce2cee08280db: Bootstrapping tablet I0703 01:41:46.138500 23910 tablet_bootstrap.cc:427] T d57e314b004b420f88b334f2cced89a1 P 1d8ef998f2044df3a15ce2cee08280db: Bootstrap starting. I0703 01:41:46.139581 23338 ts_tablet_manager.cc:1173] T d57e314b004b420f88b334f2cced89a1 P 483d3c608319410d9718f61955b1f929: Registered tablet (data state: TABLET_DATA_READY) I0703 01:41:46.139562 23583 ts_tablet_manager.cc:1173] T d57e314b004b420f88b334f2cced89a1 P 890887632c52418080b79444147136fe: Registered tablet (data state: TABLET_DATA_READY) I0703 01:41:46.140758 23911 ts_tablet_manager.cc:1028] T d57e314b004b420f88b334f2cced89a1 P 890887632c52418080b79444147136fe: Bootstrapping tablet I0703 01:41:46.141242 23911 tablet_bootstrap.cc:427] T d57e314b004b420f88b334f2cced89a1 P 890887632c52418080b79444147136fe: Bootstrap starting. I0703 01:41:46.142930 23910 tablet_bootstrap.cc:578] T d57e314b004b420f88b334f2cced89a1 P 1d8ef998f2044df3a15ce2cee08280db: No blocks or log segments found. Creating new log. I0703 01:41:46.144009 23910 log.cc:525] T d57e314b004b420f88b334f2cced89a1 P 1d8ef998f2044df3a15ce2cee08280db: Log is configured to *not* fsync() on all Append() calls I0703 01:41:46.144316 23911 tablet_bootstrap.cc:578] T d57e314b004b420f88b334f2cced89a1 P 890887632c52418080b79444147136fe: No blocks or log segments found. Creating new log. I0703 01:41:46.144596 23912 ts_tablet_manager.cc:1028] T d57e314b004b420f88b334f2cced89a1 P 483d3c608319410d9718f61955b1f929: Bootstrapping tablet I0703 01:41:46.144809 23912 tablet_bootstrap.cc:427] T d57e314b004b420f88b334f2cced89a1 P 483d3c608319410d9718f61955b1f929: Bootstrap starting. I0703 01:41:46.145191 23911 log.cc:525] T d57e314b004b420f88b334f2cced89a1 P 890887632c52418080b79444147136fe: Log is configured to *not* fsync() on all Append() calls I0703 01:41:46.147523 23912 tablet_bootstrap.cc:578] T d57e314b004b420f88b334f2cced89a1 P 483d3c608319410d9718f61955b1f929: No blocks or log segments found. Creating new log. I0703 01:41:46.148146 23911 tablet_bootstrap.cc:427] T d57e314b004b420f88b334f2cced89a1 P 890887632c52418080b79444147136fe: No bootstrap required, opened a new log I0703 01:41:46.148344 23911 ts_tablet_manager.cc:1045] T d57e314b004b420f88b334f2cced89a1 P 890887632c52418080b79444147136fe: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s I0703 01:41:46.148504 23912 log.cc:525] T d57e314b004b420f88b334f2cced89a1 P 483d3c608319410d9718f61955b1f929: Log is configured to *not* fsync() on all Append() calls I0703 01:41:46.150099 23912 tablet_bootstrap.cc:427] T d57e314b004b420f88b334f2cced89a1 P 483d3c608319410d9718f61955b1f929: No bootstrap required, opened a new log I0703 01:41:46.150254 23912 ts_tablet_manager.cc:1045] T d57e314b004b420f88b334f2cced89a1 P 483d3c608319410d9718f61955b1f929: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s I0703 01:41:46.150761 23910 tablet_bootstrap.cc:427] T d57e314b004b420f88b334f2cced89a1 P 1d8ef998f2044df3a15ce2cee08280db: No bootstrap required, opened a new log I0703 01:41:46.150954 23910 ts_tablet_manager.cc:1045] T d57e314b004b420f88b334f2cced89a1 P 1d8ef998f2044df3a15ce2cee08280db: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.005s I0703 01:41:46.156265 23912 raft_consensus.cc:325] T d57e314b004b420f88b334f2cced89a1 P 483d3c608319410d9718f61955b1f929 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "483d3c608319410d9718f61955b1f929" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 48535 } } peers { permanent_uuid: "890887632c52418080b79444147136fe" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 54776 } } peers { permanent_uuid: "1d8ef998f2044df3a15ce2cee08280db" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 33187 } } I0703 01:41:46.156574 23912 raft_consensus.cc:351] T d57e314b004b420f88b334f2cced89a1 P 483d3c608319410d9718f61955b1f929 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:41:46.156653 23912 raft_consensus.cc:628] T d57e314b004b420f88b334f2cced89a1 P 483d3c608319410d9718f61955b1f929 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 483d3c608319410d9718f61955b1f929, State: Initialized, Role: FOLLOWER I0703 01:41:46.157059 23912 consensus_queue.cc:227] T d57e314b004b420f88b334f2cced89a1 P 483d3c608319410d9718f61955b1f929 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "483d3c608319410d9718f61955b1f929" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 48535 } } peers { permanent_uuid: "890887632c52418080b79444147136fe" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 54776 } } peers { permanent_uuid: "1d8ef998f2044df3a15ce2cee08280db" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 33187 } } I0703 01:41:46.157438 23911 raft_consensus.cc:325] T d57e314b004b420f88b334f2cced89a1 P 890887632c52418080b79444147136fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "483d3c608319410d9718f61955b1f929" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 48535 } } peers { permanent_uuid: "890887632c52418080b79444147136fe" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 54776 } } peers { permanent_uuid: "1d8ef998f2044df3a15ce2cee08280db" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 33187 } } I0703 01:41:46.157866 23911 raft_consensus.cc:351] T d57e314b004b420f88b334f2cced89a1 P 890887632c52418080b79444147136fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:41:46.157956 23911 raft_consensus.cc:628] T d57e314b004b420f88b334f2cced89a1 P 890887632c52418080b79444147136fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 890887632c52418080b79444147136fe, State: Initialized, Role: FOLLOWER I0703 01:41:46.158896 23912 ts_tablet_manager.cc:1073] T d57e314b004b420f88b334f2cced89a1 P 483d3c608319410d9718f61955b1f929: Time spent starting tablet: real 0.009s user 0.005s sys 0.004s I0703 01:41:46.159046 23911 consensus_queue.cc:227] T d57e314b004b420f88b334f2cced89a1 P 890887632c52418080b79444147136fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "483d3c608319410d9718f61955b1f929" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 48535 } } peers { permanent_uuid: "890887632c52418080b79444147136fe" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 54776 } } peers { permanent_uuid: "1d8ef998f2044df3a15ce2cee08280db" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 33187 } } I0703 01:41:46.163079 23646 heartbeater.cc:467] Master 127.5.240.254:45494 was elected leader, sending a full tablet report... I0703 01:41:46.162844 23910 raft_consensus.cc:325] T d57e314b004b420f88b334f2cced89a1 P 1d8ef998f2044df3a15ce2cee08280db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "483d3c608319410d9718f61955b1f929" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 48535 } } peers { permanent_uuid: "890887632c52418080b79444147136fe" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 54776 } } peers { permanent_uuid: "1d8ef998f2044df3a15ce2cee08280db" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 33187 } } I0703 01:41:46.162991 23918 raft_consensus.cc:457] T d57e314b004b420f88b334f2cced89a1 P 890887632c52418080b79444147136fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0703 01:41:46.163172 23910 raft_consensus.cc:351] T d57e314b004b420f88b334f2cced89a1 P 1d8ef998f2044df3a15ce2cee08280db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:41:46.163231 23910 raft_consensus.cc:628] T d57e314b004b420f88b334f2cced89a1 P 1d8ef998f2044df3a15ce2cee08280db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d8ef998f2044df3a15ce2cee08280db, State: Initialized, Role: FOLLOWER I0703 01:41:46.163274 23918 raft_consensus.cc:2730] T d57e314b004b420f88b334f2cced89a1 P 890887632c52418080b79444147136fe: Snoozing failure detection for 2.078s (starting election) I0703 01:41:46.163444 23918 raft_consensus.cc:479] T d57e314b004b420f88b334f2cced89a1 P 890887632c52418080b79444147136fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "483d3c608319410d9718f61955b1f929" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 48535 } } peers { permanent_uuid: "890887632c52418080b79444147136fe" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 54776 } } peers { permanent_uuid: "1d8ef998f2044df3a15ce2cee08280db" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 33187 } } I0703 01:41:46.163671 23910 consensus_queue.cc:227] T d57e314b004b420f88b334f2cced89a1 P 1d8ef998f2044df3a15ce2cee08280db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "483d3c608319410d9718f61955b1f929" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 48535 } } peers { permanent_uuid: "890887632c52418080b79444147136fe" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 54776 } } peers { permanent_uuid: "1d8ef998f2044df3a15ce2cee08280db" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 33187 } } I0703 01:41:46.164539 23768 heartbeater.cc:467] Master 127.5.240.254:45494 was elected leader, sending a full tablet report... I0703 01:41:46.165063 23910 ts_tablet_manager.cc:1073] T d57e314b004b420f88b334f2cced89a1 P 1d8ef998f2044df3a15ce2cee08280db: Time spent starting tablet: real 0.014s user 0.007s sys 0.004s I0703 01:41:46.166359 23918 leader_election.cc:243] T d57e314b004b420f88b334f2cced89a1 P 890887632c52418080b79444147136fe [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 483d3c608319410d9718f61955b1f929 (127.5.240.193:48535) I0703 01:41:46.167013 23911 ts_tablet_manager.cc:1073] T d57e314b004b420f88b334f2cced89a1 P 890887632c52418080b79444147136fe: Time spent starting tablet: real 0.019s user 0.010s sys 0.007s I0703 01:41:46.167939 23918 leader_election.cc:243] T d57e314b004b420f88b334f2cced89a1 P 890887632c52418080b79444147136fe [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1d8ef998f2044df3a15ce2cee08280db (127.5.240.196:33187) I0703 01:41:46.177563 23358 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "d57e314b004b420f88b334f2cced89a1" candidate_uuid: "890887632c52418080b79444147136fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "483d3c608319410d9718f61955b1f929" is_pre_election: true I0703 01:41:46.178288 23358 raft_consensus.cc:2730] T d57e314b004b420f88b334f2cced89a1 P 483d3c608319410d9718f61955b1f929: Snoozing failure detection for 1.563s (vote granted) I0703 01:41:46.178396 23358 raft_consensus.cc:2243] T d57e314b004b420f88b334f2cced89a1 P 483d3c608319410d9718f61955b1f929 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 890887632c52418080b79444147136fe in term 0. I0703 01:41:46.179090 23537 leader_election.cc:379] T d57e314b004b420f88b334f2cced89a1 P 890887632c52418080b79444147136fe [CANDIDATE]: Term 1 pre-election: Vote granted by peer 483d3c608319410d9718f61955b1f929 (127.5.240.193:48535) I0703 01:41:46.179164 23537 leader_election.cc:270] T d57e314b004b420f88b334f2cced89a1 P 890887632c52418080b79444147136fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0703 01:41:46.179440 23918 raft_consensus.cc:2730] T d57e314b004b420f88b334f2cced89a1 P 890887632c52418080b79444147136fe: Snoozing failure detection for 1.880s (election complete) I0703 01:41:46.179512 23918 raft_consensus.cc:2537] T d57e314b004b420f88b334f2cced89a1 P 890887632c52418080b79444147136fe [term 0 FOLLOWER]: Leader pre-election won for term 1 I0703 01:41:46.179605 23918 raft_consensus.cc:457] T d57e314b004b420f88b334f2cced89a1 P 890887632c52418080b79444147136fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0703 01:41:46.179677 23918 raft_consensus.cc:2730] T d57e314b004b420f88b334f2cced89a1 P 890887632c52418080b79444147136fe: Snoozing failure detection for 1.812s (starting election) I0703 01:41:46.179745 23918 raft_consensus.cc:2782] T d57e314b004b420f88b334f2cced89a1 P 890887632c52418080b79444147136fe [term 0 FOLLOWER]: Advancing to term 1 I0703 01:41:46.179502 23725 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "d57e314b004b420f88b334f2cced89a1" candidate_uuid: "890887632c52418080b79444147136fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d8ef998f2044df3a15ce2cee08280db" is_pre_election: true I0703 01:41:46.180234 23725 raft_consensus.cc:2730] T d57e314b004b420f88b334f2cced89a1 P 1d8ef998f2044df3a15ce2cee08280db: Snoozing failure detection for 2.200s (vote granted) I0703 01:41:46.180327 23725 raft_consensus.cc:2243] T d57e314b004b420f88b334f2cced89a1 P 1d8ef998f2044df3a15ce2cee08280db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 890887632c52418080b79444147136fe in term 0. I0703 01:41:46.181069 23535 leader_election.cc:379] T d57e314b004b420f88b334f2cced89a1 P 890887632c52418080b79444147136fe [CANDIDATE]: Term 1 pre-election: Vote granted by peer 1d8ef998f2044df3a15ce2cee08280db (127.5.240.196:33187) I0703 01:41:46.181819 23918 raft_consensus.cc:479] T d57e314b004b420f88b334f2cced89a1 P 890887632c52418080b79444147136fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "483d3c608319410d9718f61955b1f929" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 48535 } } peers { permanent_uuid: "890887632c52418080b79444147136fe" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 54776 } } peers { permanent_uuid: "1d8ef998f2044df3a15ce2cee08280db" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 33187 } } I0703 01:41:46.182152 23918 leader_election.cc:243] T d57e314b004b420f88b334f2cced89a1 P 890887632c52418080b79444147136fe [CANDIDATE]: Term 1 election: Requesting vote from peer 483d3c608319410d9718f61955b1f929 (127.5.240.193:48535) I0703 01:41:46.182322 23918 leader_election.cc:243] T d57e314b004b420f88b334f2cced89a1 P 890887632c52418080b79444147136fe [CANDIDATE]: Term 1 election: Requesting vote from peer 1d8ef998f2044df3a15ce2cee08280db (127.5.240.196:33187) I0703 01:41:46.182924 23358 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "d57e314b004b420f88b334f2cced89a1" candidate_uuid: "890887632c52418080b79444147136fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "483d3c608319410d9718f61955b1f929" I0703 01:41:46.183043 23358 raft_consensus.cc:2782] T d57e314b004b420f88b334f2cced89a1 P 483d3c608319410d9718f61955b1f929 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:41:46.183140 23358 raft_consensus.cc:2730] T d57e314b004b420f88b334f2cced89a1 P 483d3c608319410d9718f61955b1f929: Snoozing failure detection for 2.089s (vote granted) I0703 01:41:46.183111 23725 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "d57e314b004b420f88b334f2cced89a1" candidate_uuid: "890887632c52418080b79444147136fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d8ef998f2044df3a15ce2cee08280db" I0703 01:41:46.183241 23725 raft_consensus.cc:2782] T d57e314b004b420f88b334f2cced89a1 P 1d8ef998f2044df3a15ce2cee08280db [term 0 FOLLOWER]: Advancing to term 1 I0703 01:41:46.183338 23725 raft_consensus.cc:2730] T d57e314b004b420f88b334f2cced89a1 P 1d8ef998f2044df3a15ce2cee08280db: Snoozing failure detection for 1.558s (vote granted) I0703 01:41:46.185253 23358 raft_consensus.cc:2243] T d57e314b004b420f88b334f2cced89a1 P 483d3c608319410d9718f61955b1f929 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 890887632c52418080b79444147136fe in term 1. I0703 01:41:46.185355 23725 raft_consensus.cc:2243] T d57e314b004b420f88b334f2cced89a1 P 1d8ef998f2044df3a15ce2cee08280db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 890887632c52418080b79444147136fe in term 1. I0703 01:41:46.185744 23537 leader_election.cc:379] T d57e314b004b420f88b334f2cced89a1 P 890887632c52418080b79444147136fe [CANDIDATE]: Term 1 election: Vote granted by peer 483d3c608319410d9718f61955b1f929 (127.5.240.193:48535) I0703 01:41:46.185835 23535 leader_election.cc:379] T d57e314b004b420f88b334f2cced89a1 P 890887632c52418080b79444147136fe [CANDIDATE]: Term 1 election: Vote granted by peer 1d8ef998f2044df3a15ce2cee08280db (127.5.240.196:33187) I0703 01:41:46.185902 23535 leader_election.cc:270] T d57e314b004b420f88b334f2cced89a1 P 890887632c52418080b79444147136fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:41:46.186087 23918 raft_consensus.cc:2730] T d57e314b004b420f88b334f2cced89a1 P 890887632c52418080b79444147136fe: Snoozing failure detection for 1.535s (election complete) I0703 01:41:46.186156 23918 raft_consensus.cc:2537] T d57e314b004b420f88b334f2cced89a1 P 890887632c52418080b79444147136fe [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:41:46.186272 23918 raft_consensus.cc:593] T d57e314b004b420f88b334f2cced89a1 P 890887632c52418080b79444147136fe [term 1 LEADER]: Becoming Leader. State: Replica: 890887632c52418080b79444147136fe, State: Running, Role: LEADER I0703 01:41:46.186645 23918 consensus_queue.cc:204] T d57e314b004b420f88b334f2cced89a1 P 890887632c52418080b79444147136fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "483d3c608319410d9718f61955b1f929" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 48535 } } peers { permanent_uuid: "890887632c52418080b79444147136fe" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 54776 } } peers { permanent_uuid: "1d8ef998f2044df3a15ce2cee08280db" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 33187 } } I0703 01:41:46.189797 23237 catalog_manager.cc:3798] T d57e314b004b420f88b334f2cced89a1 P 890887632c52418080b79444147136fe reported cstate change: term changed from 0 to 1, leader changed from to 890887632c52418080b79444147136fe (127.5.240.195). New cstate: current_term: 1 leader_uuid: "890887632c52418080b79444147136fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "483d3c608319410d9718f61955b1f929" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 48535 } } peers { permanent_uuid: "890887632c52418080b79444147136fe" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 54776 } } peers { permanent_uuid: "1d8ef998f2044df3a15ce2cee08280db" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 33187 } } } W0703 01:41:46.199663 23402 tablet.cc:1678] T d57e314b004b420f88b334f2cced89a1 P 483d3c608319410d9718f61955b1f929: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:41:46.209039 6083 external_mini_cluster.cc:490] 5 TS(s) registered with all masters I0703 01:41:46.210177 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 483d3c608319410d9718f61955b1f929 to finish bootstrapping I0703 01:41:46.215896 6083 ts_itest-base.cc:233] Waiting for 0 tablets on tserver b228ec228f89487b8a2aba0b6b755217 to finish bootstrapping I0703 01:41:46.222304 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 890887632c52418080b79444147136fe to finish bootstrapping I0703 01:41:46.226590 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 1d8ef998f2044df3a15ce2cee08280db to finish bootstrapping I0703 01:41:46.231946 6083 ts_itest-base.cc:233] Waiting for 0 tablets on tserver ac322d270bf445adaa0038eb0dc24410 to finish bootstrapping I0703 01:41:46.238957 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 23272 I0703 01:41:46.239548 23398 generic_service.cc:169] Checking for leaks (request via RPC) W0703 01:41:46.252861 23769 tablet.cc:1678] T d57e314b004b420f88b334f2cced89a1 P 1d8ef998f2044df3a15ce2cee08280db: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:41:46.288213 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 23272 W0703 01:41:46.296665 23537 connection.cc:512] client connection to 127.5.240.193:48535 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:48535): Connection reset by peer (error 104) W0703 01:41:46.296954 23222 connection.cc:512] server connection from 127.5.240.193:36549 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:36549): Connection reset by peer (error 104) W0703 01:41:46.366868 23647 tablet.cc:1678] T d57e314b004b420f88b334f2cced89a1 P 890887632c52418080b79444147136fe: Can't schedule compaction. Clean time has not been advanced past its initial value. Per-server replica distribution summary: Statistic | Value -----------------------+---------- Minimum Replica Count | 0 Maximum Replica Count | 1 Average Replica Count | 0.500000 Per-table replica distribution summary: Replica Skew | Value --------------+---------- Minimum | 1 Maximum | 1 Average | 1.000000 I0703 01:41:46.666457 23725 raft_consensus.cc:1081] T d57e314b004b420f88b334f2cced89a1 P 1d8ef998f2044df3a15ce2cee08280db [term 1 FOLLOWER]: Refusing update from remote peer 890887632c52418080b79444147136fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0703 01:41:46.667423 23918 consensus_queue.cc:962] T d57e314b004b420f88b334f2cced89a1 P 890887632c52418080b79444147136fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d8ef998f2044df3a15ce2cee08280db" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 33187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s W0703 01:41:46.702358 23537 consensus_peers.cc:433] T d57e314b004b420f88b334f2cced89a1 P 890887632c52418080b79444147136fe -> Peer 483d3c608319410d9718f61955b1f929 (127.5.240.193:48535): Couldn't send request to peer 483d3c608319410d9718f61955b1f929 for tablet d57e314b004b420f88b334f2cced89a1. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.193:48535: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. I0703 01:41:46.752455 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 23405 I0703 01:41:46.755677 23521 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:41:46.794492 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 23405 W0703 01:41:46.802558 23220 connection.cc:512] server connection from 127.5.240.194:48451 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:48451): Connection reset by peer (error 104) I0703 01:41:46.802809 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 23527 I0703 01:41:46.803494 23643 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:41:46.847947 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 23527 W0703 01:41:46.856746 23222 connection.cc:512] server connection from 127.5.240.195:38057 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.195:38057): Connection reset by peer (error 104) W0703 01:41:46.856940 23659 connection.cc:512] server connection from 127.5.240.195:38760 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.195:38760): Connection reset by peer (error 104) I0703 01:41:46.857059 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 23649 I0703 01:41:46.857779 23765 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:41:46.900283 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 23649 W0703 01:41:46.908843 23222 connection.cc:512] server connection from 127.5.240.196:54371 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.196:54371): Connection reset by peer (error 104) I0703 01:41:46.908951 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 23771 I0703 01:41:46.909705 23887 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:41:46.947439 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 23771 W0703 01:41:46.955590 23220 connection.cc:512] server connection from 127.5.240.197:46730 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.197:46730): Connection reset by peer (error 104) I0703 01:41:46.955668 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master pid 23212 I0703 01:41:46.956372 23268 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:41:47.018889 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master with pid 23212 [ OK ] RebalanceStartCriteriaTest.TabletServerIsDown/0 (3018 ms) [ RUN ] RebalanceStartCriteriaTest.TabletServerIsDown/1 I0703 01:41:47.041204 6083 test_util.cc:208] Using random seed: 1221831074 I0703 01:41:47.043016 6083 ts_itest-base.cc:113] Starting cluster with: I0703 01:41:47.043053 6083 ts_itest-base.cc:114] -------------- I0703 01:41:47.043097 6083 ts_itest-base.cc:115] 5 tablet servers I0703 01:41:47.043129 6083 ts_itest-base.cc:116] 3 replicas per TS I0703 01:41:47.043155 6083 ts_itest-base.cc:117] -------------- I0703 01:41:47.044286 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:38173 --webserver_interface=127.5.240.254 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:41:47.226912 23992 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:41:47.227140 23992 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:41:47.229351 23992 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0703 01:41:47.229415 23992 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0703 01:41:47.229465 23992 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:41:47.229506 23992 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0703 01:41:47.229562 23992 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0703 01:41:47.237232 23992 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --ipki_ca_key_size=1024 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:38173 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.5.240.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:41:47.243216 23992 master_main.cc:80] Initializing master server... I0703 01:41:47.243911 23992 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 207208us I0703 01:41:47.244947 23992 fs_manager.cc:261] Metadata directory not provided I0703 01:41:47.245016 23992 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:41:47.245662 23992 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:41:47.245730 23992 server_base.cc:433] Attempting to create new FS layout instead I0703 01:41:47.249855 23992 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/instance: uuid: "c83d1d2936d145dea641eb0b6b72d8cb" format_stamp: "Formatted at 2018-07-03 08:41:47 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:47.250373 23992 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "c83d1d2936d145dea641eb0b6b72d8cb" format_stamp: "Formatted at 2018-07-03 08:41:47 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:47.259011 23992 fs_manager.cc:501] Time spent creating directory manager: real 0.008s user 0.003s sys 0.004s I0703 01:41:47.260236 23992 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:41:47.260637 23992 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:41:47.263790 23992 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s I0703 01:41:47.263842 23992 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal uuid: "c83d1d2936d145dea641eb0b6b72d8cb" format_stamp: "Formatted at 2018-07-03 08:41:47 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:47.263957 23992 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:41:47.297199 23992 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:41:47.298633 23992 master_main.cc:83] Starting Master server... I0703 01:41:47.335697 23992 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.254:38173 I0703 01:41:47.336155 23992 webserver.cc:175] Starting webserver on 127.5.240.254:0 I0703 01:41:47.336195 23992 webserver.cc:186] Document root disabled I0703 01:41:47.337383 23992 webserver.cc:313] Webserver started. Bound to: http://127.5.240.254:60495/ I0703 01:41:47.337976 23992 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb I0703 01:41:47.338413 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master as pid 23992 I0703 01:41:47.339088 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.5.240.193:0 --local_ip_for_outbound_sockets=127.5.240.193 --webserver_interface=127.5.240.193 --webserver_port=0 --tserver_master_addrs=127.5.240.254:38173 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0703 01:41:47.341220 24052 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:41:47.352988 24052 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P c83d1d2936d145dea641eb0b6b72d8cb: Bootstrap starting. I0703 01:41:47.355010 24052 tablet_bootstrap.cc:578] T 00000000000000000000000000000000 P c83d1d2936d145dea641eb0b6b72d8cb: No blocks or log segments found. Creating new log. I0703 01:41:47.355760 24052 log.cc:525] T 00000000000000000000000000000000 P c83d1d2936d145dea641eb0b6b72d8cb: Log is configured to *not* fsync() on all Append() calls I0703 01:41:47.358016 24052 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P c83d1d2936d145dea641eb0b6b72d8cb: No bootstrap required, opened a new log I0703 01:41:47.364179 24052 raft_consensus.cc:325] T 00000000000000000000000000000000 P c83d1d2936d145dea641eb0b6b72d8cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c83d1d2936d145dea641eb0b6b72d8cb" member_type: VOTER } I0703 01:41:47.364372 24052 raft_consensus.cc:351] T 00000000000000000000000000000000 P c83d1d2936d145dea641eb0b6b72d8cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:41:47.364434 24052 raft_consensus.cc:628] T 00000000000000000000000000000000 P c83d1d2936d145dea641eb0b6b72d8cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c83d1d2936d145dea641eb0b6b72d8cb, State: Initialized, Role: FOLLOWER I0703 01:41:47.364899 24052 consensus_queue.cc:227] T 00000000000000000000000000000000 P c83d1d2936d145dea641eb0b6b72d8cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c83d1d2936d145dea641eb0b6b72d8cb" member_type: VOTER } I0703 01:41:47.365049 24052 raft_consensus.cc:365] T 00000000000000000000000000000000 P c83d1d2936d145dea641eb0b6b72d8cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:41:47.365129 24052 raft_consensus.cc:457] T 00000000000000000000000000000000 P c83d1d2936d145dea641eb0b6b72d8cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:41:47.365211 24052 raft_consensus.cc:2730] T 00000000000000000000000000000000 P c83d1d2936d145dea641eb0b6b72d8cb: Snoozing failure detection for 1.834s (starting election) I0703 01:41:47.365298 24052 raft_consensus.cc:2782] T 00000000000000000000000000000000 P c83d1d2936d145dea641eb0b6b72d8cb [term 0 FOLLOWER]: Advancing to term 1 I0703 01:41:47.366628 24052 raft_consensus.cc:479] T 00000000000000000000000000000000 P c83d1d2936d145dea641eb0b6b72d8cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c83d1d2936d145dea641eb0b6b72d8cb" member_type: VOTER } I0703 01:41:47.366989 24052 leader_election.cc:270] T 00000000000000000000000000000000 P c83d1d2936d145dea641eb0b6b72d8cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:41:47.367934 24055 raft_consensus.cc:2730] T 00000000000000000000000000000000 P c83d1d2936d145dea641eb0b6b72d8cb: Snoozing failure detection for 1.518s (election complete) I0703 01:41:47.368182 24055 raft_consensus.cc:2537] T 00000000000000000000000000000000 P c83d1d2936d145dea641eb0b6b72d8cb [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:41:47.370314 24055 raft_consensus.cc:593] T 00000000000000000000000000000000 P c83d1d2936d145dea641eb0b6b72d8cb [term 1 LEADER]: Becoming Leader. State: Replica: c83d1d2936d145dea641eb0b6b72d8cb, State: Running, Role: LEADER I0703 01:41:47.370771 24055 consensus_queue.cc:204] T 00000000000000000000000000000000 P c83d1d2936d145dea641eb0b6b72d8cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c83d1d2936d145dea641eb0b6b72d8cb" member_type: VOTER } I0703 01:41:47.372822 24052 sys_catalog.cc:337] T 00000000000000000000000000000000 P c83d1d2936d145dea641eb0b6b72d8cb [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "c83d1d2936d145dea641eb0b6b72d8cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c83d1d2936d145dea641eb0b6b72d8cb" member_type: VOTER } } I0703 01:41:47.373062 24052 sys_catalog.cc:340] T 00000000000000000000000000000000 P c83d1d2936d145dea641eb0b6b72d8cb [sys.catalog]: This master's current role is: LEADER I0703 01:41:47.372725 24056 sys_catalog.cc:337] T 00000000000000000000000000000000 P c83d1d2936d145dea641eb0b6b72d8cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c83d1d2936d145dea641eb0b6b72d8cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c83d1d2936d145dea641eb0b6b72d8cb" member_type: VOTER } } I0703 01:41:47.373828 24056 sys_catalog.cc:340] T 00000000000000000000000000000000 P c83d1d2936d145dea641eb0b6b72d8cb [sys.catalog]: This master's current role is: LEADER I0703 01:41:47.373011 24057 sys_catalog.cc:337] T 00000000000000000000000000000000 P c83d1d2936d145dea641eb0b6b72d8cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader c83d1d2936d145dea641eb0b6b72d8cb. Latest consensus state: current_term: 1 leader_uuid: "c83d1d2936d145dea641eb0b6b72d8cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c83d1d2936d145dea641eb0b6b72d8cb" member_type: VOTER } } I0703 01:41:47.374689 24057 sys_catalog.cc:340] T 00000000000000000000000000000000 P c83d1d2936d145dea641eb0b6b72d8cb [sys.catalog]: This master's current role is: LEADER I0703 01:41:47.375636 24052 sys_catalog.cc:424] T 00000000000000000000000000000000 P c83d1d2936d145dea641eb0b6b72d8cb [sys.catalog]: configured and running, proceeding with master startup. I0703 01:41:47.375886 24059 catalog_manager.cc:1006] Loading table and tablet metadata into memory... I0703 01:41:47.378396 23992 master_main.cc:86] Master server successfully started. I0703 01:41:47.384114 24059 catalog_manager.cc:1017] Initializing Kudu internal certificate authority... I0703 01:41:47.414271 24059 catalog_manager.cc:904] Generated new certificate authority record I0703 01:41:47.416110 24059 catalog_manager.cc:1026] Loading token signing keys... I0703 01:41:47.427247 24059 catalog_manager.cc:4102] T 00000000000000000000000000000000 P c83d1d2936d145dea641eb0b6b72d8cb: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:41:47.515306 24053 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:41:47.515511 24053 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:41:47.517534 24053 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0703 01:41:47.517601 24053 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:41:47.517729 24053 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.193 I0703 01:41:47.525108 24053 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.5.240.193 --webserver_port=0 --tserver_master_addrs=127.5.240.254:38173 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.193 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:41:47.531705 24053 tablet_server_main.cc:85] Initializing tablet server... I0703 01:41:47.533223 24053 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 207208us I0703 01:41:47.533798 24053 fs_manager.cc:261] Metadata directory not provided I0703 01:41:47.533866 24053 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:41:47.534085 24053 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:41:47.534153 24053 server_base.cc:433] Attempting to create new FS layout instead I0703 01:41:47.538255 24053 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "4ce55d2fbe26483db71343152f38abb8" format_stamp: "Formatted at 2018-07-03 08:41:47 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:47.538774 24053 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "4ce55d2fbe26483db71343152f38abb8" format_stamp: "Formatted at 2018-07-03 08:41:47 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:47.546823 24053 fs_manager.cc:501] Time spent creating directory manager: real 0.008s user 0.003s sys 0.004s I0703 01:41:47.548034 24053 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:41:47.548426 24053 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:41:47.552486 24053 fs_manager.cc:417] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s I0703 01:41:47.552536 24053 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal uuid: "4ce55d2fbe26483db71343152f38abb8" format_stamp: "Formatted at 2018-07-03 08:41:47 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:47.552696 24053 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:41:47.578753 24053 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:41:47.581264 24053 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:41:47.581691 24053 tablet_server_main.cc:90] Starting tablet server... I0703 01:41:47.682374 24053 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.193:39726 I0703 01:41:47.682862 24053 webserver.cc:175] Starting webserver on 127.5.240.193:0 I0703 01:41:47.682910 24053 webserver.cc:186] Document root disabled I0703 01:41:47.683604 24053 webserver.cc:313] Webserver started. Bound to: http://127.5.240.193:35475/ I0703 01:41:47.684135 24053 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb I0703 01:41:47.685828 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 24053 I0703 01:41:47.686669 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.5.240.194:0 --local_ip_for_outbound_sockets=127.5.240.194 --webserver_interface=127.5.240.194 --webserver_port=0 --tserver_master_addrs=127.5.240.254:38173 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0703 01:41:47.701607 24053 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:41:47.718451 24182 heartbeater.cc:346] Connected to a master server at 127.5.240.254:38173 I0703 01:41:47.718608 24182 heartbeater.cc:426] Registering TS with master... I0703 01:41:47.719389 24182 heartbeater.cc:475] Master 127.5.240.254:38173 requested a full tablet report, sending... I0703 01:41:47.720953 24019 ts_manager.cc:98] Registered new tserver with Master: 4ce55d2fbe26483db71343152f38abb8 (127.5.240.193:39726) I0703 01:41:47.722744 24019 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.193:60774 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:41:47.862396 24183 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:41:47.862639 24183 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:41:47.864653 24183 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0703 01:41:47.864714 24183 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:41:47.864816 24183 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.194 I0703 01:41:47.871364 24183 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.194:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.5.240.194 --webserver_port=0 --tserver_master_addrs=127.5.240.254:38173 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.194 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:41:47.878170 24183 tablet_server_main.cc:85] Initializing tablet server... I0703 01:41:47.879791 24183 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 207208us I0703 01:41:47.880332 24183 fs_manager.cc:261] Metadata directory not provided I0703 01:41:47.880404 24183 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:41:47.880653 24183 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:41:47.880722 24183 server_base.cc:433] Attempting to create new FS layout instead I0703 01:41:47.884770 24183 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "246f10eee67c4934b4a502a5ae95fbd9" format_stamp: "Formatted at 2018-07-03 08:41:47 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:47.885305 24183 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "246f10eee67c4934b4a502a5ae95fbd9" format_stamp: "Formatted at 2018-07-03 08:41:47 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:47.892494 24183 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.005s sys 0.002s I0703 01:41:47.893694 24183 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:41:47.894067 24183 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:41:47.897163 24183 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s I0703 01:41:47.897214 24183 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal uuid: "246f10eee67c4934b4a502a5ae95fbd9" format_stamp: "Formatted at 2018-07-03 08:41:47 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:47.897342 24183 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:41:47.925495 24183 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:41:47.927314 24183 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:41:47.927618 24183 tablet_server_main.cc:90] Starting tablet server... I0703 01:41:48.013669 24183 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.194:59210 I0703 01:41:48.014094 24183 webserver.cc:175] Starting webserver on 127.5.240.194:0 I0703 01:41:48.014140 24183 webserver.cc:186] Document root disabled I0703 01:41:48.015328 24183 webserver.cc:313] Webserver started. Bound to: http://127.5.240.194:51119/ I0703 01:41:48.015873 24183 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb I0703 01:41:48.020344 24183 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:41:48.022006 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 24183 I0703 01:41:48.022658 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.5.240.195:0 --local_ip_for_outbound_sockets=127.5.240.195 --webserver_interface=127.5.240.195 --webserver_port=0 --tserver_master_addrs=127.5.240.254:38173 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0703 01:41:48.029774 24305 heartbeater.cc:346] Connected to a master server at 127.5.240.254:38173 I0703 01:41:48.029944 24305 heartbeater.cc:426] Registering TS with master... I0703 01:41:48.030853 24305 heartbeater.cc:475] Master 127.5.240.254:38173 requested a full tablet report, sending... I0703 01:41:48.031901 24019 ts_manager.cc:98] Registered new tserver with Master: 246f10eee67c4934b4a502a5ae95fbd9 (127.5.240.194:59210) I0703 01:41:48.032749 24019 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.194:38329 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:41:48.184106 24308 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:41:48.184311 24308 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:41:48.186254 24308 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0703 01:41:48.186300 24308 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:41:48.186383 24308 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.195 I0703 01:41:48.192745 24308 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.195:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.5.240.195 --webserver_port=0 --tserver_master_addrs=127.5.240.254:38173 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.195 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:41:48.199298 24308 tablet_server_main.cc:85] Initializing tablet server... I0703 01:41:48.200783 24308 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 207708us I0703 01:41:48.201303 24308 fs_manager.cc:261] Metadata directory not provided I0703 01:41:48.201365 24308 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:41:48.201982 24308 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:41:48.202045 24308 server_base.cc:433] Attempting to create new FS layout instead I0703 01:41:48.206049 24308 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "e3c0ca1fc3254b5fa1acf121412f4b1f" format_stamp: "Formatted at 2018-07-03 08:41:48 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:48.206642 24308 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "e3c0ca1fc3254b5fa1acf121412f4b1f" format_stamp: "Formatted at 2018-07-03 08:41:48 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:48.214206 24308 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.002s sys 0.004s I0703 01:41:48.215387 24308 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:41:48.215790 24308 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:41:48.218925 24308 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0703 01:41:48.218969 24308 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal uuid: "e3c0ca1fc3254b5fa1acf121412f4b1f" format_stamp: "Formatted at 2018-07-03 08:41:48 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:48.219084 24308 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:41:48.265867 24308 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:41:48.268093 24308 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:41:48.268388 24308 tablet_server_main.cc:90] Starting tablet server... I0703 01:41:48.354979 24308 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.195:46531 I0703 01:41:48.355363 24308 webserver.cc:175] Starting webserver on 127.5.240.195:0 I0703 01:41:48.355406 24308 webserver.cc:186] Document root disabled I0703 01:41:48.356114 24308 webserver.cc:313] Webserver started. Bound to: http://127.5.240.195:33613/ I0703 01:41:48.356657 24308 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb I0703 01:41:48.358974 24308 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:41:48.359791 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 24308 I0703 01:41:48.360641 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.5.240.196:0 --local_ip_for_outbound_sockets=127.5.240.196 --webserver_interface=127.5.240.196 --webserver_port=0 --tserver_master_addrs=127.5.240.254:38173 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0703 01:41:48.370760 24427 heartbeater.cc:346] Connected to a master server at 127.5.240.254:38173 I0703 01:41:48.370945 24427 heartbeater.cc:426] Registering TS with master... I0703 01:41:48.371959 24427 heartbeater.cc:475] Master 127.5.240.254:38173 requested a full tablet report, sending... I0703 01:41:48.373234 24019 ts_manager.cc:98] Registered new tserver with Master: e3c0ca1fc3254b5fa1acf121412f4b1f (127.5.240.195:46531) I0703 01:41:48.374073 24019 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.195:55246 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:41:48.524332 24430 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:41:48.524571 24430 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:41:48.526628 24430 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0703 01:41:48.526692 24430 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:41:48.526798 24430 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.196 I0703 01:41:48.533306 24430 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.196:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb --webserver_interface=127.5.240.196 --webserver_port=0 --tserver_master_addrs=127.5.240.254:38173 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.196 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:41:48.539533 24430 tablet_server_main.cc:85] Initializing tablet server... I0703 01:41:48.541081 24430 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 207708us I0703 01:41:48.541674 24430 fs_manager.cc:261] Metadata directory not provided I0703 01:41:48.541749 24430 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:41:48.541995 24430 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:41:48.542063 24430 server_base.cc:433] Attempting to create new FS layout instead I0703 01:41:48.546548 24430 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/instance: uuid: "201b0b2c1dca433a822d9d7eadaa5a40" format_stamp: "Formatted at 2018-07-03 08:41:48 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:48.547134 24430 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal/instance: uuid: "201b0b2c1dca433a822d9d7eadaa5a40" format_stamp: "Formatted at 2018-07-03 08:41:48 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:48.554477 24430 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.004s sys 0.002s I0703 01:41:48.555675 24430 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:41:48.556053 24430 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:41:48.559690 24430 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s I0703 01:41:48.559741 24430 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal uuid: "201b0b2c1dca433a822d9d7eadaa5a40" format_stamp: "Formatted at 2018-07-03 08:41:48 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:48.559870 24430 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:41:48.592386 24430 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:41:48.594663 24430 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:41:48.595023 24430 tablet_server_main.cc:90] Starting tablet server... I0703 01:41:48.679536 24430 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.196:44176 I0703 01:41:48.679958 24430 webserver.cc:175] Starting webserver on 127.5.240.196:0 I0703 01:41:48.680003 24430 webserver.cc:186] Document root disabled I0703 01:41:48.681164 24430 webserver.cc:313] Webserver started. Bound to: http://127.5.240.196:33271/ I0703 01:41:48.681721 24430 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb I0703 01:41:48.683507 24430 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:41:48.688357 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 24430 I0703 01:41:48.689244 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data --block_manager=log --rpc_bind_addresses=127.5.240.197:0 --local_ip_for_outbound_sockets=127.5.240.197 --webserver_interface=127.5.240.197 --webserver_port=0 --tserver_master_addrs=127.5.240.254:38173 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0703 01:41:48.698297 24549 heartbeater.cc:346] Connected to a master server at 127.5.240.254:38173 I0703 01:41:48.698456 24549 heartbeater.cc:426] Registering TS with master... I0703 01:41:48.699273 24549 heartbeater.cc:475] Master 127.5.240.254:38173 requested a full tablet report, sending... I0703 01:41:48.700368 24019 ts_manager.cc:98] Registered new tserver with Master: 201b0b2c1dca433a822d9d7eadaa5a40 (127.5.240.196:44176) I0703 01:41:48.701545 24019 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.196:36154 I0703 01:41:48.724759 24182 heartbeater.cc:467] Master 127.5.240.254:38173 was elected leader, sending a full tablet report... WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:41:48.854058 24552 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:41:48.854267 24552 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:41:48.856215 24552 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0703 01:41:48.856262 24552 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:41:48.856346 24552 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.197 I0703 01:41:48.863862 24552 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.197:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb --webserver_interface=127.5.240.197 --webserver_port=0 --tserver_master_addrs=127.5.240.254:38173 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.197 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:41:48.870297 24552 tablet_server_main.cc:85] Initializing tablet server... I0703 01:41:48.871812 24552 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 207708us I0703 01:41:48.872377 24552 fs_manager.cc:261] Metadata directory not provided I0703 01:41:48.872439 24552 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:41:48.872665 24552 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:41:48.872727 24552 server_base.cc:433] Attempting to create new FS layout instead I0703 01:41:48.876706 24552 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/instance: uuid: "55c0a44b280e4015a22c8302291619c6" format_stamp: "Formatted at 2018-07-03 08:41:48 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:48.877230 24552 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal/instance: uuid: "55c0a44b280e4015a22c8302291619c6" format_stamp: "Formatted at 2018-07-03 08:41:48 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:48.884526 24552 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.004s sys 0.002s I0703 01:41:48.885740 24552 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:41:48.886124 24552 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:41:48.889750 24552 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0703 01:41:48.889802 24552 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal uuid: "55c0a44b280e4015a22c8302291619c6" format_stamp: "Formatted at 2018-07-03 08:41:48 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:48.889935 24552 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:41:48.917560 24552 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:41:48.919806 24552 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:41:48.920186 24552 tablet_server_main.cc:90] Starting tablet server... I0703 01:41:49.006018 24552 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.197:39855 I0703 01:41:49.006439 24552 webserver.cc:175] Starting webserver on 127.5.240.197:0 I0703 01:41:49.006487 24552 webserver.cc:186] Document root disabled I0703 01:41:49.007684 24552 webserver.cc:313] Webserver started. Bound to: http://127.5.240.197:33556/ I0703 01:41:49.008211 24552 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb I0703 01:41:49.009960 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 24552 I0703 01:41:49.010514 24552 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:41:49.023260 24671 heartbeater.cc:346] Connected to a master server at 127.5.240.254:38173 I0703 01:41:49.023399 24671 heartbeater.cc:426] Registering TS with master... I0703 01:41:49.024183 24671 heartbeater.cc:475] Master 127.5.240.254:38173 requested a full tablet report, sending... I0703 01:41:49.025194 24019 ts_manager.cc:98] Registered new tserver with Master: 55c0a44b280e4015a22c8302291619c6 (127.5.240.197:39855) I0703 01:41:49.025996 6083 external_mini_cluster.cc:490] 5 TS(s) registered with all masters I0703 01:41:49.026546 24019 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.197:42555 I0703 01:41:49.034927 24305 heartbeater.cc:467] Master 127.5.240.254:38173 was elected leader, sending a full tablet report... I0703 01:41:49.037008 24018 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.5.240.254:48614: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 3 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0703 01:41:49.063006 24486 tablet_service.cc:759] Processing CreateTablet for tablet 74661dce117e41d6b3e49c10fea0489d (table=TestTable [id=a2abb76e7d47441ca6b5eba32f19b719]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:41:49.063293 24119 tablet_service.cc:759] Processing CreateTablet for tablet 74661dce117e41d6b3e49c10fea0489d (table=TestTable [id=a2abb76e7d47441ca6b5eba32f19b719]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:41:49.063935 24486 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 74661dce117e41d6b3e49c10fea0489d. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:41:49.064010 24119 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 74661dce117e41d6b3e49c10fea0489d. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:41:49.064798 24242 tablet_service.cc:759] Processing CreateTablet for tablet 74661dce117e41d6b3e49c10fea0489d (table=TestTable [id=a2abb76e7d47441ca6b5eba32f19b719]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:41:49.065464 24242 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 74661dce117e41d6b3e49c10fea0489d. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:41:49.075590 24242 ts_tablet_manager.cc:1173] T 74661dce117e41d6b3e49c10fea0489d P 246f10eee67c4934b4a502a5ae95fbd9: Registered tablet (data state: TABLET_DATA_READY) I0703 01:41:49.076736 24691 ts_tablet_manager.cc:1028] T 74661dce117e41d6b3e49c10fea0489d P 246f10eee67c4934b4a502a5ae95fbd9: Bootstrapping tablet I0703 01:41:49.077283 24691 tablet_bootstrap.cc:427] T 74661dce117e41d6b3e49c10fea0489d P 246f10eee67c4934b4a502a5ae95fbd9: Bootstrap starting. I0703 01:41:49.077673 24486 ts_tablet_manager.cc:1173] T 74661dce117e41d6b3e49c10fea0489d P 201b0b2c1dca433a822d9d7eadaa5a40: Registered tablet (data state: TABLET_DATA_READY) I0703 01:41:49.078414 24692 ts_tablet_manager.cc:1028] T 74661dce117e41d6b3e49c10fea0489d P 201b0b2c1dca433a822d9d7eadaa5a40: Bootstrapping tablet I0703 01:41:49.078691 24119 ts_tablet_manager.cc:1173] T 74661dce117e41d6b3e49c10fea0489d P 4ce55d2fbe26483db71343152f38abb8: Registered tablet (data state: TABLET_DATA_READY) I0703 01:41:49.078914 24692 tablet_bootstrap.cc:427] T 74661dce117e41d6b3e49c10fea0489d P 201b0b2c1dca433a822d9d7eadaa5a40: Bootstrap starting. I0703 01:41:49.080433 24691 tablet_bootstrap.cc:578] T 74661dce117e41d6b3e49c10fea0489d P 246f10eee67c4934b4a502a5ae95fbd9: No blocks or log segments found. Creating new log. I0703 01:41:49.081475 24691 log.cc:525] T 74661dce117e41d6b3e49c10fea0489d P 246f10eee67c4934b4a502a5ae95fbd9: Log is configured to *not* fsync() on all Append() calls I0703 01:41:49.081938 24692 tablet_bootstrap.cc:578] T 74661dce117e41d6b3e49c10fea0489d P 201b0b2c1dca433a822d9d7eadaa5a40: No blocks or log segments found. Creating new log. I0703 01:41:49.083652 24693 ts_tablet_manager.cc:1028] T 74661dce117e41d6b3e49c10fea0489d P 4ce55d2fbe26483db71343152f38abb8: Bootstrapping tablet I0703 01:41:49.083864 24693 tablet_bootstrap.cc:427] T 74661dce117e41d6b3e49c10fea0489d P 4ce55d2fbe26483db71343152f38abb8: Bootstrap starting. I0703 01:41:49.082969 24692 log.cc:525] T 74661dce117e41d6b3e49c10fea0489d P 201b0b2c1dca433a822d9d7eadaa5a40: Log is configured to *not* fsync() on all Append() calls I0703 01:41:49.086336 24692 tablet_bootstrap.cc:427] T 74661dce117e41d6b3e49c10fea0489d P 201b0b2c1dca433a822d9d7eadaa5a40: No bootstrap required, opened a new log I0703 01:41:49.086380 24693 tablet_bootstrap.cc:578] T 74661dce117e41d6b3e49c10fea0489d P 4ce55d2fbe26483db71343152f38abb8: No blocks or log segments found. Creating new log. I0703 01:41:49.086585 24692 ts_tablet_manager.cc:1045] T 74661dce117e41d6b3e49c10fea0489d P 201b0b2c1dca433a822d9d7eadaa5a40: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.001s I0703 01:41:49.087070 24691 tablet_bootstrap.cc:427] T 74661dce117e41d6b3e49c10fea0489d P 246f10eee67c4934b4a502a5ae95fbd9: No bootstrap required, opened a new log I0703 01:41:49.087160 24693 log.cc:525] T 74661dce117e41d6b3e49c10fea0489d P 4ce55d2fbe26483db71343152f38abb8: Log is configured to *not* fsync() on all Append() calls I0703 01:41:49.087285 24691 ts_tablet_manager.cc:1045] T 74661dce117e41d6b3e49c10fea0489d P 246f10eee67c4934b4a502a5ae95fbd9: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s I0703 01:41:49.093648 24693 tablet_bootstrap.cc:427] T 74661dce117e41d6b3e49c10fea0489d P 4ce55d2fbe26483db71343152f38abb8: No bootstrap required, opened a new log I0703 01:41:49.093827 24693 ts_tablet_manager.cc:1045] T 74661dce117e41d6b3e49c10fea0489d P 4ce55d2fbe26483db71343152f38abb8: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.006s I0703 01:41:49.096429 24692 raft_consensus.cc:325] T 74661dce117e41d6b3e49c10fea0489d P 201b0b2c1dca433a822d9d7eadaa5a40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201b0b2c1dca433a822d9d7eadaa5a40" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 44176 } } peers { permanent_uuid: "4ce55d2fbe26483db71343152f38abb8" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 39726 } } peers { permanent_uuid: "246f10eee67c4934b4a502a5ae95fbd9" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59210 } } I0703 01:41:49.096663 24691 raft_consensus.cc:325] T 74661dce117e41d6b3e49c10fea0489d P 246f10eee67c4934b4a502a5ae95fbd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201b0b2c1dca433a822d9d7eadaa5a40" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 44176 } } peers { permanent_uuid: "4ce55d2fbe26483db71343152f38abb8" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 39726 } } peers { permanent_uuid: "246f10eee67c4934b4a502a5ae95fbd9" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59210 } } I0703 01:41:49.096938 24692 raft_consensus.cc:351] T 74661dce117e41d6b3e49c10fea0489d P 201b0b2c1dca433a822d9d7eadaa5a40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:41:49.097041 24691 raft_consensus.cc:351] T 74661dce117e41d6b3e49c10fea0489d P 246f10eee67c4934b4a502a5ae95fbd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:41:49.097100 24692 raft_consensus.cc:628] T 74661dce117e41d6b3e49c10fea0489d P 201b0b2c1dca433a822d9d7eadaa5a40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 201b0b2c1dca433a822d9d7eadaa5a40, State: Initialized, Role: FOLLOWER I0703 01:41:49.097131 24691 raft_consensus.cc:628] T 74661dce117e41d6b3e49c10fea0489d P 246f10eee67c4934b4a502a5ae95fbd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 246f10eee67c4934b4a502a5ae95fbd9, State: Initialized, Role: FOLLOWER I0703 01:41:49.097625 24692 consensus_queue.cc:227] T 74661dce117e41d6b3e49c10fea0489d P 201b0b2c1dca433a822d9d7eadaa5a40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201b0b2c1dca433a822d9d7eadaa5a40" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 44176 } } peers { permanent_uuid: "4ce55d2fbe26483db71343152f38abb8" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 39726 } } peers { permanent_uuid: "246f10eee67c4934b4a502a5ae95fbd9" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59210 } } I0703 01:41:49.097640 24691 consensus_queue.cc:227] T 74661dce117e41d6b3e49c10fea0489d P 246f10eee67c4934b4a502a5ae95fbd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201b0b2c1dca433a822d9d7eadaa5a40" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 44176 } } peers { permanent_uuid: "4ce55d2fbe26483db71343152f38abb8" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 39726 } } peers { permanent_uuid: "246f10eee67c4934b4a502a5ae95fbd9" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59210 } } I0703 01:41:49.100275 24691 ts_tablet_manager.cc:1073] T 74661dce117e41d6b3e49c10fea0489d P 246f10eee67c4934b4a502a5ae95fbd9: Time spent starting tablet: real 0.013s user 0.007s sys 0.005s I0703 01:41:49.100317 24693 raft_consensus.cc:325] T 74661dce117e41d6b3e49c10fea0489d P 4ce55d2fbe26483db71343152f38abb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201b0b2c1dca433a822d9d7eadaa5a40" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 44176 } } peers { permanent_uuid: "4ce55d2fbe26483db71343152f38abb8" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 39726 } } peers { permanent_uuid: "246f10eee67c4934b4a502a5ae95fbd9" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59210 } } I0703 01:41:49.100751 24693 raft_consensus.cc:351] T 74661dce117e41d6b3e49c10fea0489d P 4ce55d2fbe26483db71343152f38abb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:41:49.100836 24693 raft_consensus.cc:628] T 74661dce117e41d6b3e49c10fea0489d P 4ce55d2fbe26483db71343152f38abb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ce55d2fbe26483db71343152f38abb8, State: Initialized, Role: FOLLOWER I0703 01:41:49.101163 24549 heartbeater.cc:467] Master 127.5.240.254:38173 was elected leader, sending a full tablet report... I0703 01:41:49.101330 24693 consensus_queue.cc:227] T 74661dce117e41d6b3e49c10fea0489d P 4ce55d2fbe26483db71343152f38abb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201b0b2c1dca433a822d9d7eadaa5a40" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 44176 } } peers { permanent_uuid: "4ce55d2fbe26483db71343152f38abb8" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 39726 } } peers { permanent_uuid: "246f10eee67c4934b4a502a5ae95fbd9" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59210 } } I0703 01:41:49.102360 24692 ts_tablet_manager.cc:1073] T 74661dce117e41d6b3e49c10fea0489d P 201b0b2c1dca433a822d9d7eadaa5a40: Time spent starting tablet: real 0.016s user 0.009s sys 0.004s I0703 01:41:49.104092 24693 ts_tablet_manager.cc:1073] T 74661dce117e41d6b3e49c10fea0489d P 4ce55d2fbe26483db71343152f38abb8: Time spent starting tablet: real 0.010s user 0.006s sys 0.002s W0703 01:41:49.183791 24550 tablet.cc:1678] T 74661dce117e41d6b3e49c10fea0489d P 201b0b2c1dca433a822d9d7eadaa5a40: Can't schedule compaction. Clean time has not been advanced past its initial value. W0703 01:41:49.202281 24184 tablet.cc:1678] T 74661dce117e41d6b3e49c10fea0489d P 4ce55d2fbe26483db71343152f38abb8: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:41:49.216640 24697 raft_consensus.cc:457] T 74661dce117e41d6b3e49c10fea0489d P 246f10eee67c4934b4a502a5ae95fbd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0703 01:41:49.216747 24697 raft_consensus.cc:2730] T 74661dce117e41d6b3e49c10fea0489d P 246f10eee67c4934b4a502a5ae95fbd9: Snoozing failure detection for 2.217s (starting election) I0703 01:41:49.216852 24697 raft_consensus.cc:479] T 74661dce117e41d6b3e49c10fea0489d P 246f10eee67c4934b4a502a5ae95fbd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201b0b2c1dca433a822d9d7eadaa5a40" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 44176 } } peers { permanent_uuid: "4ce55d2fbe26483db71343152f38abb8" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 39726 } } peers { permanent_uuid: "246f10eee67c4934b4a502a5ae95fbd9" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59210 } } I0703 01:41:49.217574 24697 leader_election.cc:243] T 74661dce117e41d6b3e49c10fea0489d P 246f10eee67c4934b4a502a5ae95fbd9 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 201b0b2c1dca433a822d9d7eadaa5a40 (127.5.240.196:44176) I0703 01:41:49.217891 24697 leader_election.cc:243] T 74661dce117e41d6b3e49c10fea0489d P 246f10eee67c4934b4a502a5ae95fbd9 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 4ce55d2fbe26483db71343152f38abb8 (127.5.240.193:39726) I0703 01:41:49.229188 24139 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "74661dce117e41d6b3e49c10fea0489d" candidate_uuid: "246f10eee67c4934b4a502a5ae95fbd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ce55d2fbe26483db71343152f38abb8" is_pre_election: true I0703 01:41:49.229262 24506 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "74661dce117e41d6b3e49c10fea0489d" candidate_uuid: "246f10eee67c4934b4a502a5ae95fbd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "201b0b2c1dca433a822d9d7eadaa5a40" is_pre_election: true I0703 01:41:49.229408 24139 raft_consensus.cc:2730] T 74661dce117e41d6b3e49c10fea0489d P 4ce55d2fbe26483db71343152f38abb8: Snoozing failure detection for 1.795s (vote granted) I0703 01:41:49.229481 24139 raft_consensus.cc:2243] T 74661dce117e41d6b3e49c10fea0489d P 4ce55d2fbe26483db71343152f38abb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 246f10eee67c4934b4a502a5ae95fbd9 in term 0. I0703 01:41:49.229492 24506 raft_consensus.cc:2730] T 74661dce117e41d6b3e49c10fea0489d P 201b0b2c1dca433a822d9d7eadaa5a40: Snoozing failure detection for 1.634s (vote granted) I0703 01:41:49.229604 24506 raft_consensus.cc:2243] T 74661dce117e41d6b3e49c10fea0489d P 201b0b2c1dca433a822d9d7eadaa5a40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 246f10eee67c4934b4a502a5ae95fbd9 in term 0. I0703 01:41:49.230082 24195 leader_election.cc:379] T 74661dce117e41d6b3e49c10fea0489d P 246f10eee67c4934b4a502a5ae95fbd9 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 4ce55d2fbe26483db71343152f38abb8 (127.5.240.193:39726) I0703 01:41:49.230183 24195 leader_election.cc:270] T 74661dce117e41d6b3e49c10fea0489d P 246f10eee67c4934b4a502a5ae95fbd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0703 01:41:49.230299 24193 leader_election.cc:379] T 74661dce117e41d6b3e49c10fea0489d P 246f10eee67c4934b4a502a5ae95fbd9 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 201b0b2c1dca433a822d9d7eadaa5a40 (127.5.240.196:44176) I0703 01:41:49.230489 24697 raft_consensus.cc:2730] T 74661dce117e41d6b3e49c10fea0489d P 246f10eee67c4934b4a502a5ae95fbd9: Snoozing failure detection for 2.048s (election complete) I0703 01:41:49.230629 24697 raft_consensus.cc:2537] T 74661dce117e41d6b3e49c10fea0489d P 246f10eee67c4934b4a502a5ae95fbd9 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0703 01:41:49.230690 24697 raft_consensus.cc:457] T 74661dce117e41d6b3e49c10fea0489d P 246f10eee67c4934b4a502a5ae95fbd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0703 01:41:49.230748 24697 raft_consensus.cc:2730] T 74661dce117e41d6b3e49c10fea0489d P 246f10eee67c4934b4a502a5ae95fbd9: Snoozing failure detection for 2.039s (starting election) I0703 01:41:49.230808 24697 raft_consensus.cc:2782] T 74661dce117e41d6b3e49c10fea0489d P 246f10eee67c4934b4a502a5ae95fbd9 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:41:49.232250 24697 raft_consensus.cc:479] T 74661dce117e41d6b3e49c10fea0489d P 246f10eee67c4934b4a502a5ae95fbd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201b0b2c1dca433a822d9d7eadaa5a40" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 44176 } } peers { permanent_uuid: "4ce55d2fbe26483db71343152f38abb8" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 39726 } } peers { permanent_uuid: "246f10eee67c4934b4a502a5ae95fbd9" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59210 } } I0703 01:41:49.232611 24697 leader_election.cc:243] T 74661dce117e41d6b3e49c10fea0489d P 246f10eee67c4934b4a502a5ae95fbd9 [CANDIDATE]: Term 1 election: Requesting vote from peer 201b0b2c1dca433a822d9d7eadaa5a40 (127.5.240.196:44176) I0703 01:41:49.232784 24697 leader_election.cc:243] T 74661dce117e41d6b3e49c10fea0489d P 246f10eee67c4934b4a502a5ae95fbd9 [CANDIDATE]: Term 1 election: Requesting vote from peer 4ce55d2fbe26483db71343152f38abb8 (127.5.240.193:39726) I0703 01:41:49.233333 24506 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "74661dce117e41d6b3e49c10fea0489d" candidate_uuid: "246f10eee67c4934b4a502a5ae95fbd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "201b0b2c1dca433a822d9d7eadaa5a40" I0703 01:41:49.233409 24139 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "74661dce117e41d6b3e49c10fea0489d" candidate_uuid: "246f10eee67c4934b4a502a5ae95fbd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ce55d2fbe26483db71343152f38abb8" I0703 01:41:49.233456 24506 raft_consensus.cc:2782] T 74661dce117e41d6b3e49c10fea0489d P 201b0b2c1dca433a822d9d7eadaa5a40 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:41:49.233513 24139 raft_consensus.cc:2782] T 74661dce117e41d6b3e49c10fea0489d P 4ce55d2fbe26483db71343152f38abb8 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:41:49.233549 24506 raft_consensus.cc:2730] T 74661dce117e41d6b3e49c10fea0489d P 201b0b2c1dca433a822d9d7eadaa5a40: Snoozing failure detection for 1.640s (vote granted) I0703 01:41:49.233592 24139 raft_consensus.cc:2730] T 74661dce117e41d6b3e49c10fea0489d P 4ce55d2fbe26483db71343152f38abb8: Snoozing failure detection for 2.075s (vote granted) I0703 01:41:49.234850 24506 raft_consensus.cc:2243] T 74661dce117e41d6b3e49c10fea0489d P 201b0b2c1dca433a822d9d7eadaa5a40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 246f10eee67c4934b4a502a5ae95fbd9 in term 1. I0703 01:41:49.234932 24139 raft_consensus.cc:2243] T 74661dce117e41d6b3e49c10fea0489d P 4ce55d2fbe26483db71343152f38abb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 246f10eee67c4934b4a502a5ae95fbd9 in term 1. I0703 01:41:49.235280 24193 leader_election.cc:379] T 74661dce117e41d6b3e49c10fea0489d P 246f10eee67c4934b4a502a5ae95fbd9 [CANDIDATE]: Term 1 election: Vote granted by peer 201b0b2c1dca433a822d9d7eadaa5a40 (127.5.240.196:44176) I0703 01:41:49.235366 24195 leader_election.cc:379] T 74661dce117e41d6b3e49c10fea0489d P 246f10eee67c4934b4a502a5ae95fbd9 [CANDIDATE]: Term 1 election: Vote granted by peer 4ce55d2fbe26483db71343152f38abb8 (127.5.240.193:39726) I0703 01:41:49.235422 24195 leader_election.cc:270] T 74661dce117e41d6b3e49c10fea0489d P 246f10eee67c4934b4a502a5ae95fbd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:41:49.235577 24697 raft_consensus.cc:2730] T 74661dce117e41d6b3e49c10fea0489d P 246f10eee67c4934b4a502a5ae95fbd9: Snoozing failure detection for 1.926s (election complete) I0703 01:41:49.235667 24697 raft_consensus.cc:2537] T 74661dce117e41d6b3e49c10fea0489d P 246f10eee67c4934b4a502a5ae95fbd9 [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:41:49.236497 24697 raft_consensus.cc:593] T 74661dce117e41d6b3e49c10fea0489d P 246f10eee67c4934b4a502a5ae95fbd9 [term 1 LEADER]: Becoming Leader. State: Replica: 246f10eee67c4934b4a502a5ae95fbd9, State: Running, Role: LEADER I0703 01:41:49.236937 24697 consensus_queue.cc:204] T 74661dce117e41d6b3e49c10fea0489d P 246f10eee67c4934b4a502a5ae95fbd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201b0b2c1dca433a822d9d7eadaa5a40" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 44176 } } peers { permanent_uuid: "4ce55d2fbe26483db71343152f38abb8" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 39726 } } peers { permanent_uuid: "246f10eee67c4934b4a502a5ae95fbd9" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59210 } } I0703 01:41:49.240397 24018 catalog_manager.cc:3798] T 74661dce117e41d6b3e49c10fea0489d P 246f10eee67c4934b4a502a5ae95fbd9 reported cstate change: term changed from 0 to 1, leader changed from to 246f10eee67c4934b4a502a5ae95fbd9 (127.5.240.194). New cstate: current_term: 1 leader_uuid: "246f10eee67c4934b4a502a5ae95fbd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "201b0b2c1dca433a822d9d7eadaa5a40" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 44176 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ce55d2fbe26483db71343152f38abb8" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 39726 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "246f10eee67c4934b4a502a5ae95fbd9" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59210 } health_report { overall_health: HEALTHY } } } W0703 01:41:49.268548 24306 tablet.cc:1678] T 74661dce117e41d6b3e49c10fea0489d P 246f10eee67c4934b4a502a5ae95fbd9: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:41:49.285764 6083 external_mini_cluster.cc:490] 5 TS(s) registered with all masters I0703 01:41:49.286877 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 4ce55d2fbe26483db71343152f38abb8 to finish bootstrapping I0703 01:41:49.292199 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 246f10eee67c4934b4a502a5ae95fbd9 to finish bootstrapping I0703 01:41:49.297718 6083 ts_itest-base.cc:233] Waiting for 0 tablets on tserver e3c0ca1fc3254b5fa1acf121412f4b1f to finish bootstrapping I0703 01:41:49.304250 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 201b0b2c1dca433a822d9d7eadaa5a40 to finish bootstrapping I0703 01:41:49.308786 6083 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 55c0a44b280e4015a22c8302291619c6 to finish bootstrapping I0703 01:41:49.316207 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 24053 I0703 01:41:49.316823 24179 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:41:49.359874 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 24053 W0703 01:41:49.368317 24195 connection.cc:512] client connection to 127.5.240.193:39726 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:39726): Connection reset by peer (error 104) W0703 01:41:49.368505 24001 connection.cc:512] server connection from 127.5.240.193:60774 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:60774): Connection reset by peer (error 104) I0703 01:41:49.376145 24427 heartbeater.cc:467] Master 127.5.240.254:38173 was elected leader, sending a full tablet report... I0703 01:41:49.654003 24506 raft_consensus.cc:1081] T 74661dce117e41d6b3e49c10fea0489d P 201b0b2c1dca433a822d9d7eadaa5a40 [term 1 FOLLOWER]: Refusing update from remote peer 246f10eee67c4934b4a502a5ae95fbd9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0703 01:41:49.654821 24697 consensus_queue.cc:962] T 74661dce117e41d6b3e49c10fea0489d P 246f10eee67c4934b4a502a5ae95fbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "201b0b2c1dca433a822d9d7eadaa5a40" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 44176 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s W0703 01:41:49.660986 24195 consensus_peers.cc:433] T 74661dce117e41d6b3e49c10fea0489d P 246f10eee67c4934b4a502a5ae95fbd9 -> Peer 4ce55d2fbe26483db71343152f38abb8 (127.5.240.193:39726): Couldn't send request to peer 4ce55d2fbe26483db71343152f38abb8 for tablet 74661dce117e41d6b3e49c10fea0489d. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.193:39726: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. Per-server replica distribution summary: Statistic | Value -----------------------+---------- Minimum Replica Count | 0 Maximum Replica Count | 1 Average Replica Count | 0.500000 Per-table replica distribution summary: Replica Skew | Value --------------+---------- Minimum | 1 Maximum | 1 Average | 1.000000 I0703 01:41:49.828666 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 24183 I0703 01:41:49.831748 24302 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:41:49.876119 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 24183 W0703 01:41:49.884658 24002 connection.cc:512] server connection from 127.5.240.194:38329 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:38329): Connection reset by peer (error 104) W0703 01:41:49.885109 24438 connection.cc:512] server connection from 127.5.240.194:36654 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:36654): Connection reset by peer (error 104) I0703 01:41:49.885532 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 24308 I0703 01:41:49.886282 24424 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:41:49.925725 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 24308 W0703 01:41:49.933642 24002 connection.cc:512] server connection from 127.5.240.195:55246 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.195:55246): Connection reset by peer (error 104) I0703 01:41:49.933979 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 24430 I0703 01:41:49.934679 24546 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:41:49.977267 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 24430 I0703 01:41:49.985484 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 24552 W0703 01:41:49.985499 24000 connection.cc:512] server connection from 127.5.240.196:36154 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.196:36154): Connection reset by peer (error 104) I0703 01:41:49.986227 24668 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:41:50.024199 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 24552 W0703 01:41:50.032367 24000 connection.cc:512] server connection from 127.5.240.197:42555 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.197:42555): Connection reset by peer (error 104) I0703 01:41:50.032400 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master pid 23992 I0703 01:41:50.033074 24049 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:41:50.095793 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master with pid 23992 [ OK ] RebalanceStartCriteriaTest.TabletServerIsDown/1 (3077 ms) [----------] 2 tests from RebalanceStartCriteriaTest (6095 ms total) [----------] 8 tests from RebalanceParamTest [ RUN ] RebalanceParamTest.Rebalance/0 I0703 01:41:50.118599 6083 test_util.cc:208] Using random seed: 1224908474 I0703 01:41:50.120409 6083 ts_itest-base.cc:113] Starting cluster with: I0703 01:41:50.120434 6083 ts_itest-base.cc:114] -------------- I0703 01:41:50.120451 6083 ts_itest-base.cc:115] 7 tablet servers I0703 01:41:50.120468 6083 ts_itest-base.cc:116] 1 replicas per TS I0703 01:41:50.120483 6083 ts_itest-base.cc:117] -------------- I0703 01:41:50.121675 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:57131 --webserver_interface=127.5.240.254 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --allow_unsafe_replication_factor --raft_prepare_replacement_before_eviction=false --tserver_unresponsive_timeout_ms=3000 --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:41:50.305186 24774 flags.cc:406] Enabled unsafe flag: --allow_unsafe_replication_factor=true W0703 01:41:50.306185 24774 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:41:50.306246 24774 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:41:50.308445 24774 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0703 01:41:50.308504 24774 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0703 01:41:50.308538 24774 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:41:50.308571 24774 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0703 01:41:50.308625 24774 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0703 01:41:50.316761 24774 master_main.cc:73] Master server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --allow_unsafe_replication_factor=true --ipki_ca_key_size=1024 --tserver_unresponsive_timeout_ms=3000 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:57131 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.5.240.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:41:50.322391 24774 master_main.cc:80] Initializing master server... I0703 01:41:50.323092 24774 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 208708us I0703 01:41:50.324105 24774 fs_manager.cc:261] Metadata directory not provided I0703 01:41:50.324159 24774 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:41:50.324395 24774 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:41:50.324452 24774 server_base.cc:433] Attempting to create new FS layout instead I0703 01:41:50.328567 24774 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/instance: uuid: "0ed28bce615e467daedd680797177631" format_stamp: "Formatted at 2018-07-03 08:41:50 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:50.329455 24774 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "0ed28bce615e467daedd680797177631" format_stamp: "Formatted at 2018-07-03 08:41:50 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:50.337071 24774 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.004s sys 0.002s I0703 01:41:50.338258 24774 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:41:50.338657 24774 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:41:50.342278 24774 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0703 01:41:50.342329 24774 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal uuid: "0ed28bce615e467daedd680797177631" format_stamp: "Formatted at 2018-07-03 08:41:50 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:50.342429 24774 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:41:50.365247 24774 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:41:50.366688 24774 master_main.cc:83] Starting Master server... I0703 01:41:50.404300 24774 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.254:57131 I0703 01:41:50.404736 24774 webserver.cc:175] Starting webserver on 127.5.240.254:0 I0703 01:41:50.404776 24774 webserver.cc:186] Document root disabled I0703 01:41:50.405510 24774 webserver.cc:313] Webserver started. Bound to: http://127.5.240.254:49056/ I0703 01:41:50.406112 24774 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb I0703 01:41:50.409250 24833 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:41:50.416230 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master as pid 24774 I0703 01:41:50.416829 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.5.240.193:0 --local_ip_for_outbound_sockets=127.5.240.193 --webserver_interface=127.5.240.193 --webserver_port=0 --tserver_master_addrs=127.5.240.254:57131 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0703 01:41:50.420728 24833 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 0ed28bce615e467daedd680797177631: Bootstrap starting. I0703 01:41:50.422741 24833 tablet_bootstrap.cc:578] T 00000000000000000000000000000000 P 0ed28bce615e467daedd680797177631: No blocks or log segments found. Creating new log. I0703 01:41:50.423693 24833 log.cc:525] T 00000000000000000000000000000000 P 0ed28bce615e467daedd680797177631: Log is configured to *not* fsync() on all Append() calls I0703 01:41:50.425987 24833 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 0ed28bce615e467daedd680797177631: No bootstrap required, opened a new log I0703 01:41:50.432305 24833 raft_consensus.cc:325] T 00000000000000000000000000000000 P 0ed28bce615e467daedd680797177631 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ed28bce615e467daedd680797177631" member_type: VOTER } I0703 01:41:50.432538 24833 raft_consensus.cc:351] T 00000000000000000000000000000000 P 0ed28bce615e467daedd680797177631 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:41:50.432606 24833 raft_consensus.cc:628] T 00000000000000000000000000000000 P 0ed28bce615e467daedd680797177631 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed28bce615e467daedd680797177631, State: Initialized, Role: FOLLOWER I0703 01:41:50.433080 24833 consensus_queue.cc:227] T 00000000000000000000000000000000 P 0ed28bce615e467daedd680797177631 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ed28bce615e467daedd680797177631" member_type: VOTER } I0703 01:41:50.433228 24833 raft_consensus.cc:365] T 00000000000000000000000000000000 P 0ed28bce615e467daedd680797177631 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:41:50.433316 24833 raft_consensus.cc:457] T 00000000000000000000000000000000 P 0ed28bce615e467daedd680797177631 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:41:50.433400 24833 raft_consensus.cc:2730] T 00000000000000000000000000000000 P 0ed28bce615e467daedd680797177631: Snoozing failure detection for 2.014s (starting election) I0703 01:41:50.433491 24833 raft_consensus.cc:2782] T 00000000000000000000000000000000 P 0ed28bce615e467daedd680797177631 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:41:50.434937 24833 raft_consensus.cc:479] T 00000000000000000000000000000000 P 0ed28bce615e467daedd680797177631 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ed28bce615e467daedd680797177631" member_type: VOTER } I0703 01:41:50.435309 24833 leader_election.cc:270] T 00000000000000000000000000000000 P 0ed28bce615e467daedd680797177631 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:41:50.436842 24836 raft_consensus.cc:2730] T 00000000000000000000000000000000 P 0ed28bce615e467daedd680797177631: Snoozing failure detection for 1.726s (election complete) I0703 01:41:50.437096 24836 raft_consensus.cc:2537] T 00000000000000000000000000000000 P 0ed28bce615e467daedd680797177631 [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:41:50.438629 24836 raft_consensus.cc:593] T 00000000000000000000000000000000 P 0ed28bce615e467daedd680797177631 [term 1 LEADER]: Becoming Leader. State: Replica: 0ed28bce615e467daedd680797177631, State: Running, Role: LEADER I0703 01:41:50.439019 24836 consensus_queue.cc:204] T 00000000000000000000000000000000 P 0ed28bce615e467daedd680797177631 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ed28bce615e467daedd680797177631" member_type: VOTER } I0703 01:41:50.441373 24833 sys_catalog.cc:337] T 00000000000000000000000000000000 P 0ed28bce615e467daedd680797177631 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "0ed28bce615e467daedd680797177631" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ed28bce615e467daedd680797177631" member_type: VOTER } } I0703 01:41:50.441805 24833 sys_catalog.cc:340] T 00000000000000000000000000000000 P 0ed28bce615e467daedd680797177631 [sys.catalog]: This master's current role is: LEADER I0703 01:41:50.441606 24838 sys_catalog.cc:337] T 00000000000000000000000000000000 P 0ed28bce615e467daedd680797177631 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ed28bce615e467daedd680797177631. Latest consensus state: current_term: 1 leader_uuid: "0ed28bce615e467daedd680797177631" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ed28bce615e467daedd680797177631" member_type: VOTER } } I0703 01:41:50.441608 24837 sys_catalog.cc:337] T 00000000000000000000000000000000 P 0ed28bce615e467daedd680797177631 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ed28bce615e467daedd680797177631" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ed28bce615e467daedd680797177631" member_type: VOTER } } I0703 01:41:50.442684 24838 sys_catalog.cc:340] T 00000000000000000000000000000000 P 0ed28bce615e467daedd680797177631 [sys.catalog]: This master's current role is: LEADER I0703 01:41:50.442916 24837 sys_catalog.cc:340] T 00000000000000000000000000000000 P 0ed28bce615e467daedd680797177631 [sys.catalog]: This master's current role is: LEADER I0703 01:41:50.445423 24833 sys_catalog.cc:424] T 00000000000000000000000000000000 P 0ed28bce615e467daedd680797177631 [sys.catalog]: configured and running, proceeding with master startup. I0703 01:41:50.445763 24840 catalog_manager.cc:1006] Loading table and tablet metadata into memory... I0703 01:41:50.447237 24774 master_main.cc:86] Master server successfully started. I0703 01:41:50.452277 24840 catalog_manager.cc:1017] Initializing Kudu internal certificate authority... I0703 01:41:50.493706 24840 catalog_manager.cc:904] Generated new certificate authority record I0703 01:41:50.495682 24840 catalog_manager.cc:1026] Loading token signing keys... I0703 01:41:50.503614 24840 catalog_manager.cc:4102] T 00000000000000000000000000000000 P 0ed28bce615e467daedd680797177631: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:41:50.581749 24834 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:41:50.582767 24834 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:41:50.584741 24834 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0703 01:41:50.584806 24834 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:41:50.584918 24834 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.193 I0703 01:41:50.591156 24834 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.5.240.193 --webserver_port=0 --tserver_master_addrs=127.5.240.254:57131 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.193 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:41:50.597653 24834 tablet_server_main.cc:85] Initializing tablet server... I0703 01:41:50.599288 24834 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 208708us I0703 01:41:50.599880 24834 fs_manager.cc:261] Metadata directory not provided I0703 01:41:50.599949 24834 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:41:50.600548 24834 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:41:50.600642 24834 server_base.cc:433] Attempting to create new FS layout instead I0703 01:41:50.604755 24834 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "dfd7ad6293204fc89f9e72436796698b" format_stamp: "Formatted at 2018-07-03 08:41:50 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:50.605208 24834 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "dfd7ad6293204fc89f9e72436796698b" format_stamp: "Formatted at 2018-07-03 08:41:50 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:50.717783 24834 fs_manager.cc:501] Time spent creating directory manager: real 0.112s user 0.006s sys 0.002s I0703 01:41:50.719049 24834 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:41:50.719476 24834 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:41:50.723026 24834 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0703 01:41:50.723079 24834 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal uuid: "dfd7ad6293204fc89f9e72436796698b" format_stamp: "Formatted at 2018-07-03 08:41:50 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:50.723222 24834 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:41:50.768417 24834 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:41:50.769978 24834 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:41:50.770344 24834 tablet_server_main.cc:90] Starting tablet server... I0703 01:41:50.855338 24834 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.193:43198 I0703 01:41:50.855777 24834 webserver.cc:175] Starting webserver on 127.5.240.193:0 I0703 01:41:50.855823 24834 webserver.cc:186] Document root disabled I0703 01:41:50.856986 24834 webserver.cc:313] Webserver started. Bound to: http://127.5.240.193:50416/ I0703 01:41:50.857522 24834 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb I0703 01:41:50.859964 24834 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:41:50.865713 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 24834 I0703 01:41:50.866389 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.5.240.194:0 --local_ip_for_outbound_sockets=127.5.240.194 --webserver_interface=127.5.240.194 --webserver_port=0 --tserver_master_addrs=127.5.240.254:57131 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0703 01:41:50.873852 24963 heartbeater.cc:346] Connected to a master server at 127.5.240.254:57131 I0703 01:41:50.874009 24963 heartbeater.cc:426] Registering TS with master... I0703 01:41:50.874377 24963 heartbeater.cc:475] Master 127.5.240.254:57131 requested a full tablet report, sending... I0703 01:41:50.875668 24800 ts_manager.cc:98] Registered new tserver with Master: dfd7ad6293204fc89f9e72436796698b (127.5.240.193:43198) I0703 01:41:50.877478 24800 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.193:41273 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:41:51.028808 24967 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:41:51.029804 24967 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:41:51.031837 24967 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0703 01:41:51.031886 24967 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:41:51.031971 24967 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.194 I0703 01:41:51.038468 24967 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.194:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.5.240.194 --webserver_port=0 --tserver_master_addrs=127.5.240.254:57131 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.194 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:41:51.045148 24967 tablet_server_main.cc:85] Initializing tablet server... I0703 01:41:51.046689 24967 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 209208us I0703 01:41:51.047227 24967 fs_manager.cc:261] Metadata directory not provided I0703 01:41:51.047286 24967 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:41:51.047495 24967 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:41:51.047572 24967 server_base.cc:433] Attempting to create new FS layout instead I0703 01:41:51.051698 24967 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "8ec1760d41f842f398d90e47ab52973b" format_stamp: "Formatted at 2018-07-03 08:41:51 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:51.052189 24967 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "8ec1760d41f842f398d90e47ab52973b" format_stamp: "Formatted at 2018-07-03 08:41:51 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:51.059506 24967 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s I0703 01:41:51.060696 24967 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:41:51.061049 24967 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:41:51.064631 24967 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s I0703 01:41:51.064679 24967 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal uuid: "8ec1760d41f842f398d90e47ab52973b" format_stamp: "Formatted at 2018-07-03 08:41:51 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:51.064797 24967 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:41:51.114434 24967 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:41:51.116328 24967 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:41:51.116684 24967 tablet_server_main.cc:90] Starting tablet server... I0703 01:41:51.204941 24967 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.194:46721 I0703 01:41:51.205317 24967 webserver.cc:175] Starting webserver on 127.5.240.194:0 I0703 01:41:51.205354 24967 webserver.cc:186] Document root disabled I0703 01:41:51.206013 24967 webserver.cc:313] Webserver started. Bound to: http://127.5.240.194:57166/ I0703 01:41:51.206531 24967 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb I0703 01:41:51.208804 24967 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:41:51.214085 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 24967 I0703 01:41:51.214957 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.5.240.195:0 --local_ip_for_outbound_sockets=127.5.240.195 --webserver_interface=127.5.240.195 --webserver_port=0 --tserver_master_addrs=127.5.240.254:57131 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0703 01:41:51.225267 25086 heartbeater.cc:346] Connected to a master server at 127.5.240.254:57131 I0703 01:41:51.225417 25086 heartbeater.cc:426] Registering TS with master... I0703 01:41:51.225790 25086 heartbeater.cc:475] Master 127.5.240.254:57131 requested a full tablet report, sending... I0703 01:41:51.226725 24800 ts_manager.cc:98] Registered new tserver with Master: 8ec1760d41f842f398d90e47ab52973b (127.5.240.194:46721) I0703 01:41:51.227555 24800 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.194:33164 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:41:51.378145 25089 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:41:51.379148 25089 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:41:51.381157 25089 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0703 01:41:51.381206 25089 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:41:51.381286 25089 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.195 I0703 01:41:51.387434 25089 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.195:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.5.240.195 --webserver_port=0 --tserver_master_addrs=127.5.240.254:57131 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.195 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:41:51.393965 25089 tablet_server_main.cc:85] Initializing tablet server... I0703 01:41:51.395452 25089 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 209208us I0703 01:41:51.396003 25089 fs_manager.cc:261] Metadata directory not provided I0703 01:41:51.396066 25089 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:41:51.396679 25089 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:41:51.396742 25089 server_base.cc:433] Attempting to create new FS layout instead I0703 01:41:51.400904 25089 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "cff51d1a4baa4cd0a1625e1d7db94f2c" format_stamp: "Formatted at 2018-07-03 08:41:51 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:51.401422 25089 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "cff51d1a4baa4cd0a1625e1d7db94f2c" format_stamp: "Formatted at 2018-07-03 08:41:51 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:51.409147 25089 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.003s sys 0.003s I0703 01:41:51.410349 25089 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:41:51.410755 25089 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:41:51.413918 25089 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0703 01:41:51.413964 25089 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal uuid: "cff51d1a4baa4cd0a1625e1d7db94f2c" format_stamp: "Formatted at 2018-07-03 08:41:51 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:51.414083 25089 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:41:51.478857 25089 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:41:51.480695 25089 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:41:51.481010 25089 tablet_server_main.cc:90] Starting tablet server... I0703 01:41:51.566354 25089 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.195:38199 I0703 01:41:51.566750 25089 webserver.cc:175] Starting webserver on 127.5.240.195:0 I0703 01:41:51.566788 25089 webserver.cc:186] Document root disabled I0703 01:41:51.567457 25089 webserver.cc:313] Webserver started. Bound to: http://127.5.240.195:51624/ I0703 01:41:51.567997 25089 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb I0703 01:41:51.570366 25089 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:41:51.574349 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 25089 I0703 01:41:51.575212 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.5.240.196:0 --local_ip_for_outbound_sockets=127.5.240.196 --webserver_interface=127.5.240.196 --webserver_port=0 --tserver_master_addrs=127.5.240.254:57131 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0703 01:41:51.583173 25208 heartbeater.cc:346] Connected to a master server at 127.5.240.254:57131 I0703 01:41:51.583315 25208 heartbeater.cc:426] Registering TS with master... I0703 01:41:51.583686 25208 heartbeater.cc:475] Master 127.5.240.254:57131 requested a full tablet report, sending... I0703 01:41:51.584659 24800 ts_manager.cc:98] Registered new tserver with Master: cff51d1a4baa4cd0a1625e1d7db94f2c (127.5.240.195:38199) I0703 01:41:51.585484 24800 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.195:36555 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:41:51.738863 25211 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:41:51.739868 25211 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:41:51.741868 25211 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0703 01:41:51.741917 25211 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:41:51.742007 25211 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.196 I0703 01:41:51.748658 25211 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.196:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb --webserver_interface=127.5.240.196 --webserver_port=0 --tserver_master_addrs=127.5.240.254:57131 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.196 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:41:51.755228 25211 tablet_server_main.cc:85] Initializing tablet server... I0703 01:41:51.756749 25211 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 209208us I0703 01:41:51.757293 25211 fs_manager.cc:261] Metadata directory not provided I0703 01:41:51.757351 25211 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:41:51.757575 25211 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:41:51.757643 25211 server_base.cc:433] Attempting to create new FS layout instead I0703 01:41:51.761651 25211 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/instance: uuid: "ca6ae9994a78460ca6c4ae4ec3f4e9b5" format_stamp: "Formatted at 2018-07-03 08:41:51 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:51.762136 25211 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal/instance: uuid: "ca6ae9994a78460ca6c4ae4ec3f4e9b5" format_stamp: "Formatted at 2018-07-03 08:41:51 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:51.769699 25211 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.004s sys 0.003s I0703 01:41:51.771075 25211 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:41:51.771426 25211 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:41:51.774646 25211 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0703 01:41:51.774695 25211 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal uuid: "ca6ae9994a78460ca6c4ae4ec3f4e9b5" format_stamp: "Formatted at 2018-07-03 08:41:51 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:51.774806 25211 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:41:51.818951 25211 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:41:51.820328 25211 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:41:51.820641 25211 tablet_server_main.cc:90] Starting tablet server... I0703 01:41:51.879611 24963 heartbeater.cc:467] Master 127.5.240.254:57131 was elected leader, sending a full tablet report... I0703 01:41:51.908231 25211 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.196:41121 I0703 01:41:51.908625 25211 webserver.cc:175] Starting webserver on 127.5.240.196:0 I0703 01:41:51.908668 25211 webserver.cc:186] Document root disabled I0703 01:41:51.909828 25211 webserver.cc:313] Webserver started. Bound to: http://127.5.240.196:40184/ I0703 01:41:51.910357 25211 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb I0703 01:41:51.911957 25211 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:41:51.915483 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 25211 I0703 01:41:51.916373 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data --block_manager=log --rpc_bind_addresses=127.5.240.197:0 --local_ip_for_outbound_sockets=127.5.240.197 --webserver_interface=127.5.240.197 --webserver_port=0 --tserver_master_addrs=127.5.240.254:57131 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0703 01:41:51.926072 25330 heartbeater.cc:346] Connected to a master server at 127.5.240.254:57131 I0703 01:41:51.926211 25330 heartbeater.cc:426] Registering TS with master... I0703 01:41:51.926569 25330 heartbeater.cc:475] Master 127.5.240.254:57131 requested a full tablet report, sending... I0703 01:41:51.927603 24800 ts_manager.cc:98] Registered new tserver with Master: ca6ae9994a78460ca6c4ae4ec3f4e9b5 (127.5.240.196:41121) I0703 01:41:51.928419 24800 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.196:35557 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:41:52.081689 25333 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:41:52.082722 25333 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:41:52.084794 25333 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0703 01:41:52.084842 25333 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:41:52.084923 25333 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.197 I0703 01:41:52.091845 25333 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.197:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb --webserver_interface=127.5.240.197 --webserver_port=0 --tserver_master_addrs=127.5.240.254:57131 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.197 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:41:52.098523 25333 tablet_server_main.cc:85] Initializing tablet server... I0703 01:41:52.100126 25333 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 209708us I0703 01:41:52.100718 25333 fs_manager.cc:261] Metadata directory not provided I0703 01:41:52.100781 25333 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:41:52.100998 25333 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:41:52.101061 25333 server_base.cc:433] Attempting to create new FS layout instead I0703 01:41:52.105201 25333 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/instance: uuid: "2f7e2b9db52f44ba81b7393403f9ed37" format_stamp: "Formatted at 2018-07-03 08:41:52 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:52.105726 25333 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal/instance: uuid: "2f7e2b9db52f44ba81b7393403f9ed37" format_stamp: "Formatted at 2018-07-03 08:41:52 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:52.113041 25333 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.003s sys 0.003s I0703 01:41:52.114270 25333 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:41:52.114660 25333 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:41:52.118613 25333 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s I0703 01:41:52.118657 25333 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal uuid: "2f7e2b9db52f44ba81b7393403f9ed37" format_stamp: "Formatted at 2018-07-03 08:41:52 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:52.118768 25333 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:41:52.178907 25333 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:41:52.180254 25333 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:41:52.180532 25333 tablet_server_main.cc:90] Starting tablet server... I0703 01:41:52.229413 25086 heartbeater.cc:467] Master 127.5.240.254:57131 was elected leader, sending a full tablet report... I0703 01:41:52.265400 25333 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.197:45230 I0703 01:41:52.265825 25333 webserver.cc:175] Starting webserver on 127.5.240.197:0 I0703 01:41:52.265864 25333 webserver.cc:186] Document root disabled I0703 01:41:52.267019 25333 webserver.cc:313] Webserver started. Bound to: http://127.5.240.197:35582/ I0703 01:41:52.267558 25333 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb I0703 01:41:52.269408 25333 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:41:52.277896 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 25333 I0703 01:41:52.278498 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data --block_manager=log --rpc_bind_addresses=127.5.240.198:0 --local_ip_for_outbound_sockets=127.5.240.198 --webserver_interface=127.5.240.198 --webserver_port=0 --tserver_master_addrs=127.5.240.254:57131 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0703 01:41:52.281322 25452 heartbeater.cc:346] Connected to a master server at 127.5.240.254:57131 I0703 01:41:52.281461 25452 heartbeater.cc:426] Registering TS with master... I0703 01:41:52.281824 25452 heartbeater.cc:475] Master 127.5.240.254:57131 requested a full tablet report, sending... I0703 01:41:52.282814 24800 ts_manager.cc:98] Registered new tserver with Master: 2f7e2b9db52f44ba81b7393403f9ed37 (127.5.240.197:45230) I0703 01:41:52.283706 24800 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.197:49312 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:41:52.441933 25455 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:41:52.442958 25455 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:41:52.444979 25455 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0703 01:41:52.445044 25455 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:41:52.445163 25455 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.198 I0703 01:41:52.452210 25455 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.198:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data/info.pb --webserver_interface=127.5.240.198 --webserver_port=0 --tserver_master_addrs=127.5.240.254:57131 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.198 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:41:52.458819 25455 tablet_server_main.cc:85] Initializing tablet server... I0703 01:41:52.460348 25455 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 209708us I0703 01:41:52.460917 25455 fs_manager.cc:261] Metadata directory not provided I0703 01:41:52.460990 25455 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:41:52.461280 25455 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:41:52.461419 25455 server_base.cc:433] Attempting to create new FS layout instead I0703 01:41:52.465611 25455 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data/instance: uuid: "01f6144ac0484e1d9a6da7ff80ba71f0" format_stamp: "Formatted at 2018-07-03 08:41:52 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:52.466127 25455 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/wal/instance: uuid: "01f6144ac0484e1d9a6da7ff80ba71f0" format_stamp: "Formatted at 2018-07-03 08:41:52 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:52.473430 25455 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.004s sys 0.002s I0703 01:41:52.474624 25455 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:41:52.474988 25455 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:41:52.478384 25455 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0703 01:41:52.478489 25455 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/wal uuid: "01f6144ac0484e1d9a6da7ff80ba71f0" format_stamp: "Formatted at 2018-07-03 08:41:52 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:52.478708 25455 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:41:52.528110 25455 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:41:52.529475 25455 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:41:52.529778 25455 tablet_server_main.cc:90] Starting tablet server... I0703 01:41:52.587350 25208 heartbeater.cc:467] Master 127.5.240.254:57131 was elected leader, sending a full tablet report... I0703 01:41:52.615195 25455 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.198:35740 I0703 01:41:52.615572 25455 webserver.cc:175] Starting webserver on 127.5.240.198:0 I0703 01:41:52.615628 25455 webserver.cc:186] Document root disabled I0703 01:41:52.616791 25455 webserver.cc:313] Webserver started. Bound to: http://127.5.240.198:53820/ I0703 01:41:52.617310 25455 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data/info.pb I0703 01:41:52.619503 25455 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:41:52.625589 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 25455 I0703 01:41:52.626392 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data --block_manager=log --rpc_bind_addresses=127.5.240.199:0 --local_ip_for_outbound_sockets=127.5.240.199 --webserver_interface=127.5.240.199 --webserver_port=0 --tserver_master_addrs=127.5.240.254:57131 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0703 01:41:52.631726 25574 heartbeater.cc:346] Connected to a master server at 127.5.240.254:57131 I0703 01:41:52.631866 25574 heartbeater.cc:426] Registering TS with master... I0703 01:41:52.632247 25574 heartbeater.cc:475] Master 127.5.240.254:57131 requested a full tablet report, sending... I0703 01:41:52.633190 24800 ts_manager.cc:98] Registered new tserver with Master: 01f6144ac0484e1d9a6da7ff80ba71f0 (127.5.240.198:35740) I0703 01:41:52.634057 24800 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.198:44547 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:41:52.787112 25577 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:41:52.788128 25577 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:41:52.790107 25577 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0703 01:41:52.790158 25577 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:41:52.790244 25577 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.199 I0703 01:41:52.796829 25577 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.199:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data/info.pb --webserver_interface=127.5.240.199 --webserver_port=0 --tserver_master_addrs=127.5.240.254:57131 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.199 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:41:52.803325 25577 tablet_server_main.cc:85] Initializing tablet server... I0703 01:41:52.804936 25577 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 209708us I0703 01:41:52.805469 25577 fs_manager.cc:261] Metadata directory not provided I0703 01:41:52.805531 25577 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:41:52.806143 25577 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:41:52.806206 25577 server_base.cc:433] Attempting to create new FS layout instead I0703 01:41:52.810222 25577 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data/instance: uuid: "a344baa95900451dbaf349b68e662b27" format_stamp: "Formatted at 2018-07-03 08:41:52 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:52.810751 25577 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/wal/instance: uuid: "a344baa95900451dbaf349b68e662b27" format_stamp: "Formatted at 2018-07-03 08:41:52 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:52.818519 25577 fs_manager.cc:501] Time spent creating directory manager: real 0.008s user 0.002s sys 0.004s I0703 01:41:52.819731 25577 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:41:52.820086 25577 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:41:52.823246 25577 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0703 01:41:52.823292 25577 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/wal uuid: "a344baa95900451dbaf349b68e662b27" format_stamp: "Formatted at 2018-07-03 08:41:52 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:52.823437 25577 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:41:52.845643 25577 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:41:52.847445 25577 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:41:52.847811 25577 tablet_server_main.cc:90] Starting tablet server... I0703 01:41:52.930282 25330 heartbeater.cc:467] Master 127.5.240.254:57131 was elected leader, sending a full tablet report... I0703 01:41:52.934324 25577 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.199:43710 I0703 01:41:52.934734 25577 webserver.cc:175] Starting webserver on 127.5.240.199:0 I0703 01:41:52.934773 25577 webserver.cc:186] Document root disabled I0703 01:41:52.935446 25577 webserver.cc:313] Webserver started. Bound to: http://127.5.240.199:53434/ I0703 01:41:52.935972 25577 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data/info.pb I0703 01:41:52.938345 25577 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:41:52.944355 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 25577 I0703 01:41:52.950513 25696 heartbeater.cc:346] Connected to a master server at 127.5.240.254:57131 I0703 01:41:52.950678 25696 heartbeater.cc:426] Registering TS with master... I0703 01:41:52.951028 25696 heartbeater.cc:475] Master 127.5.240.254:57131 requested a full tablet report, sending... I0703 01:41:52.951982 24800 ts_manager.cc:98] Registered new tserver with Master: a344baa95900451dbaf349b68e662b27 (127.5.240.199:43710) I0703 01:41:52.952833 24800 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.199:58177 I0703 01:41:52.953915 6083 external_mini_cluster.cc:490] 7 TS(s) registered with all masters I0703 01:41:52.963878 24800 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.5.240.254:56497: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 1 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0703 01:41:52.983367 25145 tablet_service.cc:759] Processing CreateTablet for tablet fc9b869cd219497896d16275d18c81b9 (table=TestTable [id=d5d127d72867427d8612956b24cba2d2]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:41:52.984047 25145 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet fc9b869cd219497896d16275d18c81b9. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:41:52.993703 25145 ts_tablet_manager.cc:1173] T fc9b869cd219497896d16275d18c81b9 P cff51d1a4baa4cd0a1625e1d7db94f2c: Registered tablet (data state: TABLET_DATA_READY) I0703 01:41:52.994299 25712 ts_tablet_manager.cc:1028] T fc9b869cd219497896d16275d18c81b9 P cff51d1a4baa4cd0a1625e1d7db94f2c: Bootstrapping tablet I0703 01:41:52.994616 25712 tablet_bootstrap.cc:427] T fc9b869cd219497896d16275d18c81b9 P cff51d1a4baa4cd0a1625e1d7db94f2c: Bootstrap starting. I0703 01:41:52.997133 25712 tablet_bootstrap.cc:578] T fc9b869cd219497896d16275d18c81b9 P cff51d1a4baa4cd0a1625e1d7db94f2c: No blocks or log segments found. Creating new log. I0703 01:41:52.998301 25712 log.cc:525] T fc9b869cd219497896d16275d18c81b9 P cff51d1a4baa4cd0a1625e1d7db94f2c: Log is configured to *not* fsync() on all Append() calls I0703 01:41:53.000206 25712 tablet_bootstrap.cc:427] T fc9b869cd219497896d16275d18c81b9 P cff51d1a4baa4cd0a1625e1d7db94f2c: No bootstrap required, opened a new log I0703 01:41:53.000388 25712 ts_tablet_manager.cc:1045] T fc9b869cd219497896d16275d18c81b9 P cff51d1a4baa4cd0a1625e1d7db94f2c: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s I0703 01:41:53.006451 25712 raft_consensus.cc:325] T fc9b869cd219497896d16275d18c81b9 P cff51d1a4baa4cd0a1625e1d7db94f2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff51d1a4baa4cd0a1625e1d7db94f2c" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 38199 } } I0703 01:41:53.006707 25712 raft_consensus.cc:351] T fc9b869cd219497896d16275d18c81b9 P cff51d1a4baa4cd0a1625e1d7db94f2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:41:53.006775 25712 raft_consensus.cc:628] T fc9b869cd219497896d16275d18c81b9 P cff51d1a4baa4cd0a1625e1d7db94f2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cff51d1a4baa4cd0a1625e1d7db94f2c, State: Initialized, Role: FOLLOWER I0703 01:41:53.007143 25712 consensus_queue.cc:227] T fc9b869cd219497896d16275d18c81b9 P cff51d1a4baa4cd0a1625e1d7db94f2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff51d1a4baa4cd0a1625e1d7db94f2c" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 38199 } } I0703 01:41:53.007318 25712 raft_consensus.cc:365] T fc9b869cd219497896d16275d18c81b9 P cff51d1a4baa4cd0a1625e1d7db94f2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:41:53.007392 25712 raft_consensus.cc:457] T fc9b869cd219497896d16275d18c81b9 P cff51d1a4baa4cd0a1625e1d7db94f2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:41:53.007472 25712 raft_consensus.cc:2730] T fc9b869cd219497896d16275d18c81b9 P cff51d1a4baa4cd0a1625e1d7db94f2c: Snoozing failure detection for 1.899s (starting election) I0703 01:41:53.007546 25712 raft_consensus.cc:2782] T fc9b869cd219497896d16275d18c81b9 P cff51d1a4baa4cd0a1625e1d7db94f2c [term 0 FOLLOWER]: Advancing to term 1 I0703 01:41:53.008903 25712 raft_consensus.cc:479] T fc9b869cd219497896d16275d18c81b9 P cff51d1a4baa4cd0a1625e1d7db94f2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff51d1a4baa4cd0a1625e1d7db94f2c" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 38199 } } I0703 01:41:53.009224 25712 leader_election.cc:270] T fc9b869cd219497896d16275d18c81b9 P cff51d1a4baa4cd0a1625e1d7db94f2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:41:53.010361 25714 raft_consensus.cc:2730] T fc9b869cd219497896d16275d18c81b9 P cff51d1a4baa4cd0a1625e1d7db94f2c: Snoozing failure detection for 2.110s (election complete) I0703 01:41:53.010510 25714 raft_consensus.cc:2537] T fc9b869cd219497896d16275d18c81b9 P cff51d1a4baa4cd0a1625e1d7db94f2c [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:41:53.013087 25714 raft_consensus.cc:593] T fc9b869cd219497896d16275d18c81b9 P cff51d1a4baa4cd0a1625e1d7db94f2c [term 1 LEADER]: Becoming Leader. State: Replica: cff51d1a4baa4cd0a1625e1d7db94f2c, State: Running, Role: LEADER I0703 01:41:53.013121 25712 ts_tablet_manager.cc:1073] T fc9b869cd219497896d16275d18c81b9 P cff51d1a4baa4cd0a1625e1d7db94f2c: Time spent starting tablet: real 0.012s user 0.006s sys 0.006s I0703 01:41:53.013455 25714 consensus_queue.cc:204] T fc9b869cd219497896d16275d18c81b9 P cff51d1a4baa4cd0a1625e1d7db94f2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff51d1a4baa4cd0a1625e1d7db94f2c" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 38199 } } I0703 01:41:53.017309 24800 catalog_manager.cc:3798] T fc9b869cd219497896d16275d18c81b9 P cff51d1a4baa4cd0a1625e1d7db94f2c reported cstate change: term changed from 0 to 1, leader changed from to cff51d1a4baa4cd0a1625e1d7db94f2c (127.5.240.195). New cstate: current_term: 1 leader_uuid: "cff51d1a4baa4cd0a1625e1d7db94f2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff51d1a4baa4cd0a1625e1d7db94f2c" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 38199 } } } I0703 01:41:53.025269 6083 external_mini_cluster.cc:490] 7 TS(s) registered with all masters I0703 01:41:53.026360 6083 ts_itest-base.cc:233] Waiting for 0 tablets on tserver dfd7ad6293204fc89f9e72436796698b to finish bootstrapping I0703 01:41:53.033159 6083 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 8ec1760d41f842f398d90e47ab52973b to finish bootstrapping I0703 01:41:53.039474 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver cff51d1a4baa4cd0a1625e1d7db94f2c to finish bootstrapping I0703 01:41:53.044577 6083 ts_itest-base.cc:233] Waiting for 0 tablets on tserver ca6ae9994a78460ca6c4ae4ec3f4e9b5 to finish bootstrapping I0703 01:41:53.050776 6083 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 2f7e2b9db52f44ba81b7393403f9ed37 to finish bootstrapping I0703 01:41:53.056460 6083 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 01f6144ac0484e1d9a6da7ff80ba71f0 to finish bootstrapping I0703 01:41:53.061810 6083 ts_itest-base.cc:233] Waiting for 0 tablets on tserver a344baa95900451dbaf349b68e662b27 to finish bootstrapping I0703 01:41:53.069313 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 25089 I0703 01:41:53.069952 25205 generic_service.cc:169] Checking for leaks (request via RPC) W0703 01:41:53.112157 25209 tablet.cc:1678] T fc9b869cd219497896d16275d18c81b9 P cff51d1a4baa4cd0a1625e1d7db94f2c: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:41:53.112742 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 25089 I0703 01:41:53.121083 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 25211 W0703 01:41:53.121057 24781 connection.cc:512] server connection from 127.5.240.195:36555 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.195:36555): Connection reset by peer (error 104) I0703 01:41:53.122037 25327 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:41:53.160131 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 25211 W0703 01:41:53.168342 24781 connection.cc:512] server connection from 127.5.240.196:35557 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.196:35557): Connection reset by peer (error 104) I0703 01:41:53.168701 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 25333 I0703 01:41:53.169417 25449 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:41:53.206990 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 25333 I0703 01:41:53.214962 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 25455 W0703 01:41:53.215035 24783 connection.cc:512] server connection from 127.5.240.197:49312 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.197:49312): Connection reset by peer (error 104) I0703 01:41:53.215829 25571 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:41:53.253707 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 25455 W0703 01:41:53.261399 24784 connection.cc:512] server connection from 127.5.240.198:44547 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.198:44547): Connection reset by peer (error 104) I0703 01:41:53.261463 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 25577 I0703 01:41:53.262202 25693 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:41:53.300330 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 25577 W0703 01:41:53.307430 24781 connection.cc:512] server connection from 127.5.240.199:58177 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.199:58177): Connection reset by peer (error 104) I0703 01:41:57.058933 24800 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.5.240.254:56497: name: "rebalance_test_table_0" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 1 split_rows_range_bounds { } partition_schema { hash_bucket_schemas { columns { name: "key" } num_buckets: 3 seed: 0 } } I0703 01:41:57.072638 24900 tablet_service.cc:759] Processing CreateTablet for tablet ff7fa826668f419fa3c5225e5d810b9c (table=rebalance_test_table_0 [id=fa6dde76da3d4d7e904d1ea3e3be2d1d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I0703 01:41:57.073499 24900 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet ff7fa826668f419fa3c5225e5d810b9c. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:41:57.074510 25023 tablet_service.cc:759] Processing CreateTablet for tablet 46469043bcb248f6a4505bc705edcda2 (table=rebalance_test_table_0 [id=fa6dde76da3d4d7e904d1ea3e3be2d1d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I0703 01:41:57.074954 24899 tablet_service.cc:759] Processing CreateTablet for tablet 6359db2dd602463e841abc2a907c49a3 (table=rebalance_test_table_0 [id=fa6dde76da3d4d7e904d1ea3e3be2d1d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I0703 01:41:57.075249 25023 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 46469043bcb248f6a4505bc705edcda2. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:41:57.076552 24899 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 6359db2dd602463e841abc2a907c49a3. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:41:57.088973 24899 ts_tablet_manager.cc:1173] T 6359db2dd602463e841abc2a907c49a3 P dfd7ad6293204fc89f9e72436796698b: Registered tablet (data state: TABLET_DATA_READY) I0703 01:41:57.088958 25023 ts_tablet_manager.cc:1173] T 46469043bcb248f6a4505bc705edcda2 P 8ec1760d41f842f398d90e47ab52973b: Registered tablet (data state: TABLET_DATA_READY) I0703 01:41:57.089298 24900 ts_tablet_manager.cc:1173] T ff7fa826668f419fa3c5225e5d810b9c P dfd7ad6293204fc89f9e72436796698b: Registered tablet (data state: TABLET_DATA_READY) I0703 01:41:57.090215 25738 ts_tablet_manager.cc:1028] T 6359db2dd602463e841abc2a907c49a3 P dfd7ad6293204fc89f9e72436796698b: Bootstrapping tablet I0703 01:41:57.090405 25737 ts_tablet_manager.cc:1028] T 46469043bcb248f6a4505bc705edcda2 P 8ec1760d41f842f398d90e47ab52973b: Bootstrapping tablet I0703 01:41:57.090517 25738 tablet_bootstrap.cc:427] T 6359db2dd602463e841abc2a907c49a3 P dfd7ad6293204fc89f9e72436796698b: Bootstrap starting. I0703 01:41:57.090626 25737 tablet_bootstrap.cc:427] T 46469043bcb248f6a4505bc705edcda2 P 8ec1760d41f842f398d90e47ab52973b: Bootstrap starting. I0703 01:41:57.092700 25738 tablet_bootstrap.cc:578] T 6359db2dd602463e841abc2a907c49a3 P dfd7ad6293204fc89f9e72436796698b: No blocks or log segments found. Creating new log. I0703 01:41:57.092991 25737 tablet_bootstrap.cc:578] T 46469043bcb248f6a4505bc705edcda2 P 8ec1760d41f842f398d90e47ab52973b: No blocks or log segments found. Creating new log. I0703 01:41:57.093430 25738 log.cc:525] T 6359db2dd602463e841abc2a907c49a3 P dfd7ad6293204fc89f9e72436796698b: Log is configured to *not* fsync() on all Append() calls I0703 01:41:57.093679 25737 log.cc:525] T 46469043bcb248f6a4505bc705edcda2 P 8ec1760d41f842f398d90e47ab52973b: Log is configured to *not* fsync() on all Append() calls I0703 01:41:57.095685 25738 tablet_bootstrap.cc:427] T 6359db2dd602463e841abc2a907c49a3 P dfd7ad6293204fc89f9e72436796698b: No bootstrap required, opened a new log I0703 01:41:57.095736 25737 tablet_bootstrap.cc:427] T 46469043bcb248f6a4505bc705edcda2 P 8ec1760d41f842f398d90e47ab52973b: No bootstrap required, opened a new log I0703 01:41:57.095858 25738 ts_tablet_manager.cc:1045] T 6359db2dd602463e841abc2a907c49a3 P dfd7ad6293204fc89f9e72436796698b: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.002s I0703 01:41:57.095891 25737 ts_tablet_manager.cc:1045] T 46469043bcb248f6a4505bc705edcda2 P 8ec1760d41f842f398d90e47ab52973b: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s I0703 01:41:57.101641 25737 raft_consensus.cc:325] T 46469043bcb248f6a4505bc705edcda2 P 8ec1760d41f842f398d90e47ab52973b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } I0703 01:41:57.101874 25737 raft_consensus.cc:351] T 46469043bcb248f6a4505bc705edcda2 P 8ec1760d41f842f398d90e47ab52973b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:41:57.101933 25737 raft_consensus.cc:628] T 46469043bcb248f6a4505bc705edcda2 P 8ec1760d41f842f398d90e47ab52973b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ec1760d41f842f398d90e47ab52973b, State: Initialized, Role: FOLLOWER I0703 01:41:57.101848 25738 raft_consensus.cc:325] T 6359db2dd602463e841abc2a907c49a3 P dfd7ad6293204fc89f9e72436796698b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } I0703 01:41:57.102113 25738 raft_consensus.cc:351] T 6359db2dd602463e841abc2a907c49a3 P dfd7ad6293204fc89f9e72436796698b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:41:57.102175 25738 raft_consensus.cc:628] T 6359db2dd602463e841abc2a907c49a3 P dfd7ad6293204fc89f9e72436796698b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfd7ad6293204fc89f9e72436796698b, State: Initialized, Role: FOLLOWER I0703 01:41:57.102319 25737 consensus_queue.cc:227] T 46469043bcb248f6a4505bc705edcda2 P 8ec1760d41f842f398d90e47ab52973b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } I0703 01:41:57.102490 25737 raft_consensus.cc:365] T 46469043bcb248f6a4505bc705edcda2 P 8ec1760d41f842f398d90e47ab52973b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:41:57.102574 25737 raft_consensus.cc:457] T 46469043bcb248f6a4505bc705edcda2 P 8ec1760d41f842f398d90e47ab52973b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:41:57.102654 25737 raft_consensus.cc:2730] T 46469043bcb248f6a4505bc705edcda2 P 8ec1760d41f842f398d90e47ab52973b: Snoozing failure detection for 1.956s (starting election) I0703 01:41:57.102730 25737 raft_consensus.cc:2782] T 46469043bcb248f6a4505bc705edcda2 P 8ec1760d41f842f398d90e47ab52973b [term 0 FOLLOWER]: Advancing to term 1 I0703 01:41:57.102625 25738 consensus_queue.cc:227] T 6359db2dd602463e841abc2a907c49a3 P dfd7ad6293204fc89f9e72436796698b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } I0703 01:41:57.102933 25738 raft_consensus.cc:365] T 6359db2dd602463e841abc2a907c49a3 P dfd7ad6293204fc89f9e72436796698b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:41:57.103001 25738 raft_consensus.cc:457] T 6359db2dd602463e841abc2a907c49a3 P dfd7ad6293204fc89f9e72436796698b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:41:57.103070 25738 raft_consensus.cc:2730] T 6359db2dd602463e841abc2a907c49a3 P dfd7ad6293204fc89f9e72436796698b: Snoozing failure detection for 1.822s (starting election) I0703 01:41:57.103140 25738 raft_consensus.cc:2782] T 6359db2dd602463e841abc2a907c49a3 P dfd7ad6293204fc89f9e72436796698b [term 0 FOLLOWER]: Advancing to term 1 I0703 01:41:57.104140 25737 raft_consensus.cc:479] T 46469043bcb248f6a4505bc705edcda2 P 8ec1760d41f842f398d90e47ab52973b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } I0703 01:41:57.104460 25737 leader_election.cc:270] T 46469043bcb248f6a4505bc705edcda2 P 8ec1760d41f842f398d90e47ab52973b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:41:57.104508 25738 raft_consensus.cc:479] T 6359db2dd602463e841abc2a907c49a3 P dfd7ad6293204fc89f9e72436796698b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } I0703 01:41:57.105180 25738 leader_election.cc:270] T 6359db2dd602463e841abc2a907c49a3 P dfd7ad6293204fc89f9e72436796698b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:41:57.105396 25741 raft_consensus.cc:2730] T 46469043bcb248f6a4505bc705edcda2 P 8ec1760d41f842f398d90e47ab52973b: Snoozing failure detection for 1.859s (election complete) I0703 01:41:57.105654 25741 raft_consensus.cc:2537] T 46469043bcb248f6a4505bc705edcda2 P 8ec1760d41f842f398d90e47ab52973b [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:41:57.107724 25737 ts_tablet_manager.cc:1073] T 46469043bcb248f6a4505bc705edcda2 P 8ec1760d41f842f398d90e47ab52973b: Time spent starting tablet: real 0.012s user 0.008s sys 0.003s I0703 01:41:57.108057 25743 raft_consensus.cc:2730] T 6359db2dd602463e841abc2a907c49a3 P dfd7ad6293204fc89f9e72436796698b: Snoozing failure detection for 1.608s (election complete) I0703 01:41:57.108213 25743 raft_consensus.cc:2537] T 6359db2dd602463e841abc2a907c49a3 P dfd7ad6293204fc89f9e72436796698b [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:41:57.108968 25741 raft_consensus.cc:593] T 46469043bcb248f6a4505bc705edcda2 P 8ec1760d41f842f398d90e47ab52973b [term 1 LEADER]: Becoming Leader. State: Replica: 8ec1760d41f842f398d90e47ab52973b, State: Running, Role: LEADER I0703 01:41:57.109386 25741 consensus_queue.cc:204] T 46469043bcb248f6a4505bc705edcda2 P 8ec1760d41f842f398d90e47ab52973b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } I0703 01:41:57.110163 25743 raft_consensus.cc:593] T 6359db2dd602463e841abc2a907c49a3 P dfd7ad6293204fc89f9e72436796698b [term 1 LEADER]: Becoming Leader. State: Replica: dfd7ad6293204fc89f9e72436796698b, State: Running, Role: LEADER I0703 01:41:57.110625 25743 consensus_queue.cc:204] T 6359db2dd602463e841abc2a907c49a3 P dfd7ad6293204fc89f9e72436796698b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } I0703 01:41:57.113092 24800 catalog_manager.cc:3798] T 46469043bcb248f6a4505bc705edcda2 P 8ec1760d41f842f398d90e47ab52973b reported cstate change: term changed from 0 to 1, leader changed from to 8ec1760d41f842f398d90e47ab52973b (127.5.240.194). New cstate: current_term: 1 leader_uuid: "8ec1760d41f842f398d90e47ab52973b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } } I0703 01:41:57.114363 24799 catalog_manager.cc:3798] T 6359db2dd602463e841abc2a907c49a3 P dfd7ad6293204fc89f9e72436796698b reported cstate change: term changed from 0 to 1, leader changed from to dfd7ad6293204fc89f9e72436796698b (127.5.240.193). New cstate: current_term: 1 leader_uuid: "dfd7ad6293204fc89f9e72436796698b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } } I0703 01:41:57.119925 25738 ts_tablet_manager.cc:1073] T 6359db2dd602463e841abc2a907c49a3 P dfd7ad6293204fc89f9e72436796698b: Time spent starting tablet: real 0.024s user 0.008s sys 0.010s I0703 01:41:57.120095 25738 ts_tablet_manager.cc:1028] T ff7fa826668f419fa3c5225e5d810b9c P dfd7ad6293204fc89f9e72436796698b: Bootstrapping tablet I0703 01:41:57.120232 25738 tablet_bootstrap.cc:427] T ff7fa826668f419fa3c5225e5d810b9c P dfd7ad6293204fc89f9e72436796698b: Bootstrap starting. I0703 01:41:57.121865 25738 tablet_bootstrap.cc:578] T ff7fa826668f419fa3c5225e5d810b9c P dfd7ad6293204fc89f9e72436796698b: No blocks or log segments found. Creating new log. I0703 01:41:57.123594 25738 tablet_bootstrap.cc:427] T ff7fa826668f419fa3c5225e5d810b9c P dfd7ad6293204fc89f9e72436796698b: No bootstrap required, opened a new log I0703 01:41:57.123690 25738 ts_tablet_manager.cc:1045] T ff7fa826668f419fa3c5225e5d810b9c P dfd7ad6293204fc89f9e72436796698b: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s I0703 01:41:57.124264 25738 raft_consensus.cc:325] T ff7fa826668f419fa3c5225e5d810b9c P dfd7ad6293204fc89f9e72436796698b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } I0703 01:41:57.124362 25738 raft_consensus.cc:351] T ff7fa826668f419fa3c5225e5d810b9c P dfd7ad6293204fc89f9e72436796698b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:41:57.124413 25738 raft_consensus.cc:628] T ff7fa826668f419fa3c5225e5d810b9c P dfd7ad6293204fc89f9e72436796698b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfd7ad6293204fc89f9e72436796698b, State: Initialized, Role: FOLLOWER I0703 01:41:57.124609 25738 consensus_queue.cc:227] T ff7fa826668f419fa3c5225e5d810b9c P dfd7ad6293204fc89f9e72436796698b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } I0703 01:41:57.124727 25738 raft_consensus.cc:365] T ff7fa826668f419fa3c5225e5d810b9c P dfd7ad6293204fc89f9e72436796698b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:41:57.124781 25738 raft_consensus.cc:457] T ff7fa826668f419fa3c5225e5d810b9c P dfd7ad6293204fc89f9e72436796698b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:41:57.124838 25738 raft_consensus.cc:2730] T ff7fa826668f419fa3c5225e5d810b9c P dfd7ad6293204fc89f9e72436796698b: Snoozing failure detection for 1.932s (starting election) I0703 01:41:57.124899 25738 raft_consensus.cc:2782] T ff7fa826668f419fa3c5225e5d810b9c P dfd7ad6293204fc89f9e72436796698b [term 0 FOLLOWER]: Advancing to term 1 I0703 01:41:57.126196 25738 raft_consensus.cc:479] T ff7fa826668f419fa3c5225e5d810b9c P dfd7ad6293204fc89f9e72436796698b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } I0703 01:41:57.126350 25738 leader_election.cc:270] T ff7fa826668f419fa3c5225e5d810b9c P dfd7ad6293204fc89f9e72436796698b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:41:57.126577 25745 raft_consensus.cc:2730] T ff7fa826668f419fa3c5225e5d810b9c P dfd7ad6293204fc89f9e72436796698b: Snoozing failure detection for 1.992s (election complete) I0703 01:41:57.126749 25745 raft_consensus.cc:2537] T ff7fa826668f419fa3c5225e5d810b9c P dfd7ad6293204fc89f9e72436796698b [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:41:57.126946 25745 raft_consensus.cc:593] T ff7fa826668f419fa3c5225e5d810b9c P dfd7ad6293204fc89f9e72436796698b [term 1 LEADER]: Becoming Leader. State: Replica: dfd7ad6293204fc89f9e72436796698b, State: Running, Role: LEADER I0703 01:41:57.127058 25738 ts_tablet_manager.cc:1073] T ff7fa826668f419fa3c5225e5d810b9c P dfd7ad6293204fc89f9e72436796698b: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s I0703 01:41:57.127213 25745 consensus_queue.cc:204] T ff7fa826668f419fa3c5225e5d810b9c P dfd7ad6293204fc89f9e72436796698b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } I0703 01:41:57.129448 24800 catalog_manager.cc:3798] T ff7fa826668f419fa3c5225e5d810b9c P dfd7ad6293204fc89f9e72436796698b reported cstate change: term changed from 0 to 1, leader changed from to dfd7ad6293204fc89f9e72436796698b (127.5.240.193). New cstate: current_term: 1 leader_uuid: "dfd7ad6293204fc89f9e72436796698b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } } W0703 01:41:57.212745 25087 tablet.cc:1678] T 46469043bcb248f6a4505bc705edcda2 P 8ec1760d41f842f398d90e47ab52973b: Can't schedule compaction. Clean time has not been advanced past its initial value. W0703 01:41:57.358852 25751 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:41:57.365090 24964 tablet.cc:1678] T 6359db2dd602463e841abc2a907c49a3 P dfd7ad6293204fc89f9e72436796698b: Can't schedule compaction. Clean time has not been advanced past its initial value. Using table 'rebalance_test_table_0' Generator report time total : 141.265 ms time per row: 0.0706325 ms I0703 01:41:57.596096 24796 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.5.240.254:56497: name: "rebalance_test_table_1" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 1 split_rows_range_bounds { } partition_schema { hash_bucket_schemas { columns { name: "key" } num_buckets: 3 seed: 0 } } I0703 01:41:57.601863 25023 tablet_service.cc:759] Processing CreateTablet for tablet 9a818723ff79493aa17e7434040e8975 (table=rebalance_test_table_1 [id=fce1efea49b34f7fa21533a1026074fb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I0703 01:41:57.602283 25023 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 9a818723ff79493aa17e7434040e8975. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:41:57.602327 24899 tablet_service.cc:759] Processing CreateTablet for tablet 4eea38c54a6b491081040e6059ca592e (table=rebalance_test_table_1 [id=fce1efea49b34f7fa21533a1026074fb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I0703 01:41:57.602455 25022 tablet_service.cc:759] Processing CreateTablet for tablet fd7030608be84fa9b61c3ae90e04e27e (table=rebalance_test_table_1 [id=fce1efea49b34f7fa21533a1026074fb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I0703 01:41:57.602689 24899 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 4eea38c54a6b491081040e6059ca592e. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:41:57.602921 25022 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet fd7030608be84fa9b61c3ae90e04e27e. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:41:57.606695 24899 ts_tablet_manager.cc:1173] T 4eea38c54a6b491081040e6059ca592e P dfd7ad6293204fc89f9e72436796698b: Registered tablet (data state: TABLET_DATA_READY) I0703 01:41:57.606899 25738 ts_tablet_manager.cc:1028] T 4eea38c54a6b491081040e6059ca592e P dfd7ad6293204fc89f9e72436796698b: Bootstrapping tablet I0703 01:41:57.607028 25738 tablet_bootstrap.cc:427] T 4eea38c54a6b491081040e6059ca592e P dfd7ad6293204fc89f9e72436796698b: Bootstrap starting. I0703 01:41:57.607800 25023 ts_tablet_manager.cc:1173] T 9a818723ff79493aa17e7434040e8975 P 8ec1760d41f842f398d90e47ab52973b: Registered tablet (data state: TABLET_DATA_READY) I0703 01:41:57.607993 25737 ts_tablet_manager.cc:1028] T 9a818723ff79493aa17e7434040e8975 P 8ec1760d41f842f398d90e47ab52973b: Bootstrapping tablet I0703 01:41:57.608080 25738 tablet_bootstrap.cc:578] T 4eea38c54a6b491081040e6059ca592e P dfd7ad6293204fc89f9e72436796698b: No blocks or log segments found. Creating new log. I0703 01:41:57.608155 25737 tablet_bootstrap.cc:427] T 9a818723ff79493aa17e7434040e8975 P 8ec1760d41f842f398d90e47ab52973b: Bootstrap starting. I0703 01:41:57.609503 25738 tablet_bootstrap.cc:427] T 4eea38c54a6b491081040e6059ca592e P dfd7ad6293204fc89f9e72436796698b: No bootstrap required, opened a new log I0703 01:41:57.609624 25738 ts_tablet_manager.cc:1045] T 4eea38c54a6b491081040e6059ca592e P dfd7ad6293204fc89f9e72436796698b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s I0703 01:41:57.609666 25737 tablet_bootstrap.cc:578] T 9a818723ff79493aa17e7434040e8975 P 8ec1760d41f842f398d90e47ab52973b: No blocks or log segments found. Creating new log. I0703 01:41:57.610147 25738 raft_consensus.cc:325] T 4eea38c54a6b491081040e6059ca592e P dfd7ad6293204fc89f9e72436796698b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } I0703 01:41:57.610245 25738 raft_consensus.cc:351] T 4eea38c54a6b491081040e6059ca592e P dfd7ad6293204fc89f9e72436796698b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:41:57.610302 25738 raft_consensus.cc:628] T 4eea38c54a6b491081040e6059ca592e P dfd7ad6293204fc89f9e72436796698b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfd7ad6293204fc89f9e72436796698b, State: Initialized, Role: FOLLOWER I0703 01:41:57.610503 25738 consensus_queue.cc:227] T 4eea38c54a6b491081040e6059ca592e P dfd7ad6293204fc89f9e72436796698b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } I0703 01:41:57.610586 25022 ts_tablet_manager.cc:1173] T fd7030608be84fa9b61c3ae90e04e27e P 8ec1760d41f842f398d90e47ab52973b: Registered tablet (data state: TABLET_DATA_READY) I0703 01:41:57.610617 25738 raft_consensus.cc:365] T 4eea38c54a6b491081040e6059ca592e P dfd7ad6293204fc89f9e72436796698b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:41:57.610672 25738 raft_consensus.cc:457] T 4eea38c54a6b491081040e6059ca592e P dfd7ad6293204fc89f9e72436796698b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:41:57.610803 25738 raft_consensus.cc:2730] T 4eea38c54a6b491081040e6059ca592e P dfd7ad6293204fc89f9e72436796698b: Snoozing failure detection for 1.580s (starting election) I0703 01:41:57.610920 25738 raft_consensus.cc:2782] T 4eea38c54a6b491081040e6059ca592e P dfd7ad6293204fc89f9e72436796698b [term 0 FOLLOWER]: Advancing to term 1 I0703 01:41:57.611554 25737 tablet_bootstrap.cc:427] T 9a818723ff79493aa17e7434040e8975 P 8ec1760d41f842f398d90e47ab52973b: No bootstrap required, opened a new log I0703 01:41:57.611654 25737 ts_tablet_manager.cc:1045] T 9a818723ff79493aa17e7434040e8975 P 8ec1760d41f842f398d90e47ab52973b: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s I0703 01:41:57.612229 25737 raft_consensus.cc:325] T 9a818723ff79493aa17e7434040e8975 P 8ec1760d41f842f398d90e47ab52973b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } I0703 01:41:57.612341 25737 raft_consensus.cc:351] T 9a818723ff79493aa17e7434040e8975 P 8ec1760d41f842f398d90e47ab52973b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:41:57.612399 25737 raft_consensus.cc:628] T 9a818723ff79493aa17e7434040e8975 P 8ec1760d41f842f398d90e47ab52973b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ec1760d41f842f398d90e47ab52973b, State: Initialized, Role: FOLLOWER I0703 01:41:57.612391 25738 raft_consensus.cc:479] T 4eea38c54a6b491081040e6059ca592e P dfd7ad6293204fc89f9e72436796698b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } I0703 01:41:57.612560 25738 leader_election.cc:270] T 4eea38c54a6b491081040e6059ca592e P dfd7ad6293204fc89f9e72436796698b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:41:57.612537 25737 consensus_queue.cc:227] T 9a818723ff79493aa17e7434040e8975 P 8ec1760d41f842f398d90e47ab52973b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } I0703 01:41:57.612740 25737 raft_consensus.cc:365] T 9a818723ff79493aa17e7434040e8975 P 8ec1760d41f842f398d90e47ab52973b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:41:57.612782 25743 raft_consensus.cc:2730] T 4eea38c54a6b491081040e6059ca592e P dfd7ad6293204fc89f9e72436796698b: Snoozing failure detection for 1.686s (election complete) I0703 01:41:57.612807 25737 raft_consensus.cc:457] T 9a818723ff79493aa17e7434040e8975 P 8ec1760d41f842f398d90e47ab52973b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:41:57.612864 25737 raft_consensus.cc:2730] T 9a818723ff79493aa17e7434040e8975 P 8ec1760d41f842f398d90e47ab52973b: Snoozing failure detection for 1.985s (starting election) I0703 01:41:57.612891 25743 raft_consensus.cc:2537] T 4eea38c54a6b491081040e6059ca592e P dfd7ad6293204fc89f9e72436796698b [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:41:57.612918 25737 raft_consensus.cc:2782] T 9a818723ff79493aa17e7434040e8975 P 8ec1760d41f842f398d90e47ab52973b [term 0 FOLLOWER]: Advancing to term 1 I0703 01:41:57.613034 25743 raft_consensus.cc:593] T 4eea38c54a6b491081040e6059ca592e P dfd7ad6293204fc89f9e72436796698b [term 1 LEADER]: Becoming Leader. State: Replica: dfd7ad6293204fc89f9e72436796698b, State: Running, Role: LEADER I0703 01:41:57.613196 25743 consensus_queue.cc:204] T 4eea38c54a6b491081040e6059ca592e P dfd7ad6293204fc89f9e72436796698b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } I0703 01:41:57.613515 25738 ts_tablet_manager.cc:1073] T 4eea38c54a6b491081040e6059ca592e P dfd7ad6293204fc89f9e72436796698b: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s I0703 01:41:57.614761 24796 catalog_manager.cc:3798] T 4eea38c54a6b491081040e6059ca592e P dfd7ad6293204fc89f9e72436796698b reported cstate change: term changed from 0 to 1, leader changed from to dfd7ad6293204fc89f9e72436796698b (127.5.240.193). New cstate: current_term: 1 leader_uuid: "dfd7ad6293204fc89f9e72436796698b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } } I0703 01:41:57.615427 25737 raft_consensus.cc:479] T 9a818723ff79493aa17e7434040e8975 P 8ec1760d41f842f398d90e47ab52973b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } I0703 01:41:57.615561 25737 leader_election.cc:270] T 9a818723ff79493aa17e7434040e8975 P 8ec1760d41f842f398d90e47ab52973b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:41:57.615769 25741 raft_consensus.cc:2730] T 9a818723ff79493aa17e7434040e8975 P 8ec1760d41f842f398d90e47ab52973b: Snoozing failure detection for 1.539s (election complete) I0703 01:41:57.615842 25741 raft_consensus.cc:2537] T 9a818723ff79493aa17e7434040e8975 P 8ec1760d41f842f398d90e47ab52973b [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:41:57.616329 25741 raft_consensus.cc:593] T 9a818723ff79493aa17e7434040e8975 P 8ec1760d41f842f398d90e47ab52973b [term 1 LEADER]: Becoming Leader. State: Replica: 8ec1760d41f842f398d90e47ab52973b, State: Running, Role: LEADER I0703 01:41:57.616565 25741 consensus_queue.cc:204] T 9a818723ff79493aa17e7434040e8975 P 8ec1760d41f842f398d90e47ab52973b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } I0703 01:41:57.618428 24796 catalog_manager.cc:3798] T 9a818723ff79493aa17e7434040e8975 P 8ec1760d41f842f398d90e47ab52973b reported cstate change: term changed from 0 to 1, leader changed from to 8ec1760d41f842f398d90e47ab52973b (127.5.240.194). New cstate: current_term: 1 leader_uuid: "8ec1760d41f842f398d90e47ab52973b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } } I0703 01:41:57.619673 25737 ts_tablet_manager.cc:1073] T 9a818723ff79493aa17e7434040e8975 P 8ec1760d41f842f398d90e47ab52973b: Time spent starting tablet: real 0.008s user 0.004s sys 0.003s I0703 01:41:57.619786 25737 ts_tablet_manager.cc:1028] T fd7030608be84fa9b61c3ae90e04e27e P 8ec1760d41f842f398d90e47ab52973b: Bootstrapping tablet I0703 01:41:57.619901 25737 tablet_bootstrap.cc:427] T fd7030608be84fa9b61c3ae90e04e27e P 8ec1760d41f842f398d90e47ab52973b: Bootstrap starting. I0703 01:41:57.621091 25737 tablet_bootstrap.cc:578] T fd7030608be84fa9b61c3ae90e04e27e P 8ec1760d41f842f398d90e47ab52973b: No blocks or log segments found. Creating new log. I0703 01:41:57.623203 25737 tablet_bootstrap.cc:427] T fd7030608be84fa9b61c3ae90e04e27e P 8ec1760d41f842f398d90e47ab52973b: No bootstrap required, opened a new log I0703 01:41:57.623317 25737 ts_tablet_manager.cc:1045] T fd7030608be84fa9b61c3ae90e04e27e P 8ec1760d41f842f398d90e47ab52973b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s I0703 01:41:57.624161 25737 raft_consensus.cc:325] T fd7030608be84fa9b61c3ae90e04e27e P 8ec1760d41f842f398d90e47ab52973b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } I0703 01:41:57.624250 25737 raft_consensus.cc:351] T fd7030608be84fa9b61c3ae90e04e27e P 8ec1760d41f842f398d90e47ab52973b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:41:57.624302 25737 raft_consensus.cc:628] T fd7030608be84fa9b61c3ae90e04e27e P 8ec1760d41f842f398d90e47ab52973b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ec1760d41f842f398d90e47ab52973b, State: Initialized, Role: FOLLOWER I0703 01:41:57.624439 25737 consensus_queue.cc:227] T fd7030608be84fa9b61c3ae90e04e27e P 8ec1760d41f842f398d90e47ab52973b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } I0703 01:41:57.624547 25737 raft_consensus.cc:365] T fd7030608be84fa9b61c3ae90e04e27e P 8ec1760d41f842f398d90e47ab52973b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:41:57.624595 25737 raft_consensus.cc:457] T fd7030608be84fa9b61c3ae90e04e27e P 8ec1760d41f842f398d90e47ab52973b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:41:57.624663 25737 raft_consensus.cc:2730] T fd7030608be84fa9b61c3ae90e04e27e P 8ec1760d41f842f398d90e47ab52973b: Snoozing failure detection for 1.639s (starting election) I0703 01:41:57.624714 25737 raft_consensus.cc:2782] T fd7030608be84fa9b61c3ae90e04e27e P 8ec1760d41f842f398d90e47ab52973b [term 0 FOLLOWER]: Advancing to term 1 I0703 01:41:57.625927 25737 raft_consensus.cc:479] T fd7030608be84fa9b61c3ae90e04e27e P 8ec1760d41f842f398d90e47ab52973b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } I0703 01:41:57.626080 25737 leader_election.cc:270] T fd7030608be84fa9b61c3ae90e04e27e P 8ec1760d41f842f398d90e47ab52973b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:41:57.626310 25777 raft_consensus.cc:2730] T fd7030608be84fa9b61c3ae90e04e27e P 8ec1760d41f842f398d90e47ab52973b: Snoozing failure detection for 1.647s (election complete) I0703 01:41:57.626503 25777 raft_consensus.cc:2537] T fd7030608be84fa9b61c3ae90e04e27e P 8ec1760d41f842f398d90e47ab52973b [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:41:57.626827 25737 ts_tablet_manager.cc:1073] T fd7030608be84fa9b61c3ae90e04e27e P 8ec1760d41f842f398d90e47ab52973b: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s I0703 01:41:57.626827 25777 raft_consensus.cc:593] T fd7030608be84fa9b61c3ae90e04e27e P 8ec1760d41f842f398d90e47ab52973b [term 1 LEADER]: Becoming Leader. State: Replica: 8ec1760d41f842f398d90e47ab52973b, State: Running, Role: LEADER I0703 01:41:57.627444 25777 consensus_queue.cc:204] T fd7030608be84fa9b61c3ae90e04e27e P 8ec1760d41f842f398d90e47ab52973b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } I0703 01:41:57.629712 24796 catalog_manager.cc:3798] T fd7030608be84fa9b61c3ae90e04e27e P 8ec1760d41f842f398d90e47ab52973b reported cstate change: term changed from 0 to 1, leader changed from to 8ec1760d41f842f398d90e47ab52973b (127.5.240.194). New cstate: current_term: 1 leader_uuid: "8ec1760d41f842f398d90e47ab52973b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } } W0703 01:41:57.849817 25781 flags.cc:406] Enabled unsafe flag: --never_fsync=true Using table 'rebalance_test_table_1' Generator report time total : 123.3 ms time per row: 0.06165 ms I0703 01:41:58.065250 24798 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.5.240.254:56497: name: "rebalance_test_table_2" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 1 split_rows_range_bounds { } partition_schema { hash_bucket_schemas { columns { name: "key" } num_buckets: 3 seed: 0 } } I0703 01:41:58.071259 24899 tablet_service.cc:759] Processing CreateTablet for tablet 7d1747555a45487e869ed52665686c52 (table=rebalance_test_table_2 [id=92effdbc87a0486eab9ab6de8407a54f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I0703 01:41:58.071468 25022 tablet_service.cc:759] Processing CreateTablet for tablet ee4327166db34f66baa031d7e2da4999 (table=rebalance_test_table_2 [id=92effdbc87a0486eab9ab6de8407a54f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I0703 01:41:58.071647 24899 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 7d1747555a45487e869ed52665686c52. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:41:58.071768 25022 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet ee4327166db34f66baa031d7e2da4999. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:41:58.072794 24900 tablet_service.cc:759] Processing CreateTablet for tablet 0a2438297555462aa42cb91089c7a618 (table=rebalance_test_table_2 [id=92effdbc87a0486eab9ab6de8407a54f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I0703 01:41:58.073101 24900 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 0a2438297555462aa42cb91089c7a618. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:41:58.076274 24899 ts_tablet_manager.cc:1173] T 7d1747555a45487e869ed52665686c52 P dfd7ad6293204fc89f9e72436796698b: Registered tablet (data state: TABLET_DATA_READY) I0703 01:41:58.076519 25738 ts_tablet_manager.cc:1028] T 7d1747555a45487e869ed52665686c52 P dfd7ad6293204fc89f9e72436796698b: Bootstrapping tablet I0703 01:41:58.076522 25022 ts_tablet_manager.cc:1173] T ee4327166db34f66baa031d7e2da4999 P 8ec1760d41f842f398d90e47ab52973b: Registered tablet (data state: TABLET_DATA_READY) I0703 01:41:58.076696 25738 tablet_bootstrap.cc:427] T 7d1747555a45487e869ed52665686c52 P dfd7ad6293204fc89f9e72436796698b: Bootstrap starting. I0703 01:41:58.076741 25737 ts_tablet_manager.cc:1028] T ee4327166db34f66baa031d7e2da4999 P 8ec1760d41f842f398d90e47ab52973b: Bootstrapping tablet I0703 01:41:58.076891 25737 tablet_bootstrap.cc:427] T ee4327166db34f66baa031d7e2da4999 P 8ec1760d41f842f398d90e47ab52973b: Bootstrap starting. I0703 01:41:58.077867 25737 tablet_bootstrap.cc:578] T ee4327166db34f66baa031d7e2da4999 P 8ec1760d41f842f398d90e47ab52973b: No blocks or log segments found. Creating new log. I0703 01:41:58.078021 25738 tablet_bootstrap.cc:578] T 7d1747555a45487e869ed52665686c52 P dfd7ad6293204fc89f9e72436796698b: No blocks or log segments found. Creating new log. I0703 01:41:58.079949 25737 tablet_bootstrap.cc:427] T ee4327166db34f66baa031d7e2da4999 P 8ec1760d41f842f398d90e47ab52973b: No bootstrap required, opened a new log I0703 01:41:58.080054 25737 ts_tablet_manager.cc:1045] T ee4327166db34f66baa031d7e2da4999 P 8ec1760d41f842f398d90e47ab52973b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s I0703 01:41:58.080147 24900 ts_tablet_manager.cc:1173] T 0a2438297555462aa42cb91089c7a618 P dfd7ad6293204fc89f9e72436796698b: Registered tablet (data state: TABLET_DATA_READY) I0703 01:41:58.080281 25738 tablet_bootstrap.cc:427] T 7d1747555a45487e869ed52665686c52 P dfd7ad6293204fc89f9e72436796698b: No bootstrap required, opened a new log I0703 01:41:58.080407 25738 ts_tablet_manager.cc:1045] T 7d1747555a45487e869ed52665686c52 P dfd7ad6293204fc89f9e72436796698b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s I0703 01:41:58.080767 25737 raft_consensus.cc:325] T ee4327166db34f66baa031d7e2da4999 P 8ec1760d41f842f398d90e47ab52973b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } I0703 01:41:58.080909 25737 raft_consensus.cc:351] T ee4327166db34f66baa031d7e2da4999 P 8ec1760d41f842f398d90e47ab52973b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:41:58.080968 25737 raft_consensus.cc:628] T ee4327166db34f66baa031d7e2da4999 P 8ec1760d41f842f398d90e47ab52973b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ec1760d41f842f398d90e47ab52973b, State: Initialized, Role: FOLLOWER I0703 01:41:58.081151 25738 raft_consensus.cc:325] T 7d1747555a45487e869ed52665686c52 P dfd7ad6293204fc89f9e72436796698b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } I0703 01:41:58.081156 25737 consensus_queue.cc:227] T ee4327166db34f66baa031d7e2da4999 P 8ec1760d41f842f398d90e47ab52973b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } I0703 01:41:58.081279 25738 raft_consensus.cc:351] T 7d1747555a45487e869ed52665686c52 P dfd7ad6293204fc89f9e72436796698b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:41:58.081295 25737 raft_consensus.cc:365] T ee4327166db34f66baa031d7e2da4999 P 8ec1760d41f842f398d90e47ab52973b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:41:58.081331 25738 raft_consensus.cc:628] T 7d1747555a45487e869ed52665686c52 P dfd7ad6293204fc89f9e72436796698b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfd7ad6293204fc89f9e72436796698b, State: Initialized, Role: FOLLOWER I0703 01:41:58.081342 25737 raft_consensus.cc:457] T ee4327166db34f66baa031d7e2da4999 P 8ec1760d41f842f398d90e47ab52973b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:41:58.081396 25737 raft_consensus.cc:2730] T ee4327166db34f66baa031d7e2da4999 P 8ec1760d41f842f398d90e47ab52973b: Snoozing failure detection for 1.596s (starting election) I0703 01:41:58.081452 25737 raft_consensus.cc:2782] T ee4327166db34f66baa031d7e2da4999 P 8ec1760d41f842f398d90e47ab52973b [term 0 FOLLOWER]: Advancing to term 1 I0703 01:41:58.081521 25738 consensus_queue.cc:227] T 7d1747555a45487e869ed52665686c52 P dfd7ad6293204fc89f9e72436796698b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } I0703 01:41:58.081679 25738 raft_consensus.cc:365] T 7d1747555a45487e869ed52665686c52 P dfd7ad6293204fc89f9e72436796698b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:41:58.081724 25738 raft_consensus.cc:457] T 7d1747555a45487e869ed52665686c52 P dfd7ad6293204fc89f9e72436796698b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:41:58.081779 25738 raft_consensus.cc:2730] T 7d1747555a45487e869ed52665686c52 P dfd7ad6293204fc89f9e72436796698b: Snoozing failure detection for 1.667s (starting election) I0703 01:41:58.081832 25738 raft_consensus.cc:2782] T 7d1747555a45487e869ed52665686c52 P dfd7ad6293204fc89f9e72436796698b [term 0 FOLLOWER]: Advancing to term 1 I0703 01:41:58.083395 25737 raft_consensus.cc:479] T ee4327166db34f66baa031d7e2da4999 P 8ec1760d41f842f398d90e47ab52973b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } I0703 01:41:58.083585 25737 leader_election.cc:270] T ee4327166db34f66baa031d7e2da4999 P 8ec1760d41f842f398d90e47ab52973b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:41:58.083784 25738 raft_consensus.cc:479] T 7d1747555a45487e869ed52665686c52 P dfd7ad6293204fc89f9e72436796698b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } I0703 01:41:58.083884 25777 raft_consensus.cc:2730] T ee4327166db34f66baa031d7e2da4999 P 8ec1760d41f842f398d90e47ab52973b: Snoozing failure detection for 1.507s (election complete) I0703 01:41:58.084007 25777 raft_consensus.cc:2537] T ee4327166db34f66baa031d7e2da4999 P 8ec1760d41f842f398d90e47ab52973b [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:41:58.084022 25738 leader_election.cc:270] T 7d1747555a45487e869ed52665686c52 P dfd7ad6293204fc89f9e72436796698b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:41:58.084264 25743 raft_consensus.cc:2730] T 7d1747555a45487e869ed52665686c52 P dfd7ad6293204fc89f9e72436796698b: Snoozing failure detection for 1.794s (election complete) I0703 01:41:58.084272 25777 raft_consensus.cc:593] T ee4327166db34f66baa031d7e2da4999 P 8ec1760d41f842f398d90e47ab52973b [term 1 LEADER]: Becoming Leader. State: Replica: 8ec1760d41f842f398d90e47ab52973b, State: Running, Role: LEADER I0703 01:41:58.084358 25743 raft_consensus.cc:2537] T 7d1747555a45487e869ed52665686c52 P dfd7ad6293204fc89f9e72436796698b [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:41:58.084493 25743 raft_consensus.cc:593] T 7d1747555a45487e869ed52665686c52 P dfd7ad6293204fc89f9e72436796698b [term 1 LEADER]: Becoming Leader. State: Replica: dfd7ad6293204fc89f9e72436796698b, State: Running, Role: LEADER I0703 01:41:58.084441 25777 consensus_queue.cc:204] T ee4327166db34f66baa031d7e2da4999 P 8ec1760d41f842f398d90e47ab52973b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } I0703 01:41:58.084698 25743 consensus_queue.cc:204] T 7d1747555a45487e869ed52665686c52 P dfd7ad6293204fc89f9e72436796698b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } I0703 01:41:58.084977 25737 ts_tablet_manager.cc:1073] T ee4327166db34f66baa031d7e2da4999 P 8ec1760d41f842f398d90e47ab52973b: Time spent starting tablet: real 0.005s user 0.002s sys 0.002s I0703 01:41:58.085886 25738 ts_tablet_manager.cc:1073] T 7d1747555a45487e869ed52665686c52 P dfd7ad6293204fc89f9e72436796698b: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s I0703 01:41:58.086079 25738 ts_tablet_manager.cc:1028] T 0a2438297555462aa42cb91089c7a618 P dfd7ad6293204fc89f9e72436796698b: Bootstrapping tablet I0703 01:41:58.086757 24798 catalog_manager.cc:3798] T 7d1747555a45487e869ed52665686c52 P dfd7ad6293204fc89f9e72436796698b reported cstate change: term changed from 0 to 1, leader changed from to dfd7ad6293204fc89f9e72436796698b (127.5.240.193). New cstate: current_term: 1 leader_uuid: "dfd7ad6293204fc89f9e72436796698b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } } I0703 01:41:58.087425 25738 tablet_bootstrap.cc:427] T 0a2438297555462aa42cb91089c7a618 P dfd7ad6293204fc89f9e72436796698b: Bootstrap starting. I0703 01:41:58.087419 24794 catalog_manager.cc:3798] T ee4327166db34f66baa031d7e2da4999 P 8ec1760d41f842f398d90e47ab52973b reported cstate change: term changed from 0 to 1, leader changed from to 8ec1760d41f842f398d90e47ab52973b (127.5.240.194). New cstate: current_term: 1 leader_uuid: "8ec1760d41f842f398d90e47ab52973b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } } I0703 01:41:58.089699 25738 tablet_bootstrap.cc:578] T 0a2438297555462aa42cb91089c7a618 P dfd7ad6293204fc89f9e72436796698b: No blocks or log segments found. Creating new log. I0703 01:41:58.091655 25738 tablet_bootstrap.cc:427] T 0a2438297555462aa42cb91089c7a618 P dfd7ad6293204fc89f9e72436796698b: No bootstrap required, opened a new log I0703 01:41:58.091773 25738 ts_tablet_manager.cc:1045] T 0a2438297555462aa42cb91089c7a618 P dfd7ad6293204fc89f9e72436796698b: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s I0703 01:41:58.092310 25738 raft_consensus.cc:325] T 0a2438297555462aa42cb91089c7a618 P dfd7ad6293204fc89f9e72436796698b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } I0703 01:41:58.092402 25738 raft_consensus.cc:351] T 0a2438297555462aa42cb91089c7a618 P dfd7ad6293204fc89f9e72436796698b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:41:58.092456 25738 raft_consensus.cc:628] T 0a2438297555462aa42cb91089c7a618 P dfd7ad6293204fc89f9e72436796698b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfd7ad6293204fc89f9e72436796698b, State: Initialized, Role: FOLLOWER I0703 01:41:58.092643 25738 consensus_queue.cc:227] T 0a2438297555462aa42cb91089c7a618 P dfd7ad6293204fc89f9e72436796698b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } I0703 01:41:58.092759 25738 raft_consensus.cc:365] T 0a2438297555462aa42cb91089c7a618 P dfd7ad6293204fc89f9e72436796698b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:41:58.092813 25738 raft_consensus.cc:457] T 0a2438297555462aa42cb91089c7a618 P dfd7ad6293204fc89f9e72436796698b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:41:58.092866 25738 raft_consensus.cc:2730] T 0a2438297555462aa42cb91089c7a618 P dfd7ad6293204fc89f9e72436796698b: Snoozing failure detection for 1.685s (starting election) I0703 01:41:58.092919 25738 raft_consensus.cc:2782] T 0a2438297555462aa42cb91089c7a618 P dfd7ad6293204fc89f9e72436796698b [term 0 FOLLOWER]: Advancing to term 1 I0703 01:41:58.094144 25738 raft_consensus.cc:479] T 0a2438297555462aa42cb91089c7a618 P dfd7ad6293204fc89f9e72436796698b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } I0703 01:41:58.094287 25738 leader_election.cc:270] T 0a2438297555462aa42cb91089c7a618 P dfd7ad6293204fc89f9e72436796698b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:41:58.094529 25743 raft_consensus.cc:2730] T 0a2438297555462aa42cb91089c7a618 P dfd7ad6293204fc89f9e72436796698b: Snoozing failure detection for 2.109s (election complete) I0703 01:41:58.094650 25743 raft_consensus.cc:2537] T 0a2438297555462aa42cb91089c7a618 P dfd7ad6293204fc89f9e72436796698b [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:41:58.094758 25738 ts_tablet_manager.cc:1073] T 0a2438297555462aa42cb91089c7a618 P dfd7ad6293204fc89f9e72436796698b: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s I0703 01:41:58.094871 25743 raft_consensus.cc:593] T 0a2438297555462aa42cb91089c7a618 P dfd7ad6293204fc89f9e72436796698b [term 1 LEADER]: Becoming Leader. State: Replica: dfd7ad6293204fc89f9e72436796698b, State: Running, Role: LEADER I0703 01:41:58.095016 25743 consensus_queue.cc:204] T 0a2438297555462aa42cb91089c7a618 P dfd7ad6293204fc89f9e72436796698b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } I0703 01:41:58.096571 24794 catalog_manager.cc:3798] T 0a2438297555462aa42cb91089c7a618 P dfd7ad6293204fc89f9e72436796698b reported cstate change: term changed from 0 to 1, leader changed from to dfd7ad6293204fc89f9e72436796698b (127.5.240.193). New cstate: current_term: 1 leader_uuid: "dfd7ad6293204fc89f9e72436796698b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } } W0703 01:41:58.324353 25801 flags.cc:406] Enabled unsafe flag: --never_fsync=true Using table 'rebalance_test_table_2' Generator report time total : 121.429 ms time per row: 0.0607144 ms I0703 01:41:58.538627 24792 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.5.240.254:56497: name: "rebalance_test_table_3" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 1 split_rows_range_bounds { } partition_schema { hash_bucket_schemas { columns { name: "key" } num_buckets: 3 seed: 0 } } I0703 01:41:58.544878 24900 tablet_service.cc:759] Processing CreateTablet for tablet dcabbf8c71c14f248522b11a1426555d (table=rebalance_test_table_3 [id=72ffb8cd105e4058a3b2caf04d27ee7f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I0703 01:41:58.544922 25022 tablet_service.cc:759] Processing CreateTablet for tablet dae8746553524b6a9eb2e3f177f658f1 (table=rebalance_test_table_3 [id=72ffb8cd105e4058a3b2caf04d27ee7f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I0703 01:41:58.545065 25023 tablet_service.cc:759] Processing CreateTablet for tablet 72ebab6b7393457db959684913708522 (table=rebalance_test_table_3 [id=72ffb8cd105e4058a3b2caf04d27ee7f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I0703 01:41:58.545179 24900 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet dcabbf8c71c14f248522b11a1426555d. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:41:58.545338 25022 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet dae8746553524b6a9eb2e3f177f658f1. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:41:58.545543 25023 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 72ebab6b7393457db959684913708522. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:41:58.548663 24900 ts_tablet_manager.cc:1173] T dcabbf8c71c14f248522b11a1426555d P dfd7ad6293204fc89f9e72436796698b: Registered tablet (data state: TABLET_DATA_READY) I0703 01:41:58.548830 25738 ts_tablet_manager.cc:1028] T dcabbf8c71c14f248522b11a1426555d P dfd7ad6293204fc89f9e72436796698b: Bootstrapping tablet I0703 01:41:58.548985 25738 tablet_bootstrap.cc:427] T dcabbf8c71c14f248522b11a1426555d P dfd7ad6293204fc89f9e72436796698b: Bootstrap starting. I0703 01:41:58.549970 25738 tablet_bootstrap.cc:578] T dcabbf8c71c14f248522b11a1426555d P dfd7ad6293204fc89f9e72436796698b: No blocks or log segments found. Creating new log. I0703 01:41:58.550879 25022 ts_tablet_manager.cc:1173] T dae8746553524b6a9eb2e3f177f658f1 P 8ec1760d41f842f398d90e47ab52973b: Registered tablet (data state: TABLET_DATA_READY) I0703 01:41:58.551051 25737 ts_tablet_manager.cc:1028] T dae8746553524b6a9eb2e3f177f658f1 P 8ec1760d41f842f398d90e47ab52973b: Bootstrapping tablet I0703 01:41:58.551196 25737 tablet_bootstrap.cc:427] T dae8746553524b6a9eb2e3f177f658f1 P 8ec1760d41f842f398d90e47ab52973b: Bootstrap starting. I0703 01:41:58.551371 25023 ts_tablet_manager.cc:1173] T 72ebab6b7393457db959684913708522 P 8ec1760d41f842f398d90e47ab52973b: Registered tablet (data state: TABLET_DATA_READY) I0703 01:41:58.551399 25738 tablet_bootstrap.cc:427] T dcabbf8c71c14f248522b11a1426555d P dfd7ad6293204fc89f9e72436796698b: No bootstrap required, opened a new log I0703 01:41:58.551481 25738 ts_tablet_manager.cc:1045] T dcabbf8c71c14f248522b11a1426555d P dfd7ad6293204fc89f9e72436796698b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s I0703 01:41:58.552014 25738 raft_consensus.cc:325] T dcabbf8c71c14f248522b11a1426555d P dfd7ad6293204fc89f9e72436796698b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } I0703 01:41:58.552125 25738 raft_consensus.cc:351] T dcabbf8c71c14f248522b11a1426555d P dfd7ad6293204fc89f9e72436796698b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:41:58.552179 25738 raft_consensus.cc:628] T dcabbf8c71c14f248522b11a1426555d P dfd7ad6293204fc89f9e72436796698b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfd7ad6293204fc89f9e72436796698b, State: Initialized, Role: FOLLOWER I0703 01:41:58.552356 25738 consensus_queue.cc:227] T dcabbf8c71c14f248522b11a1426555d P dfd7ad6293204fc89f9e72436796698b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } I0703 01:41:58.552460 25738 raft_consensus.cc:365] T dcabbf8c71c14f248522b11a1426555d P dfd7ad6293204fc89f9e72436796698b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:41:58.552494 25738 raft_consensus.cc:457] T dcabbf8c71c14f248522b11a1426555d P dfd7ad6293204fc89f9e72436796698b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:41:58.552553 25738 raft_consensus.cc:2730] T dcabbf8c71c14f248522b11a1426555d P dfd7ad6293204fc89f9e72436796698b: Snoozing failure detection for 1.670s (starting election) I0703 01:41:58.552600 25738 raft_consensus.cc:2782] T dcabbf8c71c14f248522b11a1426555d P dfd7ad6293204fc89f9e72436796698b [term 0 FOLLOWER]: Advancing to term 1 I0703 01:41:58.552920 25737 tablet_bootstrap.cc:578] T dae8746553524b6a9eb2e3f177f658f1 P 8ec1760d41f842f398d90e47ab52973b: No blocks or log segments found. Creating new log. I0703 01:41:58.554144 25738 raft_consensus.cc:479] T dcabbf8c71c14f248522b11a1426555d P dfd7ad6293204fc89f9e72436796698b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } I0703 01:41:58.554311 25738 leader_election.cc:270] T dcabbf8c71c14f248522b11a1426555d P dfd7ad6293204fc89f9e72436796698b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:41:58.554415 25737 tablet_bootstrap.cc:427] T dae8746553524b6a9eb2e3f177f658f1 P 8ec1760d41f842f398d90e47ab52973b: No bootstrap required, opened a new log I0703 01:41:58.554510 25737 ts_tablet_manager.cc:1045] T dae8746553524b6a9eb2e3f177f658f1 P 8ec1760d41f842f398d90e47ab52973b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s I0703 01:41:58.554563 25743 raft_consensus.cc:2730] T dcabbf8c71c14f248522b11a1426555d P dfd7ad6293204fc89f9e72436796698b: Snoozing failure detection for 1.827s (election complete) I0703 01:41:58.554661 25743 raft_consensus.cc:2537] T dcabbf8c71c14f248522b11a1426555d P dfd7ad6293204fc89f9e72436796698b [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:41:58.554898 25738 ts_tablet_manager.cc:1073] T dcabbf8c71c14f248522b11a1426555d P dfd7ad6293204fc89f9e72436796698b: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s I0703 01:41:58.554898 25743 raft_consensus.cc:593] T dcabbf8c71c14f248522b11a1426555d P dfd7ad6293204fc89f9e72436796698b [term 1 LEADER]: Becoming Leader. State: Replica: dfd7ad6293204fc89f9e72436796698b, State: Running, Role: LEADER I0703 01:41:58.555081 25743 consensus_queue.cc:204] T dcabbf8c71c14f248522b11a1426555d P dfd7ad6293204fc89f9e72436796698b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } I0703 01:41:58.555158 25737 raft_consensus.cc:325] T dae8746553524b6a9eb2e3f177f658f1 P 8ec1760d41f842f398d90e47ab52973b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } I0703 01:41:58.555243 25737 raft_consensus.cc:351] T dae8746553524b6a9eb2e3f177f658f1 P 8ec1760d41f842f398d90e47ab52973b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:41:58.555277 25737 raft_consensus.cc:628] T dae8746553524b6a9eb2e3f177f658f1 P 8ec1760d41f842f398d90e47ab52973b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ec1760d41f842f398d90e47ab52973b, State: Initialized, Role: FOLLOWER I0703 01:41:58.555416 25737 consensus_queue.cc:227] T dae8746553524b6a9eb2e3f177f658f1 P 8ec1760d41f842f398d90e47ab52973b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } I0703 01:41:58.555555 25737 raft_consensus.cc:365] T dae8746553524b6a9eb2e3f177f658f1 P 8ec1760d41f842f398d90e47ab52973b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:41:58.555595 25737 raft_consensus.cc:457] T dae8746553524b6a9eb2e3f177f658f1 P 8ec1760d41f842f398d90e47ab52973b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:41:58.555649 25737 raft_consensus.cc:2730] T dae8746553524b6a9eb2e3f177f658f1 P 8ec1760d41f842f398d90e47ab52973b: Snoozing failure detection for 2.210s (starting election) I0703 01:41:58.555689 25737 raft_consensus.cc:2782] T dae8746553524b6a9eb2e3f177f658f1 P 8ec1760d41f842f398d90e47ab52973b [term 0 FOLLOWER]: Advancing to term 1 I0703 01:41:58.556877 24792 catalog_manager.cc:3798] T dcabbf8c71c14f248522b11a1426555d P dfd7ad6293204fc89f9e72436796698b reported cstate change: term changed from 0 to 1, leader changed from to dfd7ad6293204fc89f9e72436796698b (127.5.240.193). New cstate: current_term: 1 leader_uuid: "dfd7ad6293204fc89f9e72436796698b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } } I0703 01:41:58.557179 25737 raft_consensus.cc:479] T dae8746553524b6a9eb2e3f177f658f1 P 8ec1760d41f842f398d90e47ab52973b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } I0703 01:41:58.557395 25737 leader_election.cc:270] T dae8746553524b6a9eb2e3f177f658f1 P 8ec1760d41f842f398d90e47ab52973b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:41:58.557684 25777 raft_consensus.cc:2730] T dae8746553524b6a9eb2e3f177f658f1 P 8ec1760d41f842f398d90e47ab52973b: Snoozing failure detection for 1.769s (election complete) I0703 01:41:58.557806 25777 raft_consensus.cc:2537] T dae8746553524b6a9eb2e3f177f658f1 P 8ec1760d41f842f398d90e47ab52973b [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:41:58.558286 25777 raft_consensus.cc:593] T dae8746553524b6a9eb2e3f177f658f1 P 8ec1760d41f842f398d90e47ab52973b [term 1 LEADER]: Becoming Leader. State: Replica: 8ec1760d41f842f398d90e47ab52973b, State: Running, Role: LEADER I0703 01:41:58.558136 25737 ts_tablet_manager.cc:1073] T dae8746553524b6a9eb2e3f177f658f1 P 8ec1760d41f842f398d90e47ab52973b: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s I0703 01:41:58.558599 25737 ts_tablet_manager.cc:1028] T 72ebab6b7393457db959684913708522 P 8ec1760d41f842f398d90e47ab52973b: Bootstrapping tablet I0703 01:41:58.558502 25777 consensus_queue.cc:204] T dae8746553524b6a9eb2e3f177f658f1 P 8ec1760d41f842f398d90e47ab52973b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } I0703 01:41:58.558985 25737 tablet_bootstrap.cc:427] T 72ebab6b7393457db959684913708522 P 8ec1760d41f842f398d90e47ab52973b: Bootstrap starting. I0703 01:41:58.560349 24792 catalog_manager.cc:3798] T dae8746553524b6a9eb2e3f177f658f1 P 8ec1760d41f842f398d90e47ab52973b reported cstate change: term changed from 0 to 1, leader changed from to 8ec1760d41f842f398d90e47ab52973b (127.5.240.194). New cstate: current_term: 1 leader_uuid: "8ec1760d41f842f398d90e47ab52973b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } } I0703 01:41:58.560916 25737 tablet_bootstrap.cc:578] T 72ebab6b7393457db959684913708522 P 8ec1760d41f842f398d90e47ab52973b: No blocks or log segments found. Creating new log. I0703 01:41:58.562620 25737 tablet_bootstrap.cc:427] T 72ebab6b7393457db959684913708522 P 8ec1760d41f842f398d90e47ab52973b: No bootstrap required, opened a new log I0703 01:41:58.562739 25737 ts_tablet_manager.cc:1045] T 72ebab6b7393457db959684913708522 P 8ec1760d41f842f398d90e47ab52973b: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s I0703 01:41:58.563289 25737 raft_consensus.cc:325] T 72ebab6b7393457db959684913708522 P 8ec1760d41f842f398d90e47ab52973b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } I0703 01:41:58.563444 25737 raft_consensus.cc:351] T 72ebab6b7393457db959684913708522 P 8ec1760d41f842f398d90e47ab52973b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:41:58.563494 25737 raft_consensus.cc:628] T 72ebab6b7393457db959684913708522 P 8ec1760d41f842f398d90e47ab52973b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ec1760d41f842f398d90e47ab52973b, State: Initialized, Role: FOLLOWER I0703 01:41:58.563670 25737 consensus_queue.cc:227] T 72ebab6b7393457db959684913708522 P 8ec1760d41f842f398d90e47ab52973b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } I0703 01:41:58.563828 25737 raft_consensus.cc:365] T 72ebab6b7393457db959684913708522 P 8ec1760d41f842f398d90e47ab52973b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:41:58.563902 25737 raft_consensus.cc:457] T 72ebab6b7393457db959684913708522 P 8ec1760d41f842f398d90e47ab52973b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:41:58.563982 25737 raft_consensus.cc:2730] T 72ebab6b7393457db959684913708522 P 8ec1760d41f842f398d90e47ab52973b: Snoozing failure detection for 1.640s (starting election) I0703 01:41:58.564061 25737 raft_consensus.cc:2782] T 72ebab6b7393457db959684913708522 P 8ec1760d41f842f398d90e47ab52973b [term 0 FOLLOWER]: Advancing to term 1 I0703 01:41:58.565419 25737 raft_consensus.cc:479] T 72ebab6b7393457db959684913708522 P 8ec1760d41f842f398d90e47ab52973b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } I0703 01:41:58.565559 25737 leader_election.cc:270] T 72ebab6b7393457db959684913708522 P 8ec1760d41f842f398d90e47ab52973b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:41:58.565850 25777 raft_consensus.cc:2730] T 72ebab6b7393457db959684913708522 P 8ec1760d41f842f398d90e47ab52973b: Snoozing failure detection for 1.744s (election complete) I0703 01:41:58.565958 25777 raft_consensus.cc:2537] T 72ebab6b7393457db959684913708522 P 8ec1760d41f842f398d90e47ab52973b [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:41:58.566128 25777 raft_consensus.cc:593] T 72ebab6b7393457db959684913708522 P 8ec1760d41f842f398d90e47ab52973b [term 1 LEADER]: Becoming Leader. State: Replica: 8ec1760d41f842f398d90e47ab52973b, State: Running, Role: LEADER I0703 01:41:58.566253 25737 ts_tablet_manager.cc:1073] T 72ebab6b7393457db959684913708522 P 8ec1760d41f842f398d90e47ab52973b: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s I0703 01:41:58.566324 25777 consensus_queue.cc:204] T 72ebab6b7393457db959684913708522 P 8ec1760d41f842f398d90e47ab52973b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } I0703 01:41:58.567754 24792 catalog_manager.cc:3798] T 72ebab6b7393457db959684913708522 P 8ec1760d41f842f398d90e47ab52973b reported cstate change: term changed from 0 to 1, leader changed from to 8ec1760d41f842f398d90e47ab52973b (127.5.240.194). New cstate: current_term: 1 leader_uuid: "8ec1760d41f842f398d90e47ab52973b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } } W0703 01:41:58.787382 25821 flags.cc:406] Enabled unsafe flag: --never_fsync=true Using table 'rebalance_test_table_3' Generator report time total : 126.766 ms time per row: 0.0633828 ms I0703 01:41:59.005741 24792 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.5.240.254:56497: name: "rebalance_test_table_4" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 1 split_rows_range_bounds { } partition_schema { hash_bucket_schemas { columns { name: "key" } num_buckets: 3 seed: 0 } } I0703 01:41:59.011327 24900 tablet_service.cc:759] Processing CreateTablet for tablet 53242d0289d644faa0149e0a54c0984c (table=rebalance_test_table_4 [id=199091349c4b44e3a05e6e999c26968e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I0703 01:41:59.011440 24899 tablet_service.cc:759] Processing CreateTablet for tablet 93faa898182e46b2a6aab9df75230c0b (table=rebalance_test_table_4 [id=199091349c4b44e3a05e6e999c26968e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I0703 01:41:59.011713 24900 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 53242d0289d644faa0149e0a54c0984c. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:41:59.011898 24899 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 93faa898182e46b2a6aab9df75230c0b. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:41:59.012070 25022 tablet_service.cc:759] Processing CreateTablet for tablet c7f0c202bc0c4351afc57e7efbbd5e02 (table=rebalance_test_table_4 [id=199091349c4b44e3a05e6e999c26968e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I0703 01:41:59.012398 25022 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet c7f0c202bc0c4351afc57e7efbbd5e02. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:41:59.016129 24900 ts_tablet_manager.cc:1173] T 53242d0289d644faa0149e0a54c0984c P dfd7ad6293204fc89f9e72436796698b: Registered tablet (data state: TABLET_DATA_READY) I0703 01:41:59.016330 25738 ts_tablet_manager.cc:1028] T 53242d0289d644faa0149e0a54c0984c P dfd7ad6293204fc89f9e72436796698b: Bootstrapping tablet I0703 01:41:59.016578 25738 tablet_bootstrap.cc:427] T 53242d0289d644faa0149e0a54c0984c P dfd7ad6293204fc89f9e72436796698b: Bootstrap starting. I0703 01:41:59.017581 25738 tablet_bootstrap.cc:578] T 53242d0289d644faa0149e0a54c0984c P dfd7ad6293204fc89f9e72436796698b: No blocks or log segments found. Creating new log. I0703 01:41:59.017710 25022 ts_tablet_manager.cc:1173] T c7f0c202bc0c4351afc57e7efbbd5e02 P 8ec1760d41f842f398d90e47ab52973b: Registered tablet (data state: TABLET_DATA_READY) I0703 01:41:59.017901 25737 ts_tablet_manager.cc:1028] T c7f0c202bc0c4351afc57e7efbbd5e02 P 8ec1760d41f842f398d90e47ab52973b: Bootstrapping tablet I0703 01:41:59.018023 24899 ts_tablet_manager.cc:1173] T 93faa898182e46b2a6aab9df75230c0b P dfd7ad6293204fc89f9e72436796698b: Registered tablet (data state: TABLET_DATA_READY) I0703 01:41:59.018052 25737 tablet_bootstrap.cc:427] T c7f0c202bc0c4351afc57e7efbbd5e02 P 8ec1760d41f842f398d90e47ab52973b: Bootstrap starting. I0703 01:41:59.019182 25737 tablet_bootstrap.cc:578] T c7f0c202bc0c4351afc57e7efbbd5e02 P 8ec1760d41f842f398d90e47ab52973b: No blocks or log segments found. Creating new log. I0703 01:41:59.019315 25738 tablet_bootstrap.cc:427] T 53242d0289d644faa0149e0a54c0984c P dfd7ad6293204fc89f9e72436796698b: No bootstrap required, opened a new log I0703 01:41:59.019412 25738 ts_tablet_manager.cc:1045] T 53242d0289d644faa0149e0a54c0984c P dfd7ad6293204fc89f9e72436796698b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s I0703 01:41:59.020077 25738 raft_consensus.cc:325] T 53242d0289d644faa0149e0a54c0984c P dfd7ad6293204fc89f9e72436796698b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } I0703 01:41:59.020216 25738 raft_consensus.cc:351] T 53242d0289d644faa0149e0a54c0984c P dfd7ad6293204fc89f9e72436796698b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:41:59.020272 25738 raft_consensus.cc:628] T 53242d0289d644faa0149e0a54c0984c P dfd7ad6293204fc89f9e72436796698b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfd7ad6293204fc89f9e72436796698b, State: Initialized, Role: FOLLOWER I0703 01:41:59.020443 25738 consensus_queue.cc:227] T 53242d0289d644faa0149e0a54c0984c P dfd7ad6293204fc89f9e72436796698b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } I0703 01:41:59.020639 25738 raft_consensus.cc:365] T 53242d0289d644faa0149e0a54c0984c P dfd7ad6293204fc89f9e72436796698b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:41:59.020712 25738 raft_consensus.cc:457] T 53242d0289d644faa0149e0a54c0984c P dfd7ad6293204fc89f9e72436796698b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:41:59.020794 25738 raft_consensus.cc:2730] T 53242d0289d644faa0149e0a54c0984c P dfd7ad6293204fc89f9e72436796698b: Snoozing failure detection for 1.854s (starting election) I0703 01:41:59.020880 25737 tablet_bootstrap.cc:427] T c7f0c202bc0c4351afc57e7efbbd5e02 P 8ec1760d41f842f398d90e47ab52973b: No bootstrap required, opened a new log I0703 01:41:59.020880 25738 raft_consensus.cc:2782] T 53242d0289d644faa0149e0a54c0984c P dfd7ad6293204fc89f9e72436796698b [term 0 FOLLOWER]: Advancing to term 1 I0703 01:41:59.020982 25737 ts_tablet_manager.cc:1045] T c7f0c202bc0c4351afc57e7efbbd5e02 P 8ec1760d41f842f398d90e47ab52973b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s I0703 01:41:59.021731 25737 raft_consensus.cc:325] T c7f0c202bc0c4351afc57e7efbbd5e02 P 8ec1760d41f842f398d90e47ab52973b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } I0703 01:41:59.021905 25737 raft_consensus.cc:351] T c7f0c202bc0c4351afc57e7efbbd5e02 P 8ec1760d41f842f398d90e47ab52973b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:41:59.021981 25737 raft_consensus.cc:628] T c7f0c202bc0c4351afc57e7efbbd5e02 P 8ec1760d41f842f398d90e47ab52973b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ec1760d41f842f398d90e47ab52973b, State: Initialized, Role: FOLLOWER I0703 01:41:59.022208 25737 consensus_queue.cc:227] T c7f0c202bc0c4351afc57e7efbbd5e02 P 8ec1760d41f842f398d90e47ab52973b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } I0703 01:41:59.022383 25737 raft_consensus.cc:365] T c7f0c202bc0c4351afc57e7efbbd5e02 P 8ec1760d41f842f398d90e47ab52973b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:41:59.022450 25737 raft_consensus.cc:457] T c7f0c202bc0c4351afc57e7efbbd5e02 P 8ec1760d41f842f398d90e47ab52973b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:41:59.022524 25737 raft_consensus.cc:2730] T c7f0c202bc0c4351afc57e7efbbd5e02 P 8ec1760d41f842f398d90e47ab52973b: Snoozing failure detection for 2.067s (starting election) I0703 01:41:59.022606 25737 raft_consensus.cc:2782] T c7f0c202bc0c4351afc57e7efbbd5e02 P 8ec1760d41f842f398d90e47ab52973b [term 0 FOLLOWER]: Advancing to term 1 I0703 01:41:59.022830 25738 raft_consensus.cc:479] T 53242d0289d644faa0149e0a54c0984c P dfd7ad6293204fc89f9e72436796698b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } I0703 01:41:59.023037 25738 leader_election.cc:270] T 53242d0289d644faa0149e0a54c0984c P dfd7ad6293204fc89f9e72436796698b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:41:59.023308 25743 raft_consensus.cc:2730] T 53242d0289d644faa0149e0a54c0984c P dfd7ad6293204fc89f9e72436796698b: Snoozing failure detection for 2.191s (election complete) I0703 01:41:59.023427 25743 raft_consensus.cc:2537] T 53242d0289d644faa0149e0a54c0984c P dfd7ad6293204fc89f9e72436796698b [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:41:59.023612 25743 raft_consensus.cc:593] T 53242d0289d644faa0149e0a54c0984c P dfd7ad6293204fc89f9e72436796698b [term 1 LEADER]: Becoming Leader. State: Replica: dfd7ad6293204fc89f9e72436796698b, State: Running, Role: LEADER I0703 01:41:59.023892 25738 ts_tablet_manager.cc:1073] T 53242d0289d644faa0149e0a54c0984c P dfd7ad6293204fc89f9e72436796698b: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s I0703 01:41:59.023823 25743 consensus_queue.cc:204] T 53242d0289d644faa0149e0a54c0984c P dfd7ad6293204fc89f9e72436796698b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } I0703 01:41:59.024107 25738 ts_tablet_manager.cc:1028] T 93faa898182e46b2a6aab9df75230c0b P dfd7ad6293204fc89f9e72436796698b: Bootstrapping tablet I0703 01:41:59.024377 25738 tablet_bootstrap.cc:427] T 93faa898182e46b2a6aab9df75230c0b P dfd7ad6293204fc89f9e72436796698b: Bootstrap starting. I0703 01:41:59.025630 24792 catalog_manager.cc:3798] T 53242d0289d644faa0149e0a54c0984c P dfd7ad6293204fc89f9e72436796698b reported cstate change: term changed from 0 to 1, leader changed from to dfd7ad6293204fc89f9e72436796698b (127.5.240.193). New cstate: current_term: 1 leader_uuid: "dfd7ad6293204fc89f9e72436796698b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } } I0703 01:41:59.025938 25737 raft_consensus.cc:479] T c7f0c202bc0c4351afc57e7efbbd5e02 P 8ec1760d41f842f398d90e47ab52973b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } I0703 01:41:59.026070 25738 tablet_bootstrap.cc:578] T 93faa898182e46b2a6aab9df75230c0b P dfd7ad6293204fc89f9e72436796698b: No blocks or log segments found. Creating new log. I0703 01:41:59.026317 25737 leader_election.cc:270] T c7f0c202bc0c4351afc57e7efbbd5e02 P 8ec1760d41f842f398d90e47ab52973b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:41:59.026648 25817 raft_consensus.cc:2730] T c7f0c202bc0c4351afc57e7efbbd5e02 P 8ec1760d41f842f398d90e47ab52973b: Snoozing failure detection for 1.735s (election complete) I0703 01:41:59.026820 25817 raft_consensus.cc:2537] T c7f0c202bc0c4351afc57e7efbbd5e02 P 8ec1760d41f842f398d90e47ab52973b [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:41:59.026955 25817 raft_consensus.cc:593] T c7f0c202bc0c4351afc57e7efbbd5e02 P 8ec1760d41f842f398d90e47ab52973b [term 1 LEADER]: Becoming Leader. State: Replica: 8ec1760d41f842f398d90e47ab52973b, State: Running, Role: LEADER I0703 01:41:59.027081 25737 ts_tablet_manager.cc:1073] T c7f0c202bc0c4351afc57e7efbbd5e02 P 8ec1760d41f842f398d90e47ab52973b: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s I0703 01:41:59.027202 25817 consensus_queue.cc:204] T c7f0c202bc0c4351afc57e7efbbd5e02 P 8ec1760d41f842f398d90e47ab52973b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } I0703 01:41:59.029042 24792 catalog_manager.cc:3798] T c7f0c202bc0c4351afc57e7efbbd5e02 P 8ec1760d41f842f398d90e47ab52973b reported cstate change: term changed from 0 to 1, leader changed from to 8ec1760d41f842f398d90e47ab52973b (127.5.240.194). New cstate: current_term: 1 leader_uuid: "8ec1760d41f842f398d90e47ab52973b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } } I0703 01:41:59.030066 25738 tablet_bootstrap.cc:427] T 93faa898182e46b2a6aab9df75230c0b P dfd7ad6293204fc89f9e72436796698b: No bootstrap required, opened a new log I0703 01:41:59.030148 25738 ts_tablet_manager.cc:1045] T 93faa898182e46b2a6aab9df75230c0b P dfd7ad6293204fc89f9e72436796698b: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s I0703 01:41:59.030661 25738 raft_consensus.cc:325] T 93faa898182e46b2a6aab9df75230c0b P dfd7ad6293204fc89f9e72436796698b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } I0703 01:41:59.030799 25738 raft_consensus.cc:351] T 93faa898182e46b2a6aab9df75230c0b P dfd7ad6293204fc89f9e72436796698b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:41:59.030867 25738 raft_consensus.cc:628] T 93faa898182e46b2a6aab9df75230c0b P dfd7ad6293204fc89f9e72436796698b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfd7ad6293204fc89f9e72436796698b, State: Initialized, Role: FOLLOWER I0703 01:41:59.031033 25738 consensus_queue.cc:227] T 93faa898182e46b2a6aab9df75230c0b P dfd7ad6293204fc89f9e72436796698b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } I0703 01:41:59.031167 25738 raft_consensus.cc:365] T 93faa898182e46b2a6aab9df75230c0b P dfd7ad6293204fc89f9e72436796698b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:41:59.031244 25738 raft_consensus.cc:457] T 93faa898182e46b2a6aab9df75230c0b P dfd7ad6293204fc89f9e72436796698b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:41:59.031296 25738 raft_consensus.cc:2730] T 93faa898182e46b2a6aab9df75230c0b P dfd7ad6293204fc89f9e72436796698b: Snoozing failure detection for 1.557s (starting election) I0703 01:41:59.031357 25738 raft_consensus.cc:2782] T 93faa898182e46b2a6aab9df75230c0b P dfd7ad6293204fc89f9e72436796698b [term 0 FOLLOWER]: Advancing to term 1 I0703 01:41:59.032658 25738 raft_consensus.cc:479] T 93faa898182e46b2a6aab9df75230c0b P dfd7ad6293204fc89f9e72436796698b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } I0703 01:41:59.032791 25738 leader_election.cc:270] T 93faa898182e46b2a6aab9df75230c0b P dfd7ad6293204fc89f9e72436796698b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:41:59.033012 25743 raft_consensus.cc:2730] T 93faa898182e46b2a6aab9df75230c0b P dfd7ad6293204fc89f9e72436796698b: Snoozing failure detection for 1.533s (election complete) I0703 01:41:59.033123 25743 raft_consensus.cc:2537] T 93faa898182e46b2a6aab9df75230c0b P dfd7ad6293204fc89f9e72436796698b [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:41:59.033303 25743 raft_consensus.cc:593] T 93faa898182e46b2a6aab9df75230c0b P dfd7ad6293204fc89f9e72436796698b [term 1 LEADER]: Becoming Leader. State: Replica: dfd7ad6293204fc89f9e72436796698b, State: Running, Role: LEADER I0703 01:41:59.033388 25738 ts_tablet_manager.cc:1073] T 93faa898182e46b2a6aab9df75230c0b P dfd7ad6293204fc89f9e72436796698b: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s I0703 01:41:59.033488 25743 consensus_queue.cc:204] T 93faa898182e46b2a6aab9df75230c0b P dfd7ad6293204fc89f9e72436796698b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } I0703 01:41:59.034852 24792 catalog_manager.cc:3798] T 93faa898182e46b2a6aab9df75230c0b P dfd7ad6293204fc89f9e72436796698b reported cstate change: term changed from 0 to 1, leader changed from to dfd7ad6293204fc89f9e72436796698b (127.5.240.193). New cstate: current_term: 1 leader_uuid: "dfd7ad6293204fc89f9e72436796698b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } } W0703 01:41:59.260270 25841 flags.cc:406] Enabled unsafe flag: --never_fsync=true Using table 'rebalance_test_table_4' Generator report time total : 125.761 ms time per row: 0.0628804 ms I0703 01:41:59.477643 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.5.240.195:38199 --local_ip_for_outbound_sockets=127.5.240.195 --webserver_port=51624 --webserver_interface=127.5.240.195 --tserver_master_addrs=127.5.240.254:57131 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:41:59.638713 25853 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:41:59.639741 25853 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:41:59.641805 25853 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0703 01:41:59.641870 25853 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:41:59.641978 25853 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.195 I0703 01:41:59.648881 25853 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.195:38199 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.5.240.195 --webserver_port=51624 --tserver_master_addrs=127.5.240.254:57131 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.195 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:41:59.655556 25853 tablet_server_main.cc:85] Initializing tablet server... I0703 01:41:59.657119 25853 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 213208us I0703 01:41:59.657699 25853 fs_manager.cc:261] Metadata directory not provided I0703 01:41:59.657778 25853 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:41:59.663525 25853 fs_manager.cc:397] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s I0703 01:41:59.664376 25853 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:41:59.664775 25853 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:41:59.668241 25853 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0703 01:41:59.668304 25853 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal uuid: "cff51d1a4baa4cd0a1625e1d7db94f2c" format_stamp: "Formatted at 2018-07-03 08:41:51 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:41:59.669989 25853 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:41:59.741694 25853 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:41:59.743633 25853 ts_tablet_manager.cc:344] Loading tablet metadata (0/1 complete) I0703 01:41:59.743682 25853 ts_tablet_manager.cc:1005] T fc9b869cd219497896d16275d18c81b9 P cff51d1a4baa4cd0a1625e1d7db94f2c: Loading tablet metadata I0703 01:41:59.747632 25853 ts_tablet_manager.cc:356] Loaded tablet metadata (1 live tablets) I0703 01:41:59.750007 25853 ts_tablet_manager.cc:1173] T fc9b869cd219497896d16275d18c81b9 P cff51d1a4baa4cd0a1625e1d7db94f2c: Registered tablet (data state: TABLET_DATA_READY) I0703 01:41:59.751376 25868 ts_tablet_manager.cc:1028] T fc9b869cd219497896d16275d18c81b9 P cff51d1a4baa4cd0a1625e1d7db94f2c: Bootstrapping tablet I0703 01:41:59.751770 25868 tablet_bootstrap.cc:427] T fc9b869cd219497896d16275d18c81b9 P cff51d1a4baa4cd0a1625e1d7db94f2c: Bootstrap starting. I0703 01:41:59.752368 25853 tablet_server_main.cc:90] Starting tablet server... I0703 01:41:59.779767 25868 log.cc:525] T fc9b869cd219497896d16275d18c81b9 P cff51d1a4baa4cd0a1625e1d7db94f2c: Log is configured to *not* fsync() on all Append() calls I0703 01:41:59.795792 25868 tablet_bootstrap.cc:427] T fc9b869cd219497896d16275d18c81b9 P cff51d1a4baa4cd0a1625e1d7db94f2c: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:41:59.797154 25868 tablet_bootstrap.cc:427] T fc9b869cd219497896d16275d18c81b9 P cff51d1a4baa4cd0a1625e1d7db94f2c: Bootstrap complete. I0703 01:41:59.798844 25868 ts_tablet_manager.cc:1045] T fc9b869cd219497896d16275d18c81b9 P cff51d1a4baa4cd0a1625e1d7db94f2c: Time spent bootstrapping tablet: real 0.047s user 0.010s sys 0.022s I0703 01:41:59.842854 25868 raft_consensus.cc:325] T fc9b869cd219497896d16275d18c81b9 P cff51d1a4baa4cd0a1625e1d7db94f2c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff51d1a4baa4cd0a1625e1d7db94f2c" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 38199 } } I0703 01:41:59.854344 25868 raft_consensus.cc:628] T fc9b869cd219497896d16275d18c81b9 P cff51d1a4baa4cd0a1625e1d7db94f2c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cff51d1a4baa4cd0a1625e1d7db94f2c, State: Initialized, Role: FOLLOWER I0703 01:41:59.855664 25868 consensus_queue.cc:227] T fc9b869cd219497896d16275d18c81b9 P cff51d1a4baa4cd0a1625e1d7db94f2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff51d1a4baa4cd0a1625e1d7db94f2c" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 38199 } } I0703 01:41:59.856092 25868 raft_consensus.cc:365] T fc9b869cd219497896d16275d18c81b9 P cff51d1a4baa4cd0a1625e1d7db94f2c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:41:59.856853 25868 raft_consensus.cc:457] T fc9b869cd219497896d16275d18c81b9 P cff51d1a4baa4cd0a1625e1d7db94f2c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:41:59.857287 25868 raft_consensus.cc:2730] T fc9b869cd219497896d16275d18c81b9 P cff51d1a4baa4cd0a1625e1d7db94f2c: Snoozing failure detection for 1.679s (starting election) I0703 01:41:59.857857 25868 raft_consensus.cc:2782] T fc9b869cd219497896d16275d18c81b9 P cff51d1a4baa4cd0a1625e1d7db94f2c [term 1 FOLLOWER]: Advancing to term 2 I0703 01:41:59.870764 25868 raft_consensus.cc:479] T fc9b869cd219497896d16275d18c81b9 P cff51d1a4baa4cd0a1625e1d7db94f2c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff51d1a4baa4cd0a1625e1d7db94f2c" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 38199 } } I0703 01:41:59.871363 25868 leader_election.cc:270] T fc9b869cd219497896d16275d18c81b9 P cff51d1a4baa4cd0a1625e1d7db94f2c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0703 01:41:59.877492 25953 raft_consensus.cc:2730] T fc9b869cd219497896d16275d18c81b9 P cff51d1a4baa4cd0a1625e1d7db94f2c: Snoozing failure detection for 1.749s (election complete) I0703 01:41:59.878849 25953 raft_consensus.cc:2537] T fc9b869cd219497896d16275d18c81b9 P cff51d1a4baa4cd0a1625e1d7db94f2c [term 2 FOLLOWER]: Leader election won for term 2 I0703 01:41:59.884222 25868 ts_tablet_manager.cc:1073] T fc9b869cd219497896d16275d18c81b9 P cff51d1a4baa4cd0a1625e1d7db94f2c: Time spent starting tablet: real 0.085s user 0.016s sys 0.034s I0703 01:41:59.885896 25953 raft_consensus.cc:593] T fc9b869cd219497896d16275d18c81b9 P cff51d1a4baa4cd0a1625e1d7db94f2c [term 2 LEADER]: Becoming Leader. State: Replica: cff51d1a4baa4cd0a1625e1d7db94f2c, State: Running, Role: LEADER I0703 01:41:59.887500 25953 consensus_queue.cc:204] T fc9b869cd219497896d16275d18c81b9 P cff51d1a4baa4cd0a1625e1d7db94f2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff51d1a4baa4cd0a1625e1d7db94f2c" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 38199 } } I0703 01:41:59.904748 25853 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.195:38199 I0703 01:41:59.905194 25853 webserver.cc:175] Starting webserver on 127.5.240.195:51624 I0703 01:41:59.905237 25853 webserver.cc:186] Document root disabled I0703 01:41:59.906047 25853 webserver.cc:313] Webserver started. Bound to: http://127.5.240.195:51624/ I0703 01:41:59.906602 25853 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb W0703 01:41:59.908941 25979 tablet.cc:1678] T fc9b869cd219497896d16275d18c81b9 P cff51d1a4baa4cd0a1625e1d7db94f2c: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:41:59.909009 25853 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:41:59.915323 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 25853 I0703 01:41:59.915755 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.5.240.196:41121 --local_ip_for_outbound_sockets=127.5.240.196 --webserver_port=40184 --webserver_interface=127.5.240.196 --tserver_master_addrs=127.5.240.254:57131 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0703 01:41:59.918444 25978 heartbeater.cc:346] Connected to a master server at 127.5.240.254:57131 I0703 01:41:59.918599 25978 heartbeater.cc:426] Registering TS with master... I0703 01:41:59.918967 25978 heartbeater.cc:475] Master 127.5.240.254:57131 requested a full tablet report, sending... I0703 01:41:59.920408 24792 ts_manager.cc:104] Re-registered known tserver with Master: cff51d1a4baa4cd0a1625e1d7db94f2c (127.5.240.195:38199) I0703 01:41:59.920616 24792 catalog_manager.cc:3798] T fc9b869cd219497896d16275d18c81b9 P cff51d1a4baa4cd0a1625e1d7db94f2c reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "cff51d1a4baa4cd0a1625e1d7db94f2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff51d1a4baa4cd0a1625e1d7db94f2c" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 38199 } } } I0703 01:41:59.924471 24792 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.195:42980 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:42:00.080317 25985 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:42:00.081341 25985 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:42:00.083333 25985 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0703 01:42:00.083401 25985 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:42:00.083521 25985 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.196 I0703 01:42:00.090412 25985 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.196:41121 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb --webserver_interface=127.5.240.196 --webserver_port=40184 --tserver_master_addrs=127.5.240.254:57131 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.196 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:42:00.096995 25985 tablet_server_main.cc:85] Initializing tablet server... I0703 01:42:00.098538 25985 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 213708us I0703 01:42:00.099114 25985 fs_manager.cc:261] Metadata directory not provided I0703 01:42:00.099192 25985 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:42:00.105115 25985 fs_manager.cc:397] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s I0703 01:42:00.105911 25985 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:42:00.106312 25985 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:42:00.109859 25985 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0703 01:42:00.109925 25985 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal uuid: "ca6ae9994a78460ca6c4ae4ec3f4e9b5" format_stamp: "Formatted at 2018-07-03 08:41:51 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:42:00.111552 25985 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:42:00.184813 25985 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:42:00.186719 25985 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:42:00.187052 25985 tablet_server_main.cc:90] Starting tablet server... I0703 01:42:00.272243 25985 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.196:41121 I0703 01:42:00.272711 25985 webserver.cc:175] Starting webserver on 127.5.240.196:40184 I0703 01:42:00.272758 25985 webserver.cc:186] Document root disabled I0703 01:42:00.274091 25985 webserver.cc:313] Webserver started. Bound to: http://127.5.240.196:40184/ I0703 01:42:00.274750 25985 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb I0703 01:42:00.276054 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 25985 I0703 01:42:00.276603 25985 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:42:00.276638 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data --block_manager=log --rpc_bind_addresses=127.5.240.197:45230 --local_ip_for_outbound_sockets=127.5.240.197 --webserver_port=35582 --webserver_interface=127.5.240.197 --tserver_master_addrs=127.5.240.254:57131 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0703 01:42:00.290832 26104 heartbeater.cc:346] Connected to a master server at 127.5.240.254:57131 I0703 01:42:00.291007 26104 heartbeater.cc:426] Registering TS with master... I0703 01:42:00.291443 26104 heartbeater.cc:475] Master 127.5.240.254:57131 requested a full tablet report, sending... I0703 01:42:00.292480 24792 ts_manager.cc:104] Re-registered known tserver with Master: ca6ae9994a78460ca6c4ae4ec3f4e9b5 (127.5.240.196:41121) I0703 01:42:00.293354 24792 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.196:38048 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:42:00.450668 26107 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:42:00.451685 26107 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:42:00.453675 26107 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0703 01:42:00.453740 26107 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:42:00.453855 26107 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.197 I0703 01:42:00.460520 26107 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.197:45230 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb --webserver_interface=127.5.240.197 --webserver_port=35582 --tserver_master_addrs=127.5.240.254:57131 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.197 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:42:00.467093 26107 tablet_server_main.cc:85] Initializing tablet server... I0703 01:42:00.468652 26107 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 213708us I0703 01:42:00.469209 26107 fs_manager.cc:261] Metadata directory not provided I0703 01:42:00.469286 26107 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:42:00.475806 26107 fs_manager.cc:397] Time spent opening directory manager: real 0.005s user 0.001s sys 0.003s I0703 01:42:00.476601 26107 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:42:00.476981 26107 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:42:00.480113 26107 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s I0703 01:42:00.480177 26107 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal uuid: "2f7e2b9db52f44ba81b7393403f9ed37" format_stamp: "Formatted at 2018-07-03 08:41:52 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:42:00.481848 26107 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:42:00.543166 26107 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:42:00.545145 26107 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:42:00.545461 26107 tablet_server_main.cc:90] Starting tablet server... I0703 01:42:00.630780 26107 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.197:45230 I0703 01:42:00.631201 26107 webserver.cc:175] Starting webserver on 127.5.240.197:35582 I0703 01:42:00.631240 26107 webserver.cc:186] Document root disabled I0703 01:42:00.632421 26107 webserver.cc:313] Webserver started. Bound to: http://127.5.240.197:35582/ I0703 01:42:00.633056 26107 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb I0703 01:42:00.636229 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 26107 I0703 01:42:00.636848 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data --block_manager=log --rpc_bind_addresses=127.5.240.198:35740 --local_ip_for_outbound_sockets=127.5.240.198 --webserver_port=53820 --webserver_interface=127.5.240.198 --tserver_master_addrs=127.5.240.254:57131 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0703 01:42:00.643353 26107 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:42:00.647081 26226 heartbeater.cc:346] Connected to a master server at 127.5.240.254:57131 I0703 01:42:00.647233 26226 heartbeater.cc:426] Registering TS with master... I0703 01:42:00.647580 26226 heartbeater.cc:475] Master 127.5.240.254:57131 requested a full tablet report, sending... I0703 01:42:00.648676 24792 ts_manager.cc:104] Re-registered known tserver with Master: 2f7e2b9db52f44ba81b7393403f9ed37 (127.5.240.197:45230) I0703 01:42:00.650035 24792 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.197:43609 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:42:00.801383 26229 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:42:00.802366 26229 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:42:00.804311 26229 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0703 01:42:00.804358 26229 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:42:00.804440 26229 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.198 I0703 01:42:00.810530 26229 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.198:35740 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data/info.pb --webserver_interface=127.5.240.198 --webserver_port=53820 --tserver_master_addrs=127.5.240.254:57131 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.198 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:42:00.817026 26229 tablet_server_main.cc:85] Initializing tablet server... I0703 01:42:00.818589 26229 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 213708us I0703 01:42:00.819195 26229 fs_manager.cc:261] Metadata directory not provided I0703 01:42:00.819260 26229 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:42:00.825691 26229 fs_manager.cc:397] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s I0703 01:42:00.826447 26229 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:42:00.826841 26229 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:42:00.829854 26229 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0703 01:42:00.829910 26229 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/wal uuid: "01f6144ac0484e1d9a6da7ff80ba71f0" format_stamp: "Formatted at 2018-07-03 08:41:52 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:42:00.831544 26229 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:42:00.855501 26229 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:42:00.856930 26229 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:42:00.857228 26229 tablet_server_main.cc:90] Starting tablet server... I0703 01:42:00.926800 25978 heartbeater.cc:467] Master 127.5.240.254:57131 was elected leader, sending a full tablet report... I0703 01:42:00.943959 26229 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.198:35740 I0703 01:42:00.944345 26229 webserver.cc:175] Starting webserver on 127.5.240.198:53820 I0703 01:42:00.944386 26229 webserver.cc:186] Document root disabled I0703 01:42:00.945582 26229 webserver.cc:313] Webserver started. Bound to: http://127.5.240.198:53820/ I0703 01:42:00.946203 26229 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data/info.pb I0703 01:42:00.948082 26229 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:42:00.955752 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 26229 I0703 01:42:00.956169 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data --block_manager=log --rpc_bind_addresses=127.5.240.199:43710 --local_ip_for_outbound_sockets=127.5.240.199 --webserver_port=53434 --webserver_interface=127.5.240.199 --tserver_master_addrs=127.5.240.254:57131 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0703 01:42:00.959872 26348 heartbeater.cc:346] Connected to a master server at 127.5.240.254:57131 I0703 01:42:00.960173 26348 heartbeater.cc:426] Registering TS with master... I0703 01:42:00.960618 26348 heartbeater.cc:475] Master 127.5.240.254:57131 requested a full tablet report, sending... I0703 01:42:00.961709 24792 ts_manager.cc:104] Re-registered known tserver with Master: 01f6144ac0484e1d9a6da7ff80ba71f0 (127.5.240.198:35740) I0703 01:42:00.962843 24792 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.198:56782 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:42:01.121412 26351 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:42:01.122442 26351 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:42:01.124423 26351 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0703 01:42:01.124487 26351 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:42:01.124619 26351 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.199 I0703 01:42:01.130762 26351 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.199:43710 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data/info.pb --webserver_interface=127.5.240.199 --webserver_port=53434 --tserver_master_addrs=127.5.240.254:57131 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.199 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:42:01.137351 26351 tablet_server_main.cc:85] Initializing tablet server... I0703 01:42:01.138906 26351 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 214208us I0703 01:42:01.139473 26351 fs_manager.cc:261] Metadata directory not provided I0703 01:42:01.139546 26351 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:42:01.145743 26351 fs_manager.cc:397] Time spent opening directory manager: real 0.005s user 0.002s sys 0.002s I0703 01:42:01.146562 26351 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:42:01.146940 26351 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:42:01.150343 26351 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0703 01:42:01.150409 26351 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/wal uuid: "a344baa95900451dbaf349b68e662b27" format_stamp: "Formatted at 2018-07-03 08:41:52 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:42:01.152107 26351 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:42:01.192947 26351 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:42:01.194355 26351 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:42:01.194671 26351 tablet_server_main.cc:90] Starting tablet server... I0703 01:42:01.280345 26351 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.199:43710 I0703 01:42:01.280828 26351 webserver.cc:175] Starting webserver on 127.5.240.199:53434 I0703 01:42:01.280869 26351 webserver.cc:186] Document root disabled I0703 01:42:01.282027 26351 webserver.cc:313] Webserver started. Bound to: http://127.5.240.199:53434/ I0703 01:42:01.282670 26351 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data/info.pb I0703 01:42:01.284503 26351 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:42:01.285883 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 26351 I0703 01:42:01.295691 26104 heartbeater.cc:467] Master 127.5.240.254:57131 was elected leader, sending a full tablet report... I0703 01:42:01.296350 26470 heartbeater.cc:346] Connected to a master server at 127.5.240.254:57131 I0703 01:42:01.296528 26470 heartbeater.cc:426] Registering TS with master... I0703 01:42:01.296967 26470 heartbeater.cc:475] Master 127.5.240.254:57131 requested a full tablet report, sending... I0703 01:42:01.297945 24792 ts_manager.cc:104] Re-registered known tserver with Master: a344baa95900451dbaf349b68e662b27 (127.5.240.199:43710) I0703 01:42:01.298776 24792 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.199:36877 I0703 01:42:01.652007 26226 heartbeater.cc:467] Master 127.5.240.254:57131 was elected leader, sending a full tablet report... I0703 01:42:01.792752 24920 consensus_queue.cc:204] T 7d1747555a45487e869ed52665686c52 P dfd7ad6293204fc89f9e72436796698b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } peers { permanent_uuid: "2f7e2b9db52f44ba81b7393403f9ed37" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 45230 } } W0703 01:42:01.799430 24852 consensus_peers.cc:433] T 7d1747555a45487e869ed52665686c52 P dfd7ad6293204fc89f9e72436796698b -> Peer 2f7e2b9db52f44ba81b7393403f9ed37 (127.5.240.197:45230): Couldn't send request to peer 2f7e2b9db52f44ba81b7393403f9ed37 for tablet 7d1747555a45487e869ed52665686c52. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7d1747555a45487e869ed52665686c52. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:42:01.964975 26348 heartbeater.cc:467] Master 127.5.240.254:57131 was elected leader, sending a full tablet report... I0703 01:42:02.300591 26470 heartbeater.cc:467] Master 127.5.240.254:57131 was elected leader, sending a full tablet report... I0703 01:42:02.394721 26556 ts_tablet_manager.cc:669] T 7d1747555a45487e869ed52665686c52 P 2f7e2b9db52f44ba81b7393403f9ed37: Initiating tablet copy from peer dfd7ad6293204fc89f9e72436796698b (127.5.240.193:43198) I0703 01:42:02.394976 26556 tablet_copy_client.cc:239] T 7d1747555a45487e869ed52665686c52 P 2f7e2b9db52f44ba81b7393403f9ed37: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.193:43198 I0703 01:42:02.399911 24940 tablet_copy_service.cc:135] P dfd7ad6293204fc89f9e72436796698b: Received BeginTabletCopySession request for tablet 7d1747555a45487e869ed52665686c52 from peer 2f7e2b9db52f44ba81b7393403f9ed37 ({username='jenkins'} at 127.5.240.197:49946) I0703 01:42:02.400060 24940 tablet_copy_service.cc:156] P dfd7ad6293204fc89f9e72436796698b: Beginning new tablet copy session on tablet 7d1747555a45487e869ed52665686c52 from peer 2f7e2b9db52f44ba81b7393403f9ed37 at {username='jenkins'} at 127.5.240.197:49946: session id = 2f7e2b9db52f44ba81b7393403f9ed37-7d1747555a45487e869ed52665686c52 I0703 01:42:02.402037 24940 tablet_copy_source_session.cc:202] T 7d1747555a45487e869ed52665686c52 P dfd7ad6293204fc89f9e72436796698b: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:42:02.405072 26556 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 7d1747555a45487e869ed52665686c52. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:02.415738 26556 ts_tablet_manager.cc:1173] T 7d1747555a45487e869ed52665686c52 P 2f7e2b9db52f44ba81b7393403f9ed37: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:42:02.415990 26556 tablet_copy_client.cc:566] T 7d1747555a45487e869ed52665686c52 P 2f7e2b9db52f44ba81b7393403f9ed37: tablet copy: Starting download of 0 data blocks... I0703 01:42:02.416221 26556 tablet_copy_client.cc:529] T 7d1747555a45487e869ed52665686c52 P 2f7e2b9db52f44ba81b7393403f9ed37: tablet copy: Starting download of 1 WAL segments... I0703 01:42:02.418344 26556 tablet_copy_client.cc:405] T 7d1747555a45487e869ed52665686c52 P 2f7e2b9db52f44ba81b7393403f9ed37: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:42:02.420100 26556 ts_tablet_manager.cc:1028] T 7d1747555a45487e869ed52665686c52 P 2f7e2b9db52f44ba81b7393403f9ed37: Bootstrapping tablet I0703 01:42:02.420341 26556 tablet_bootstrap.cc:427] T 7d1747555a45487e869ed52665686c52 P 2f7e2b9db52f44ba81b7393403f9ed37: Bootstrap starting. I0703 01:42:02.427654 26556 log.cc:525] T 7d1747555a45487e869ed52665686c52 P 2f7e2b9db52f44ba81b7393403f9ed37: Log is configured to *not* fsync() on all Append() calls I0703 01:42:02.466581 26556 tablet_bootstrap.cc:427] T 7d1747555a45487e869ed52665686c52 P 2f7e2b9db52f44ba81b7393403f9ed37: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=691 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates I0703 01:42:02.466992 26556 tablet_bootstrap.cc:427] T 7d1747555a45487e869ed52665686c52 P 2f7e2b9db52f44ba81b7393403f9ed37: Bootstrap complete. I0703 01:42:02.467222 26556 ts_tablet_manager.cc:1045] T 7d1747555a45487e869ed52665686c52 P 2f7e2b9db52f44ba81b7393403f9ed37: Time spent bootstrapping tablet: real 0.047s user 0.042s sys 0.004s I0703 01:42:02.470942 26556 raft_consensus.cc:325] T 7d1747555a45487e869ed52665686c52 P 2f7e2b9db52f44ba81b7393403f9ed37 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } I0703 01:42:02.471652 26556 raft_consensus.cc:628] T 7d1747555a45487e869ed52665686c52 P 2f7e2b9db52f44ba81b7393403f9ed37 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f7e2b9db52f44ba81b7393403f9ed37, State: Initialized, Role: FOLLOWER I0703 01:42:02.471889 26556 consensus_queue.cc:227] T 7d1747555a45487e869ed52665686c52 P 2f7e2b9db52f44ba81b7393403f9ed37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } peers { permanent_uuid: "2f7e2b9db52f44ba81b7393403f9ed37" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 45230 } } I0703 01:42:02.473585 26556 ts_tablet_manager.cc:1073] T 7d1747555a45487e869ed52665686c52 P 2f7e2b9db52f44ba81b7393403f9ed37: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s I0703 01:42:02.474577 24940 tablet_copy_service.cc:325] P dfd7ad6293204fc89f9e72436796698b: Request end of tablet copy session 2f7e2b9db52f44ba81b7393403f9ed37-7d1747555a45487e869ed52665686c52 received from {username='jenkins'} at 127.5.240.197:49946 I0703 01:42:02.474669 24940 tablet_copy_service.cc:413] P dfd7ad6293204fc89f9e72436796698b: ending tablet copy session 2f7e2b9db52f44ba81b7393403f9ed37-7d1747555a45487e869ed52665686c52 on tablet 7d1747555a45487e869ed52665686c52 with peer 2f7e2b9db52f44ba81b7393403f9ed37 I0703 01:42:02.792253 26183 raft_consensus.cc:1026] T 7d1747555a45487e869ed52665686c52 P 2f7e2b9db52f44ba81b7393403f9ed37 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[] I0703 01:42:02.794736 26183 raft_consensus.cc:2684] T 7d1747555a45487e869ed52665686c52 P 2f7e2b9db52f44ba81b7393403f9ed37 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 2f7e2b9db52f44ba81b7393403f9ed37 (127.5.240.197) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } peers { permanent_uuid: "2f7e2b9db52f44ba81b7393403f9ed37" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 45230 } } } I0703 01:42:02.794025 26561 raft_consensus.cc:2684] T 7d1747555a45487e869ed52665686c52 P dfd7ad6293204fc89f9e72436796698b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 2f7e2b9db52f44ba81b7393403f9ed37 (127.5.240.197) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } peers { permanent_uuid: "2f7e2b9db52f44ba81b7393403f9ed37" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 45230 } } } I0703 01:42:02.797610 26561 rpcz_store.cc:265] Call kudu.consensus.ConsensusService.ChangeConfig from 127.5.240.193:49926 (request call id 1) took 1005ms. Request Metrics: {"dns_us":59,"raft.queue_time_us":453,"raft.run_cpu_time_us":273,"raft.run_wall_time_us":272,"thread_start_us":469,"threads_started":2,"wal-append.queue_time_us":292,"wal-append.run_cpu_time_us":463,"wal-append.run_wall_time_us":1000603} I0703 01:42:02.799681 24792 catalog_manager.cc:3798] T 7d1747555a45487e869ed52665686c52 P dfd7ad6293204fc89f9e72436796698b reported cstate change: config changed from index -1 to 4, VOTER 2f7e2b9db52f44ba81b7393403f9ed37 (127.5.240.197) added. New cstate: current_term: 1 leader_uuid: "dfd7ad6293204fc89f9e72436796698b" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } peers { permanent_uuid: "2f7e2b9db52f44ba81b7393403f9ed37" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 45230 } } } I0703 01:42:02.821960 25043 consensus_queue.cc:204] T fd7030608be84fa9b61c3ae90e04e27e P 8ec1760d41f842f398d90e47ab52973b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } peers { permanent_uuid: "a344baa95900451dbaf349b68e662b27" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 43710 } } W0703 01:42:02.829099 24977 consensus_peers.cc:433] T fd7030608be84fa9b61c3ae90e04e27e P 8ec1760d41f842f398d90e47ab52973b -> Peer a344baa95900451dbaf349b68e662b27 (127.5.240.199:43710): Couldn't send request to peer a344baa95900451dbaf349b68e662b27 for tablet fd7030608be84fa9b61c3ae90e04e27e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fd7030608be84fa9b61c3ae90e04e27e. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:42:03.298483 26589 ts_tablet_manager.cc:669] T fd7030608be84fa9b61c3ae90e04e27e P a344baa95900451dbaf349b68e662b27: Initiating tablet copy from peer 8ec1760d41f842f398d90e47ab52973b (127.5.240.194:46721) I0703 01:42:03.298772 26589 tablet_copy_client.cc:239] T fd7030608be84fa9b61c3ae90e04e27e P a344baa95900451dbaf349b68e662b27: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.194:46721 I0703 01:42:03.304523 25063 tablet_copy_service.cc:135] P 8ec1760d41f842f398d90e47ab52973b: Received BeginTabletCopySession request for tablet fd7030608be84fa9b61c3ae90e04e27e from peer a344baa95900451dbaf349b68e662b27 ({username='jenkins'} at 127.5.240.199:53181) I0703 01:42:03.304718 25063 tablet_copy_service.cc:156] P 8ec1760d41f842f398d90e47ab52973b: Beginning new tablet copy session on tablet fd7030608be84fa9b61c3ae90e04e27e from peer a344baa95900451dbaf349b68e662b27 at {username='jenkins'} at 127.5.240.199:53181: session id = a344baa95900451dbaf349b68e662b27-fd7030608be84fa9b61c3ae90e04e27e I0703 01:42:03.306648 25063 tablet_copy_source_session.cc:202] T fd7030608be84fa9b61c3ae90e04e27e P 8ec1760d41f842f398d90e47ab52973b: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:42:03.309504 26589 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet fd7030608be84fa9b61c3ae90e04e27e. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:03.317251 26589 ts_tablet_manager.cc:1173] T fd7030608be84fa9b61c3ae90e04e27e P a344baa95900451dbaf349b68e662b27: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:42:03.317497 26589 tablet_copy_client.cc:566] T fd7030608be84fa9b61c3ae90e04e27e P a344baa95900451dbaf349b68e662b27: tablet copy: Starting download of 0 data blocks... I0703 01:42:03.317901 26589 tablet_copy_client.cc:529] T fd7030608be84fa9b61c3ae90e04e27e P a344baa95900451dbaf349b68e662b27: tablet copy: Starting download of 1 WAL segments... I0703 01:42:03.319970 26589 tablet_copy_client.cc:405] T fd7030608be84fa9b61c3ae90e04e27e P a344baa95900451dbaf349b68e662b27: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:42:03.321581 26589 ts_tablet_manager.cc:1028] T fd7030608be84fa9b61c3ae90e04e27e P a344baa95900451dbaf349b68e662b27: Bootstrapping tablet I0703 01:42:03.321794 26589 tablet_bootstrap.cc:427] T fd7030608be84fa9b61c3ae90e04e27e P a344baa95900451dbaf349b68e662b27: Bootstrap starting. I0703 01:42:03.327651 26589 log.cc:525] T fd7030608be84fa9b61c3ae90e04e27e P a344baa95900451dbaf349b68e662b27: Log is configured to *not* fsync() on all Append() calls I0703 01:42:03.363364 26589 tablet_bootstrap.cc:427] T fd7030608be84fa9b61c3ae90e04e27e P a344baa95900451dbaf349b68e662b27: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=656 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates I0703 01:42:03.363749 26589 tablet_bootstrap.cc:427] T fd7030608be84fa9b61c3ae90e04e27e P a344baa95900451dbaf349b68e662b27: Bootstrap complete. I0703 01:42:03.363963 26589 ts_tablet_manager.cc:1045] T fd7030608be84fa9b61c3ae90e04e27e P a344baa95900451dbaf349b68e662b27: Time spent bootstrapping tablet: real 0.042s user 0.037s sys 0.004s I0703 01:42:03.368644 26589 raft_consensus.cc:325] T fd7030608be84fa9b61c3ae90e04e27e P a344baa95900451dbaf349b68e662b27 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } I0703 01:42:03.369477 26589 raft_consensus.cc:628] T fd7030608be84fa9b61c3ae90e04e27e P a344baa95900451dbaf349b68e662b27 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a344baa95900451dbaf349b68e662b27, State: Initialized, Role: FOLLOWER I0703 01:42:03.369793 26589 consensus_queue.cc:227] T fd7030608be84fa9b61c3ae90e04e27e P a344baa95900451dbaf349b68e662b27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } peers { permanent_uuid: "a344baa95900451dbaf349b68e662b27" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 43710 } } I0703 01:42:03.371070 26589 ts_tablet_manager.cc:1073] T fd7030608be84fa9b61c3ae90e04e27e P a344baa95900451dbaf349b68e662b27: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s I0703 01:42:03.371939 25063 tablet_copy_service.cc:325] P 8ec1760d41f842f398d90e47ab52973b: Request end of tablet copy session a344baa95900451dbaf349b68e662b27-fd7030608be84fa9b61c3ae90e04e27e received from {username='jenkins'} at 127.5.240.199:53181 I0703 01:42:03.372022 25063 tablet_copy_service.cc:413] P 8ec1760d41f842f398d90e47ab52973b: ending tablet copy session a344baa95900451dbaf349b68e662b27-fd7030608be84fa9b61c3ae90e04e27e on tablet fd7030608be84fa9b61c3ae90e04e27e with peer a344baa95900451dbaf349b68e662b27 I0703 01:42:03.901060 26427 raft_consensus.cc:1026] T fd7030608be84fa9b61c3ae90e04e27e P a344baa95900451dbaf349b68e662b27 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[] I0703 01:42:03.902776 26596 raft_consensus.cc:2684] T fd7030608be84fa9b61c3ae90e04e27e P 8ec1760d41f842f398d90e47ab52973b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a344baa95900451dbaf349b68e662b27 (127.5.240.199) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } peers { permanent_uuid: "a344baa95900451dbaf349b68e662b27" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 43710 } } } I0703 01:42:03.903602 26427 raft_consensus.cc:2684] T fd7030608be84fa9b61c3ae90e04e27e P a344baa95900451dbaf349b68e662b27 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a344baa95900451dbaf349b68e662b27 (127.5.240.199) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } peers { permanent_uuid: "a344baa95900451dbaf349b68e662b27" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 43710 } } } I0703 01:42:03.905970 26596 rpcz_store.cc:265] Call kudu.consensus.ConsensusService.ChangeConfig from 127.5.240.194:45739 (request call id 1) took 1084ms. Request Metrics: {"dns_us":47,"raft.queue_time_us":395,"raft.run_cpu_time_us":295,"raft.run_wall_time_us":295,"spinlock_wait_cycles":43776,"thread_start_us":390,"threads_started":2,"wal-append.queue_time_us":243,"wal-append.run_cpu_time_us":469,"wal-append.run_wall_time_us":1000571} I0703 01:42:03.907922 24792 catalog_manager.cc:3798] T fd7030608be84fa9b61c3ae90e04e27e P a344baa95900451dbaf349b68e662b27 reported cstate change: config changed from index -1 to 4, VOTER a344baa95900451dbaf349b68e662b27 (127.5.240.199) added. New cstate: current_term: 1 leader_uuid: "8ec1760d41f842f398d90e47ab52973b" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } peers { permanent_uuid: "a344baa95900451dbaf349b68e662b27" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 43710 } } } I0703 01:42:03.929988 24920 consensus_queue.cc:204] T ff7fa826668f419fa3c5225e5d810b9c P dfd7ad6293204fc89f9e72436796698b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } peers { permanent_uuid: "ca6ae9994a78460ca6c4ae4ec3f4e9b5" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 41121 } } W0703 01:42:03.936939 24854 consensus_peers.cc:433] T ff7fa826668f419fa3c5225e5d810b9c P dfd7ad6293204fc89f9e72436796698b -> Peer ca6ae9994a78460ca6c4ae4ec3f4e9b5 (127.5.240.196:41121): Couldn't send request to peer ca6ae9994a78460ca6c4ae4ec3f4e9b5 for tablet ff7fa826668f419fa3c5225e5d810b9c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ff7fa826668f419fa3c5225e5d810b9c. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:42:04.475533 26624 ts_tablet_manager.cc:669] T ff7fa826668f419fa3c5225e5d810b9c P ca6ae9994a78460ca6c4ae4ec3f4e9b5: Initiating tablet copy from peer dfd7ad6293204fc89f9e72436796698b (127.5.240.193:43198) I0703 01:42:04.475821 26624 tablet_copy_client.cc:239] T ff7fa826668f419fa3c5225e5d810b9c P ca6ae9994a78460ca6c4ae4ec3f4e9b5: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.193:43198 I0703 01:42:04.480382 24940 tablet_copy_service.cc:135] P dfd7ad6293204fc89f9e72436796698b: Received BeginTabletCopySession request for tablet ff7fa826668f419fa3c5225e5d810b9c from peer ca6ae9994a78460ca6c4ae4ec3f4e9b5 ({username='jenkins'} at 127.5.240.196:57899) I0703 01:42:04.480482 24940 tablet_copy_service.cc:156] P dfd7ad6293204fc89f9e72436796698b: Beginning new tablet copy session on tablet ff7fa826668f419fa3c5225e5d810b9c from peer ca6ae9994a78460ca6c4ae4ec3f4e9b5 at {username='jenkins'} at 127.5.240.196:57899: session id = ca6ae9994a78460ca6c4ae4ec3f4e9b5-ff7fa826668f419fa3c5225e5d810b9c I0703 01:42:04.482203 24940 tablet_copy_source_session.cc:202] T ff7fa826668f419fa3c5225e5d810b9c P dfd7ad6293204fc89f9e72436796698b: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:42:04.485087 26624 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet ff7fa826668f419fa3c5225e5d810b9c. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:04.492895 26624 ts_tablet_manager.cc:1173] T ff7fa826668f419fa3c5225e5d810b9c P ca6ae9994a78460ca6c4ae4ec3f4e9b5: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:42:04.493147 26624 tablet_copy_client.cc:566] T ff7fa826668f419fa3c5225e5d810b9c P ca6ae9994a78460ca6c4ae4ec3f4e9b5: tablet copy: Starting download of 0 data blocks... I0703 01:42:04.493472 26624 tablet_copy_client.cc:529] T ff7fa826668f419fa3c5225e5d810b9c P ca6ae9994a78460ca6c4ae4ec3f4e9b5: tablet copy: Starting download of 1 WAL segments... I0703 01:42:04.495450 26624 tablet_copy_client.cc:405] T ff7fa826668f419fa3c5225e5d810b9c P ca6ae9994a78460ca6c4ae4ec3f4e9b5: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:42:04.497042 26624 ts_tablet_manager.cc:1028] T ff7fa826668f419fa3c5225e5d810b9c P ca6ae9994a78460ca6c4ae4ec3f4e9b5: Bootstrapping tablet I0703 01:42:04.497253 26624 tablet_bootstrap.cc:427] T ff7fa826668f419fa3c5225e5d810b9c P ca6ae9994a78460ca6c4ae4ec3f4e9b5: Bootstrap starting. I0703 01:42:04.503182 26624 log.cc:525] T ff7fa826668f419fa3c5225e5d810b9c P ca6ae9994a78460ca6c4ae4ec3f4e9b5: Log is configured to *not* fsync() on all Append() calls I0703 01:42:04.539499 26624 tablet_bootstrap.cc:427] T ff7fa826668f419fa3c5225e5d810b9c P ca6ae9994a78460ca6c4ae4ec3f4e9b5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=691 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates I0703 01:42:04.539925 26624 tablet_bootstrap.cc:427] T ff7fa826668f419fa3c5225e5d810b9c P ca6ae9994a78460ca6c4ae4ec3f4e9b5: Bootstrap complete. I0703 01:42:04.540158 26624 ts_tablet_manager.cc:1045] T ff7fa826668f419fa3c5225e5d810b9c P ca6ae9994a78460ca6c4ae4ec3f4e9b5: Time spent bootstrapping tablet: real 0.043s user 0.036s sys 0.006s I0703 01:42:04.543965 26624 raft_consensus.cc:325] T ff7fa826668f419fa3c5225e5d810b9c P ca6ae9994a78460ca6c4ae4ec3f4e9b5 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } I0703 01:42:04.544689 26624 raft_consensus.cc:628] T ff7fa826668f419fa3c5225e5d810b9c P ca6ae9994a78460ca6c4ae4ec3f4e9b5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca6ae9994a78460ca6c4ae4ec3f4e9b5, State: Initialized, Role: FOLLOWER I0703 01:42:04.544929 26624 consensus_queue.cc:227] T ff7fa826668f419fa3c5225e5d810b9c P ca6ae9994a78460ca6c4ae4ec3f4e9b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } peers { permanent_uuid: "ca6ae9994a78460ca6c4ae4ec3f4e9b5" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 41121 } } I0703 01:42:04.546775 26624 ts_tablet_manager.cc:1073] T ff7fa826668f419fa3c5225e5d810b9c P ca6ae9994a78460ca6c4ae4ec3f4e9b5: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s I0703 01:42:04.547624 24940 tablet_copy_service.cc:325] P dfd7ad6293204fc89f9e72436796698b: Request end of tablet copy session ca6ae9994a78460ca6c4ae4ec3f4e9b5-ff7fa826668f419fa3c5225e5d810b9c received from {username='jenkins'} at 127.5.240.196:57899 I0703 01:42:04.547703 24940 tablet_copy_service.cc:413] P dfd7ad6293204fc89f9e72436796698b: ending tablet copy session ca6ae9994a78460ca6c4ae4ec3f4e9b5-ff7fa826668f419fa3c5225e5d810b9c on tablet ff7fa826668f419fa3c5225e5d810b9c with peer ca6ae9994a78460ca6c4ae4ec3f4e9b5 I0703 01:42:05.096254 26061 raft_consensus.cc:1026] T ff7fa826668f419fa3c5225e5d810b9c P ca6ae9994a78460ca6c4ae4ec3f4e9b5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[] I0703 01:42:05.097910 26630 raft_consensus.cc:2684] T ff7fa826668f419fa3c5225e5d810b9c P dfd7ad6293204fc89f9e72436796698b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ca6ae9994a78460ca6c4ae4ec3f4e9b5 (127.5.240.196) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } peers { permanent_uuid: "ca6ae9994a78460ca6c4ae4ec3f4e9b5" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 41121 } } } I0703 01:42:05.098649 26061 raft_consensus.cc:2684] T ff7fa826668f419fa3c5225e5d810b9c P ca6ae9994a78460ca6c4ae4ec3f4e9b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ca6ae9994a78460ca6c4ae4ec3f4e9b5 (127.5.240.196) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } peers { permanent_uuid: "ca6ae9994a78460ca6c4ae4ec3f4e9b5" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 41121 } } } I0703 01:42:05.100260 26630 rpcz_store.cc:265] Call kudu.consensus.ConsensusService.ChangeConfig from 127.5.240.193:49932 (request call id 1) took 1170ms. Request Metrics: {"dns_us":55,"raft.queue_time_us":56,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":157,"spinlock_wait_cycles":8960,"thread_start_us":137,"threads_started":1,"wal-append.queue_time_us":294,"wal-append.run_cpu_time_us":271,"wal-append.run_wall_time_us":1000374} I0703 01:42:05.101691 24792 catalog_manager.cc:3798] T ff7fa826668f419fa3c5225e5d810b9c P dfd7ad6293204fc89f9e72436796698b reported cstate change: config changed from index -1 to 4, VOTER ca6ae9994a78460ca6c4ae4ec3f4e9b5 (127.5.240.196) added. New cstate: current_term: 1 leader_uuid: "dfd7ad6293204fc89f9e72436796698b" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } peers { permanent_uuid: "ca6ae9994a78460ca6c4ae4ec3f4e9b5" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 41121 } } } I0703 01:42:05.122912 24920 consensus_queue.cc:204] T 6359db2dd602463e841abc2a907c49a3 P dfd7ad6293204fc89f9e72436796698b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } peers { permanent_uuid: "01f6144ac0484e1d9a6da7ff80ba71f0" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 35740 } } W0703 01:42:05.129266 24854 consensus_peers.cc:433] T 6359db2dd602463e841abc2a907c49a3 P dfd7ad6293204fc89f9e72436796698b -> Peer 01f6144ac0484e1d9a6da7ff80ba71f0 (127.5.240.198:35740): Couldn't send request to peer 01f6144ac0484e1d9a6da7ff80ba71f0 for tablet 6359db2dd602463e841abc2a907c49a3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6359db2dd602463e841abc2a907c49a3. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:42:05.629271 26658 ts_tablet_manager.cc:669] T 6359db2dd602463e841abc2a907c49a3 P 01f6144ac0484e1d9a6da7ff80ba71f0: Initiating tablet copy from peer dfd7ad6293204fc89f9e72436796698b (127.5.240.193:43198) I0703 01:42:05.629544 26658 tablet_copy_client.cc:239] T 6359db2dd602463e841abc2a907c49a3 P 01f6144ac0484e1d9a6da7ff80ba71f0: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.193:43198 I0703 01:42:05.634584 24940 tablet_copy_service.cc:135] P dfd7ad6293204fc89f9e72436796698b: Received BeginTabletCopySession request for tablet 6359db2dd602463e841abc2a907c49a3 from peer 01f6144ac0484e1d9a6da7ff80ba71f0 ({username='jenkins'} at 127.5.240.198:49776) I0703 01:42:05.634680 24940 tablet_copy_service.cc:156] P dfd7ad6293204fc89f9e72436796698b: Beginning new tablet copy session on tablet 6359db2dd602463e841abc2a907c49a3 from peer 01f6144ac0484e1d9a6da7ff80ba71f0 at {username='jenkins'} at 127.5.240.198:49776: session id = 01f6144ac0484e1d9a6da7ff80ba71f0-6359db2dd602463e841abc2a907c49a3 I0703 01:42:05.636291 24940 tablet_copy_source_session.cc:202] T 6359db2dd602463e841abc2a907c49a3 P dfd7ad6293204fc89f9e72436796698b: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:42:05.639132 26658 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 6359db2dd602463e841abc2a907c49a3. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:05.646682 26658 ts_tablet_manager.cc:1173] T 6359db2dd602463e841abc2a907c49a3 P 01f6144ac0484e1d9a6da7ff80ba71f0: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:42:05.646935 26658 tablet_copy_client.cc:566] T 6359db2dd602463e841abc2a907c49a3 P 01f6144ac0484e1d9a6da7ff80ba71f0: tablet copy: Starting download of 0 data blocks... I0703 01:42:05.647214 26658 tablet_copy_client.cc:529] T 6359db2dd602463e841abc2a907c49a3 P 01f6144ac0484e1d9a6da7ff80ba71f0: tablet copy: Starting download of 1 WAL segments... I0703 01:42:05.649152 26658 tablet_copy_client.cc:405] T 6359db2dd602463e841abc2a907c49a3 P 01f6144ac0484e1d9a6da7ff80ba71f0: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:42:05.650794 26658 ts_tablet_manager.cc:1028] T 6359db2dd602463e841abc2a907c49a3 P 01f6144ac0484e1d9a6da7ff80ba71f0: Bootstrapping tablet I0703 01:42:05.650996 26658 tablet_bootstrap.cc:427] T 6359db2dd602463e841abc2a907c49a3 P 01f6144ac0484e1d9a6da7ff80ba71f0: Bootstrap starting. I0703 01:42:05.656782 26658 log.cc:525] T 6359db2dd602463e841abc2a907c49a3 P 01f6144ac0484e1d9a6da7ff80ba71f0: Log is configured to *not* fsync() on all Append() calls I0703 01:42:05.691730 26658 tablet_bootstrap.cc:427] T 6359db2dd602463e841abc2a907c49a3 P 01f6144ac0484e1d9a6da7ff80ba71f0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=653 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates I0703 01:42:05.692095 26658 tablet_bootstrap.cc:427] T 6359db2dd602463e841abc2a907c49a3 P 01f6144ac0484e1d9a6da7ff80ba71f0: Bootstrap complete. I0703 01:42:05.692297 26658 ts_tablet_manager.cc:1045] T 6359db2dd602463e841abc2a907c49a3 P 01f6144ac0484e1d9a6da7ff80ba71f0: Time spent bootstrapping tablet: real 0.041s user 0.035s sys 0.005s I0703 01:42:05.695869 26658 raft_consensus.cc:325] T 6359db2dd602463e841abc2a907c49a3 P 01f6144ac0484e1d9a6da7ff80ba71f0 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } I0703 01:42:05.696475 26658 raft_consensus.cc:628] T 6359db2dd602463e841abc2a907c49a3 P 01f6144ac0484e1d9a6da7ff80ba71f0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01f6144ac0484e1d9a6da7ff80ba71f0, State: Initialized, Role: FOLLOWER I0703 01:42:05.696728 26658 consensus_queue.cc:227] T 6359db2dd602463e841abc2a907c49a3 P 01f6144ac0484e1d9a6da7ff80ba71f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } peers { permanent_uuid: "01f6144ac0484e1d9a6da7ff80ba71f0" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 35740 } } I0703 01:42:05.697890 26658 ts_tablet_manager.cc:1073] T 6359db2dd602463e841abc2a907c49a3 P 01f6144ac0484e1d9a6da7ff80ba71f0: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s I0703 01:42:05.698616 24940 tablet_copy_service.cc:325] P dfd7ad6293204fc89f9e72436796698b: Request end of tablet copy session 01f6144ac0484e1d9a6da7ff80ba71f0-6359db2dd602463e841abc2a907c49a3 received from {username='jenkins'} at 127.5.240.198:49776 I0703 01:42:05.698693 24940 tablet_copy_service.cc:413] P dfd7ad6293204fc89f9e72436796698b: ending tablet copy session 01f6144ac0484e1d9a6da7ff80ba71f0-6359db2dd602463e841abc2a907c49a3 on tablet 6359db2dd602463e841abc2a907c49a3 with peer 01f6144ac0484e1d9a6da7ff80ba71f0 I0703 01:42:06.080152 26305 raft_consensus.cc:1026] T 6359db2dd602463e841abc2a907c49a3 P 01f6144ac0484e1d9a6da7ff80ba71f0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[] I0703 01:42:06.081688 26630 raft_consensus.cc:2684] T 6359db2dd602463e841abc2a907c49a3 P dfd7ad6293204fc89f9e72436796698b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 01f6144ac0484e1d9a6da7ff80ba71f0 (127.5.240.198) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } peers { permanent_uuid: "01f6144ac0484e1d9a6da7ff80ba71f0" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 35740 } } } I0703 01:42:06.082285 26305 raft_consensus.cc:2684] T 6359db2dd602463e841abc2a907c49a3 P 01f6144ac0484e1d9a6da7ff80ba71f0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 01f6144ac0484e1d9a6da7ff80ba71f0 (127.5.240.198) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } peers { permanent_uuid: "01f6144ac0484e1d9a6da7ff80ba71f0" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 35740 } } } I0703 01:42:06.085459 24792 catalog_manager.cc:3798] T 6359db2dd602463e841abc2a907c49a3 P dfd7ad6293204fc89f9e72436796698b reported cstate change: config changed from index -1 to 4, VOTER 01f6144ac0484e1d9a6da7ff80ba71f0 (127.5.240.198) added. New cstate: current_term: 1 leader_uuid: "dfd7ad6293204fc89f9e72436796698b" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } peers { permanent_uuid: "01f6144ac0484e1d9a6da7ff80ba71f0" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 35740 } } } I0703 01:42:06.106398 24920 consensus_queue.cc:204] T 53242d0289d644faa0149e0a54c0984c P dfd7ad6293204fc89f9e72436796698b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } peers { permanent_uuid: "cff51d1a4baa4cd0a1625e1d7db94f2c" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 38199 } } W0703 01:42:06.112495 24852 consensus_peers.cc:433] T 53242d0289d644faa0149e0a54c0984c P dfd7ad6293204fc89f9e72436796698b -> Peer cff51d1a4baa4cd0a1625e1d7db94f2c (127.5.240.195:38199): Couldn't send request to peer cff51d1a4baa4cd0a1625e1d7db94f2c for tablet 53242d0289d644faa0149e0a54c0984c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 53242d0289d644faa0149e0a54c0984c. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:42:06.511631 26688 ts_tablet_manager.cc:669] T 53242d0289d644faa0149e0a54c0984c P cff51d1a4baa4cd0a1625e1d7db94f2c: Initiating tablet copy from peer dfd7ad6293204fc89f9e72436796698b (127.5.240.193:43198) I0703 01:42:06.511898 26688 tablet_copy_client.cc:239] T 53242d0289d644faa0149e0a54c0984c P cff51d1a4baa4cd0a1625e1d7db94f2c: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.193:43198 I0703 01:42:06.516458 24940 tablet_copy_service.cc:135] P dfd7ad6293204fc89f9e72436796698b: Received BeginTabletCopySession request for tablet 53242d0289d644faa0149e0a54c0984c from peer cff51d1a4baa4cd0a1625e1d7db94f2c ({username='jenkins'} at 127.5.240.195:59782) I0703 01:42:06.516592 24940 tablet_copy_service.cc:156] P dfd7ad6293204fc89f9e72436796698b: Beginning new tablet copy session on tablet 53242d0289d644faa0149e0a54c0984c from peer cff51d1a4baa4cd0a1625e1d7db94f2c at {username='jenkins'} at 127.5.240.195:59782: session id = cff51d1a4baa4cd0a1625e1d7db94f2c-53242d0289d644faa0149e0a54c0984c I0703 01:42:06.518229 24940 tablet_copy_source_session.cc:202] T 53242d0289d644faa0149e0a54c0984c P dfd7ad6293204fc89f9e72436796698b: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:42:06.519850 26688 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 53242d0289d644faa0149e0a54c0984c. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:06.523244 26688 ts_tablet_manager.cc:1173] T 53242d0289d644faa0149e0a54c0984c P cff51d1a4baa4cd0a1625e1d7db94f2c: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:42:06.523454 26688 tablet_copy_client.cc:566] T 53242d0289d644faa0149e0a54c0984c P cff51d1a4baa4cd0a1625e1d7db94f2c: tablet copy: Starting download of 0 data blocks... I0703 01:42:06.523717 26688 tablet_copy_client.cc:529] T 53242d0289d644faa0149e0a54c0984c P cff51d1a4baa4cd0a1625e1d7db94f2c: tablet copy: Starting download of 1 WAL segments... I0703 01:42:06.525573 26688 tablet_copy_client.cc:405] T 53242d0289d644faa0149e0a54c0984c P cff51d1a4baa4cd0a1625e1d7db94f2c: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:42:06.527047 26688 ts_tablet_manager.cc:1028] T 53242d0289d644faa0149e0a54c0984c P cff51d1a4baa4cd0a1625e1d7db94f2c: Bootstrapping tablet I0703 01:42:06.527165 26688 tablet_bootstrap.cc:427] T 53242d0289d644faa0149e0a54c0984c P cff51d1a4baa4cd0a1625e1d7db94f2c: Bootstrap starting. I0703 01:42:06.583361 26688 tablet_bootstrap.cc:427] T 53242d0289d644faa0149e0a54c0984c P cff51d1a4baa4cd0a1625e1d7db94f2c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=691 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates I0703 01:42:06.583701 26688 tablet_bootstrap.cc:427] T 53242d0289d644faa0149e0a54c0984c P cff51d1a4baa4cd0a1625e1d7db94f2c: Bootstrap complete. I0703 01:42:06.583853 26688 ts_tablet_manager.cc:1045] T 53242d0289d644faa0149e0a54c0984c P cff51d1a4baa4cd0a1625e1d7db94f2c: Time spent bootstrapping tablet: real 0.057s user 0.037s sys 0.003s I0703 01:42:06.584533 26688 raft_consensus.cc:325] T 53242d0289d644faa0149e0a54c0984c P cff51d1a4baa4cd0a1625e1d7db94f2c [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } I0703 01:42:06.584962 26688 raft_consensus.cc:628] T 53242d0289d644faa0149e0a54c0984c P cff51d1a4baa4cd0a1625e1d7db94f2c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cff51d1a4baa4cd0a1625e1d7db94f2c, State: Initialized, Role: FOLLOWER I0703 01:42:06.585070 26688 consensus_queue.cc:227] T 53242d0289d644faa0149e0a54c0984c P cff51d1a4baa4cd0a1625e1d7db94f2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } peers { permanent_uuid: "cff51d1a4baa4cd0a1625e1d7db94f2c" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 38199 } } I0703 01:42:06.585772 26688 ts_tablet_manager.cc:1073] T 53242d0289d644faa0149e0a54c0984c P cff51d1a4baa4cd0a1625e1d7db94f2c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s I0703 01:42:06.586566 24940 tablet_copy_service.cc:325] P dfd7ad6293204fc89f9e72436796698b: Request end of tablet copy session cff51d1a4baa4cd0a1625e1d7db94f2c-53242d0289d644faa0149e0a54c0984c received from {username='jenkins'} at 127.5.240.195:59782 I0703 01:42:06.586643 24940 tablet_copy_service.cc:413] P dfd7ad6293204fc89f9e72436796698b: ending tablet copy session cff51d1a4baa4cd0a1625e1d7db94f2c-53242d0289d644faa0149e0a54c0984c on tablet 53242d0289d644faa0149e0a54c0984c with peer cff51d1a4baa4cd0a1625e1d7db94f2c I0703 01:42:07.100355 25931 raft_consensus.cc:1026] T 53242d0289d644faa0149e0a54c0984c P cff51d1a4baa4cd0a1625e1d7db94f2c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[] I0703 01:42:07.102011 26694 raft_consensus.cc:2684] T 53242d0289d644faa0149e0a54c0984c P dfd7ad6293204fc89f9e72436796698b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER cff51d1a4baa4cd0a1625e1d7db94f2c (127.5.240.195) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } peers { permanent_uuid: "cff51d1a4baa4cd0a1625e1d7db94f2c" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 38199 } } } I0703 01:42:07.102514 25931 raft_consensus.cc:2684] T 53242d0289d644faa0149e0a54c0984c P cff51d1a4baa4cd0a1625e1d7db94f2c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER cff51d1a4baa4cd0a1625e1d7db94f2c (127.5.240.195) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } peers { permanent_uuid: "cff51d1a4baa4cd0a1625e1d7db94f2c" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 38199 } } } I0703 01:42:07.105962 24793 catalog_manager.cc:3798] T 53242d0289d644faa0149e0a54c0984c P dfd7ad6293204fc89f9e72436796698b reported cstate change: config changed from index -1 to 4, VOTER cff51d1a4baa4cd0a1625e1d7db94f2c (127.5.240.195) added. New cstate: current_term: 1 leader_uuid: "dfd7ad6293204fc89f9e72436796698b" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } peers { permanent_uuid: "cff51d1a4baa4cd0a1625e1d7db94f2c" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 38199 } } } I0703 01:42:07.128252 25043 consensus_queue.cc:204] T 9a818723ff79493aa17e7434040e8975 P 8ec1760d41f842f398d90e47ab52973b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } peers { permanent_uuid: "2f7e2b9db52f44ba81b7393403f9ed37" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 45230 } } W0703 01:42:07.134328 24975 consensus_peers.cc:433] T 9a818723ff79493aa17e7434040e8975 P 8ec1760d41f842f398d90e47ab52973b -> Peer 2f7e2b9db52f44ba81b7393403f9ed37 (127.5.240.197:45230): Couldn't send request to peer 2f7e2b9db52f44ba81b7393403f9ed37 for tablet 9a818723ff79493aa17e7434040e8975. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9a818723ff79493aa17e7434040e8975. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:42:07.593762 26721 ts_tablet_manager.cc:669] T 9a818723ff79493aa17e7434040e8975 P 2f7e2b9db52f44ba81b7393403f9ed37: Initiating tablet copy from peer 8ec1760d41f842f398d90e47ab52973b (127.5.240.194:46721) I0703 01:42:07.593924 26721 tablet_copy_client.cc:239] T 9a818723ff79493aa17e7434040e8975 P 2f7e2b9db52f44ba81b7393403f9ed37: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.194:46721 I0703 01:42:07.598199 25063 tablet_copy_service.cc:135] P 8ec1760d41f842f398d90e47ab52973b: Received BeginTabletCopySession request for tablet 9a818723ff79493aa17e7434040e8975 from peer 2f7e2b9db52f44ba81b7393403f9ed37 ({username='jenkins'} at 127.5.240.197:51614) I0703 01:42:07.598301 25063 tablet_copy_service.cc:156] P 8ec1760d41f842f398d90e47ab52973b: Beginning new tablet copy session on tablet 9a818723ff79493aa17e7434040e8975 from peer 2f7e2b9db52f44ba81b7393403f9ed37 at {username='jenkins'} at 127.5.240.197:51614: session id = 2f7e2b9db52f44ba81b7393403f9ed37-9a818723ff79493aa17e7434040e8975 I0703 01:42:07.599931 25063 tablet_copy_source_session.cc:202] T 9a818723ff79493aa17e7434040e8975 P 8ec1760d41f842f398d90e47ab52973b: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:42:07.600945 26721 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 9a818723ff79493aa17e7434040e8975. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:07.603718 26721 ts_tablet_manager.cc:1173] T 9a818723ff79493aa17e7434040e8975 P 2f7e2b9db52f44ba81b7393403f9ed37: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:42:07.603901 26721 tablet_copy_client.cc:566] T 9a818723ff79493aa17e7434040e8975 P 2f7e2b9db52f44ba81b7393403f9ed37: tablet copy: Starting download of 0 data blocks... I0703 01:42:07.604074 26721 tablet_copy_client.cc:529] T 9a818723ff79493aa17e7434040e8975 P 2f7e2b9db52f44ba81b7393403f9ed37: tablet copy: Starting download of 1 WAL segments... I0703 01:42:07.605851 26721 tablet_copy_client.cc:405] T 9a818723ff79493aa17e7434040e8975 P 2f7e2b9db52f44ba81b7393403f9ed37: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:42:07.607190 26721 ts_tablet_manager.cc:1028] T 9a818723ff79493aa17e7434040e8975 P 2f7e2b9db52f44ba81b7393403f9ed37: Bootstrapping tablet I0703 01:42:07.607306 26721 tablet_bootstrap.cc:427] T 9a818723ff79493aa17e7434040e8975 P 2f7e2b9db52f44ba81b7393403f9ed37: Bootstrap starting. I0703 01:42:07.646081 26721 tablet_bootstrap.cc:427] T 9a818723ff79493aa17e7434040e8975 P 2f7e2b9db52f44ba81b7393403f9ed37: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=691 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates I0703 01:42:07.646360 26721 tablet_bootstrap.cc:427] T 9a818723ff79493aa17e7434040e8975 P 2f7e2b9db52f44ba81b7393403f9ed37: Bootstrap complete. I0703 01:42:07.646497 26721 ts_tablet_manager.cc:1045] T 9a818723ff79493aa17e7434040e8975 P 2f7e2b9db52f44ba81b7393403f9ed37: Time spent bootstrapping tablet: real 0.039s user 0.038s sys 0.000s I0703 01:42:07.647061 26721 raft_consensus.cc:325] T 9a818723ff79493aa17e7434040e8975 P 2f7e2b9db52f44ba81b7393403f9ed37 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } I0703 01:42:07.647353 26721 raft_consensus.cc:628] T 9a818723ff79493aa17e7434040e8975 P 2f7e2b9db52f44ba81b7393403f9ed37 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f7e2b9db52f44ba81b7393403f9ed37, State: Initialized, Role: FOLLOWER I0703 01:42:07.647469 26721 consensus_queue.cc:227] T 9a818723ff79493aa17e7434040e8975 P 2f7e2b9db52f44ba81b7393403f9ed37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } peers { permanent_uuid: "2f7e2b9db52f44ba81b7393403f9ed37" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 45230 } } I0703 01:42:07.648140 26721 ts_tablet_manager.cc:1073] T 9a818723ff79493aa17e7434040e8975 P 2f7e2b9db52f44ba81b7393403f9ed37: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s I0703 01:42:07.648891 25063 tablet_copy_service.cc:325] P 8ec1760d41f842f398d90e47ab52973b: Request end of tablet copy session 2f7e2b9db52f44ba81b7393403f9ed37-9a818723ff79493aa17e7434040e8975 received from {username='jenkins'} at 127.5.240.197:51614 I0703 01:42:07.648964 25063 tablet_copy_service.cc:413] P 8ec1760d41f842f398d90e47ab52973b: ending tablet copy session 2f7e2b9db52f44ba81b7393403f9ed37-9a818723ff79493aa17e7434040e8975 on tablet 9a818723ff79493aa17e7434040e8975 with peer 2f7e2b9db52f44ba81b7393403f9ed37 I0703 01:42:08.141695 26183 raft_consensus.cc:1026] T 9a818723ff79493aa17e7434040e8975 P 2f7e2b9db52f44ba81b7393403f9ed37 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[] I0703 01:42:08.142933 26726 raft_consensus.cc:2684] T 9a818723ff79493aa17e7434040e8975 P 8ec1760d41f842f398d90e47ab52973b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 2f7e2b9db52f44ba81b7393403f9ed37 (127.5.240.197) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } peers { permanent_uuid: "2f7e2b9db52f44ba81b7393403f9ed37" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 45230 } } } I0703 01:42:08.143551 26183 raft_consensus.cc:2684] T 9a818723ff79493aa17e7434040e8975 P 2f7e2b9db52f44ba81b7393403f9ed37 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 2f7e2b9db52f44ba81b7393403f9ed37 (127.5.240.197) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } peers { permanent_uuid: "2f7e2b9db52f44ba81b7393403f9ed37" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 45230 } } } I0703 01:42:08.145182 26726 rpcz_store.cc:265] Call kudu.consensus.ConsensusService.ChangeConfig from 127.5.240.194:45751 (request call id 1) took 1017ms. Request Metrics: {"dns_us":51,"raft.queue_time_us":41,"raft.run_cpu_time_us":97,"raft.run_wall_time_us":97,"spinlock_wait_cycles":72448,"thread_start_us":160,"threads_started":1,"wal-append.queue_time_us":250,"wal-append.run_cpu_time_us":363,"wal-append.run_wall_time_us":1000468} I0703 01:42:08.147564 24797 catalog_manager.cc:3798] T 9a818723ff79493aa17e7434040e8975 P 8ec1760d41f842f398d90e47ab52973b reported cstate change: config changed from index -1 to 4, VOTER 2f7e2b9db52f44ba81b7393403f9ed37 (127.5.240.197) added. New cstate: current_term: 1 leader_uuid: "8ec1760d41f842f398d90e47ab52973b" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } peers { permanent_uuid: "2f7e2b9db52f44ba81b7393403f9ed37" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 45230 } } } I0703 01:42:08.167737 25043 consensus_queue.cc:204] T dae8746553524b6a9eb2e3f177f658f1 P 8ec1760d41f842f398d90e47ab52973b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } peers { permanent_uuid: "a344baa95900451dbaf349b68e662b27" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 43710 } } W0703 01:42:08.169903 24977 consensus_peers.cc:433] T dae8746553524b6a9eb2e3f177f658f1 P 8ec1760d41f842f398d90e47ab52973b -> Peer a344baa95900451dbaf349b68e662b27 (127.5.240.199:43710): Couldn't send request to peer a344baa95900451dbaf349b68e662b27 for tablet dae8746553524b6a9eb2e3f177f658f1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dae8746553524b6a9eb2e3f177f658f1. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:42:08.723906 26751 ts_tablet_manager.cc:669] T dae8746553524b6a9eb2e3f177f658f1 P a344baa95900451dbaf349b68e662b27: Initiating tablet copy from peer 8ec1760d41f842f398d90e47ab52973b (127.5.240.194:46721) I0703 01:42:08.724076 26751 tablet_copy_client.cc:239] T dae8746553524b6a9eb2e3f177f658f1 P a344baa95900451dbaf349b68e662b27: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.194:46721 I0703 01:42:08.724769 25063 tablet_copy_service.cc:135] P 8ec1760d41f842f398d90e47ab52973b: Received BeginTabletCopySession request for tablet dae8746553524b6a9eb2e3f177f658f1 from peer a344baa95900451dbaf349b68e662b27 ({username='jenkins'} at 127.5.240.199:53181) I0703 01:42:08.724875 25063 tablet_copy_service.cc:156] P 8ec1760d41f842f398d90e47ab52973b: Beginning new tablet copy session on tablet dae8746553524b6a9eb2e3f177f658f1 from peer a344baa95900451dbaf349b68e662b27 at {username='jenkins'} at 127.5.240.199:53181: session id = a344baa95900451dbaf349b68e662b27-dae8746553524b6a9eb2e3f177f658f1 I0703 01:42:08.726477 25063 tablet_copy_source_session.cc:202] T dae8746553524b6a9eb2e3f177f658f1 P 8ec1760d41f842f398d90e47ab52973b: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:42:08.727521 26751 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet dae8746553524b6a9eb2e3f177f658f1. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:08.730377 26751 ts_tablet_manager.cc:1173] T dae8746553524b6a9eb2e3f177f658f1 P a344baa95900451dbaf349b68e662b27: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:42:08.730548 26751 tablet_copy_client.cc:566] T dae8746553524b6a9eb2e3f177f658f1 P a344baa95900451dbaf349b68e662b27: tablet copy: Starting download of 0 data blocks... I0703 01:42:08.730819 26751 tablet_copy_client.cc:529] T dae8746553524b6a9eb2e3f177f658f1 P a344baa95900451dbaf349b68e662b27: tablet copy: Starting download of 1 WAL segments... I0703 01:42:08.732520 26751 tablet_copy_client.cc:405] T dae8746553524b6a9eb2e3f177f658f1 P a344baa95900451dbaf349b68e662b27: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:42:08.733937 26751 ts_tablet_manager.cc:1028] T dae8746553524b6a9eb2e3f177f658f1 P a344baa95900451dbaf349b68e662b27: Bootstrapping tablet I0703 01:42:08.734060 26751 tablet_bootstrap.cc:427] T dae8746553524b6a9eb2e3f177f658f1 P a344baa95900451dbaf349b68e662b27: Bootstrap starting. I0703 01:42:08.771302 26751 tablet_bootstrap.cc:427] T dae8746553524b6a9eb2e3f177f658f1 P a344baa95900451dbaf349b68e662b27: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=691 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates I0703 01:42:08.771579 26751 tablet_bootstrap.cc:427] T dae8746553524b6a9eb2e3f177f658f1 P a344baa95900451dbaf349b68e662b27: Bootstrap complete. I0703 01:42:08.771785 26751 ts_tablet_manager.cc:1045] T dae8746553524b6a9eb2e3f177f658f1 P a344baa95900451dbaf349b68e662b27: Time spent bootstrapping tablet: real 0.038s user 0.036s sys 0.001s I0703 01:42:08.772441 26751 raft_consensus.cc:325] T dae8746553524b6a9eb2e3f177f658f1 P a344baa95900451dbaf349b68e662b27 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } I0703 01:42:08.772744 26751 raft_consensus.cc:628] T dae8746553524b6a9eb2e3f177f658f1 P a344baa95900451dbaf349b68e662b27 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a344baa95900451dbaf349b68e662b27, State: Initialized, Role: FOLLOWER I0703 01:42:08.772857 26751 consensus_queue.cc:227] T dae8746553524b6a9eb2e3f177f658f1 P a344baa95900451dbaf349b68e662b27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } peers { permanent_uuid: "a344baa95900451dbaf349b68e662b27" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 43710 } } I0703 01:42:08.773515 26751 ts_tablet_manager.cc:1073] T dae8746553524b6a9eb2e3f177f658f1 P a344baa95900451dbaf349b68e662b27: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s I0703 01:42:08.774209 25063 tablet_copy_service.cc:325] P 8ec1760d41f842f398d90e47ab52973b: Request end of tablet copy session a344baa95900451dbaf349b68e662b27-dae8746553524b6a9eb2e3f177f658f1 received from {username='jenkins'} at 127.5.240.199:53181 I0703 01:42:08.774282 25063 tablet_copy_service.cc:413] P 8ec1760d41f842f398d90e47ab52973b: ending tablet copy session a344baa95900451dbaf349b68e662b27-dae8746553524b6a9eb2e3f177f658f1 on tablet dae8746553524b6a9eb2e3f177f658f1 with peer a344baa95900451dbaf349b68e662b27 I0703 01:42:09.218636 26426 raft_consensus.cc:1026] T dae8746553524b6a9eb2e3f177f658f1 P a344baa95900451dbaf349b68e662b27 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[] I0703 01:42:09.219995 26754 raft_consensus.cc:2684] T dae8746553524b6a9eb2e3f177f658f1 P 8ec1760d41f842f398d90e47ab52973b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a344baa95900451dbaf349b68e662b27 (127.5.240.199) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } peers { permanent_uuid: "a344baa95900451dbaf349b68e662b27" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 43710 } } } I0703 01:42:09.221097 26426 raft_consensus.cc:2684] T dae8746553524b6a9eb2e3f177f658f1 P a344baa95900451dbaf349b68e662b27 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER a344baa95900451dbaf349b68e662b27 (127.5.240.199) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } peers { permanent_uuid: "a344baa95900451dbaf349b68e662b27" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 43710 } } } I0703 01:42:09.223081 26754 rpcz_store.cc:265] Call kudu.consensus.ConsensusService.ChangeConfig from 127.5.240.194:45754 (request call id 1) took 1055ms. Request Metrics: {"dns_us":55,"raft.queue_time_us":60,"raft.run_cpu_time_us":206,"raft.run_wall_time_us":206,"spinlock_wait_cycles":168192,"thread_start_us":173,"threads_started":1,"wal-append.queue_time_us":290,"wal-append.run_cpu_time_us":452,"wal-append.run_wall_time_us":1000560} I0703 01:42:09.224824 24793 catalog_manager.cc:3798] T dae8746553524b6a9eb2e3f177f658f1 P 8ec1760d41f842f398d90e47ab52973b reported cstate change: config changed from index -1 to 4, VOTER a344baa95900451dbaf349b68e662b27 (127.5.240.199) added. New cstate: current_term: 1 leader_uuid: "8ec1760d41f842f398d90e47ab52973b" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } peers { permanent_uuid: "a344baa95900451dbaf349b68e662b27" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 43710 } } } I0703 01:42:09.246991 25043 consensus_queue.cc:204] T 72ebab6b7393457db959684913708522 P 8ec1760d41f842f398d90e47ab52973b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } peers { permanent_uuid: "ca6ae9994a78460ca6c4ae4ec3f4e9b5" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 41121 } } W0703 01:42:09.252779 24977 consensus_peers.cc:433] T 72ebab6b7393457db959684913708522 P 8ec1760d41f842f398d90e47ab52973b -> Peer ca6ae9994a78460ca6c4ae4ec3f4e9b5 (127.5.240.196:41121): Couldn't send request to peer ca6ae9994a78460ca6c4ae4ec3f4e9b5 for tablet 72ebab6b7393457db959684913708522. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 72ebab6b7393457db959684913708522. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:42:09.765314 26781 ts_tablet_manager.cc:669] T 72ebab6b7393457db959684913708522 P ca6ae9994a78460ca6c4ae4ec3f4e9b5: Initiating tablet copy from peer 8ec1760d41f842f398d90e47ab52973b (127.5.240.194:46721) I0703 01:42:09.765476 26781 tablet_copy_client.cc:239] T 72ebab6b7393457db959684913708522 P ca6ae9994a78460ca6c4ae4ec3f4e9b5: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.194:46721 I0703 01:42:09.769775 25063 tablet_copy_service.cc:135] P 8ec1760d41f842f398d90e47ab52973b: Received BeginTabletCopySession request for tablet 72ebab6b7393457db959684913708522 from peer ca6ae9994a78460ca6c4ae4ec3f4e9b5 ({username='jenkins'} at 127.5.240.196:32861) I0703 01:42:09.769877 25063 tablet_copy_service.cc:156] P 8ec1760d41f842f398d90e47ab52973b: Beginning new tablet copy session on tablet 72ebab6b7393457db959684913708522 from peer ca6ae9994a78460ca6c4ae4ec3f4e9b5 at {username='jenkins'} at 127.5.240.196:32861: session id = ca6ae9994a78460ca6c4ae4ec3f4e9b5-72ebab6b7393457db959684913708522 I0703 01:42:09.771670 25063 tablet_copy_source_session.cc:202] T 72ebab6b7393457db959684913708522 P 8ec1760d41f842f398d90e47ab52973b: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:42:09.772795 26781 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 72ebab6b7393457db959684913708522. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:09.775832 26781 ts_tablet_manager.cc:1173] T 72ebab6b7393457db959684913708522 P ca6ae9994a78460ca6c4ae4ec3f4e9b5: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:42:09.776018 26781 tablet_copy_client.cc:566] T 72ebab6b7393457db959684913708522 P ca6ae9994a78460ca6c4ae4ec3f4e9b5: tablet copy: Starting download of 0 data blocks... I0703 01:42:09.776182 26781 tablet_copy_client.cc:529] T 72ebab6b7393457db959684913708522 P ca6ae9994a78460ca6c4ae4ec3f4e9b5: tablet copy: Starting download of 1 WAL segments... I0703 01:42:09.777798 26781 tablet_copy_client.cc:405] T 72ebab6b7393457db959684913708522 P ca6ae9994a78460ca6c4ae4ec3f4e9b5: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:42:09.779196 26781 ts_tablet_manager.cc:1028] T 72ebab6b7393457db959684913708522 P ca6ae9994a78460ca6c4ae4ec3f4e9b5: Bootstrapping tablet I0703 01:42:09.779311 26781 tablet_bootstrap.cc:427] T 72ebab6b7393457db959684913708522 P ca6ae9994a78460ca6c4ae4ec3f4e9b5: Bootstrap starting. I0703 01:42:09.814187 26781 tablet_bootstrap.cc:427] T 72ebab6b7393457db959684913708522 P ca6ae9994a78460ca6c4ae4ec3f4e9b5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=656 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates I0703 01:42:09.814465 26781 tablet_bootstrap.cc:427] T 72ebab6b7393457db959684913708522 P ca6ae9994a78460ca6c4ae4ec3f4e9b5: Bootstrap complete. I0703 01:42:09.814625 26781 ts_tablet_manager.cc:1045] T 72ebab6b7393457db959684913708522 P ca6ae9994a78460ca6c4ae4ec3f4e9b5: Time spent bootstrapping tablet: real 0.035s user 0.033s sys 0.002s I0703 01:42:09.815171 26781 raft_consensus.cc:325] T 72ebab6b7393457db959684913708522 P ca6ae9994a78460ca6c4ae4ec3f4e9b5 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } I0703 01:42:09.815471 26781 raft_consensus.cc:628] T 72ebab6b7393457db959684913708522 P ca6ae9994a78460ca6c4ae4ec3f4e9b5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca6ae9994a78460ca6c4ae4ec3f4e9b5, State: Initialized, Role: FOLLOWER I0703 01:42:09.815577 26781 consensus_queue.cc:227] T 72ebab6b7393457db959684913708522 P ca6ae9994a78460ca6c4ae4ec3f4e9b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } peers { permanent_uuid: "ca6ae9994a78460ca6c4ae4ec3f4e9b5" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 41121 } } I0703 01:42:09.816401 26781 ts_tablet_manager.cc:1073] T 72ebab6b7393457db959684913708522 P ca6ae9994a78460ca6c4ae4ec3f4e9b5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s I0703 01:42:09.817078 25063 tablet_copy_service.cc:325] P 8ec1760d41f842f398d90e47ab52973b: Request end of tablet copy session ca6ae9994a78460ca6c4ae4ec3f4e9b5-72ebab6b7393457db959684913708522 received from {username='jenkins'} at 127.5.240.196:32861 I0703 01:42:09.817142 25063 tablet_copy_service.cc:413] P 8ec1760d41f842f398d90e47ab52973b: ending tablet copy session ca6ae9994a78460ca6c4ae4ec3f4e9b5-72ebab6b7393457db959684913708522 on tablet 72ebab6b7393457db959684913708522 with peer ca6ae9994a78460ca6c4ae4ec3f4e9b5 I0703 01:42:10.358184 26061 raft_consensus.cc:1026] T 72ebab6b7393457db959684913708522 P ca6ae9994a78460ca6c4ae4ec3f4e9b5 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[] I0703 01:42:10.359423 26787 raft_consensus.cc:2684] T 72ebab6b7393457db959684913708522 P 8ec1760d41f842f398d90e47ab52973b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ca6ae9994a78460ca6c4ae4ec3f4e9b5 (127.5.240.196) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } peers { permanent_uuid: "ca6ae9994a78460ca6c4ae4ec3f4e9b5" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 41121 } } } I0703 01:42:10.360364 26061 raft_consensus.cc:2684] T 72ebab6b7393457db959684913708522 P ca6ae9994a78460ca6c4ae4ec3f4e9b5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ca6ae9994a78460ca6c4ae4ec3f4e9b5 (127.5.240.196) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } peers { permanent_uuid: "ca6ae9994a78460ca6c4ae4ec3f4e9b5" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 41121 } } } I0703 01:42:10.362753 26787 rpcz_store.cc:265] Call kudu.consensus.ConsensusService.ChangeConfig from 127.5.240.194:45757 (request call id 1) took 1116ms. Request Metrics: {"dns_us":53,"raft.queue_time_us":54,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":145,"spinlock_wait_cycles":4992,"thread_start_us":226,"threads_started":1,"wal-append.queue_time_us":348,"wal-append.run_cpu_time_us":386,"wal-append.run_wall_time_us":1000487} I0703 01:42:10.364801 24793 catalog_manager.cc:3798] T 72ebab6b7393457db959684913708522 P ca6ae9994a78460ca6c4ae4ec3f4e9b5 reported cstate change: config changed from index -1 to 4, VOTER ca6ae9994a78460ca6c4ae4ec3f4e9b5 (127.5.240.196) added. New cstate: current_term: 1 leader_uuid: "8ec1760d41f842f398d90e47ab52973b" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } peers { permanent_uuid: "ca6ae9994a78460ca6c4ae4ec3f4e9b5" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 41121 } } } I0703 01:42:10.396658 24920 consensus_queue.cc:204] T 0a2438297555462aa42cb91089c7a618 P dfd7ad6293204fc89f9e72436796698b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } peers { permanent_uuid: "01f6144ac0484e1d9a6da7ff80ba71f0" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 35740 } } W0703 01:42:10.398501 24854 consensus_peers.cc:433] T 0a2438297555462aa42cb91089c7a618 P dfd7ad6293204fc89f9e72436796698b -> Peer 01f6144ac0484e1d9a6da7ff80ba71f0 (127.5.240.198:35740): Couldn't send request to peer 01f6144ac0484e1d9a6da7ff80ba71f0 for tablet 0a2438297555462aa42cb91089c7a618. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0a2438297555462aa42cb91089c7a618. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:42:10.940637 26812 ts_tablet_manager.cc:669] T 0a2438297555462aa42cb91089c7a618 P 01f6144ac0484e1d9a6da7ff80ba71f0: Initiating tablet copy from peer dfd7ad6293204fc89f9e72436796698b (127.5.240.193:43198) I0703 01:42:10.940784 26812 tablet_copy_client.cc:239] T 0a2438297555462aa42cb91089c7a618 P 01f6144ac0484e1d9a6da7ff80ba71f0: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.193:43198 I0703 01:42:10.941417 24940 tablet_copy_service.cc:135] P dfd7ad6293204fc89f9e72436796698b: Received BeginTabletCopySession request for tablet 0a2438297555462aa42cb91089c7a618 from peer 01f6144ac0484e1d9a6da7ff80ba71f0 ({username='jenkins'} at 127.5.240.198:49776) I0703 01:42:10.941527 24940 tablet_copy_service.cc:156] P dfd7ad6293204fc89f9e72436796698b: Beginning new tablet copy session on tablet 0a2438297555462aa42cb91089c7a618 from peer 01f6144ac0484e1d9a6da7ff80ba71f0 at {username='jenkins'} at 127.5.240.198:49776: session id = 01f6144ac0484e1d9a6da7ff80ba71f0-0a2438297555462aa42cb91089c7a618 I0703 01:42:10.943223 24940 tablet_copy_source_session.cc:202] T 0a2438297555462aa42cb91089c7a618 P dfd7ad6293204fc89f9e72436796698b: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:42:10.944290 26812 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 0a2438297555462aa42cb91089c7a618. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:10.947065 26812 ts_tablet_manager.cc:1173] T 0a2438297555462aa42cb91089c7a618 P 01f6144ac0484e1d9a6da7ff80ba71f0: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:42:10.947235 26812 tablet_copy_client.cc:566] T 0a2438297555462aa42cb91089c7a618 P 01f6144ac0484e1d9a6da7ff80ba71f0: tablet copy: Starting download of 0 data blocks... I0703 01:42:10.947463 26812 tablet_copy_client.cc:529] T 0a2438297555462aa42cb91089c7a618 P 01f6144ac0484e1d9a6da7ff80ba71f0: tablet copy: Starting download of 1 WAL segments... I0703 01:42:10.949162 26812 tablet_copy_client.cc:405] T 0a2438297555462aa42cb91089c7a618 P 01f6144ac0484e1d9a6da7ff80ba71f0: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:42:10.950551 26812 ts_tablet_manager.cc:1028] T 0a2438297555462aa42cb91089c7a618 P 01f6144ac0484e1d9a6da7ff80ba71f0: Bootstrapping tablet I0703 01:42:10.950690 26812 tablet_bootstrap.cc:427] T 0a2438297555462aa42cb91089c7a618 P 01f6144ac0484e1d9a6da7ff80ba71f0: Bootstrap starting. I0703 01:42:10.985556 26812 tablet_bootstrap.cc:427] T 0a2438297555462aa42cb91089c7a618 P 01f6144ac0484e1d9a6da7ff80ba71f0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=653 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates I0703 01:42:10.985942 26812 tablet_bootstrap.cc:427] T 0a2438297555462aa42cb91089c7a618 P 01f6144ac0484e1d9a6da7ff80ba71f0: Bootstrap complete. I0703 01:42:10.986090 26812 ts_tablet_manager.cc:1045] T 0a2438297555462aa42cb91089c7a618 P 01f6144ac0484e1d9a6da7ff80ba71f0: Time spent bootstrapping tablet: real 0.035s user 0.036s sys 0.000s I0703 01:42:10.986692 26812 raft_consensus.cc:325] T 0a2438297555462aa42cb91089c7a618 P 01f6144ac0484e1d9a6da7ff80ba71f0 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } I0703 01:42:10.987000 26812 raft_consensus.cc:628] T 0a2438297555462aa42cb91089c7a618 P 01f6144ac0484e1d9a6da7ff80ba71f0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01f6144ac0484e1d9a6da7ff80ba71f0, State: Initialized, Role: FOLLOWER I0703 01:42:10.987112 26812 consensus_queue.cc:227] T 0a2438297555462aa42cb91089c7a618 P 01f6144ac0484e1d9a6da7ff80ba71f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } peers { permanent_uuid: "01f6144ac0484e1d9a6da7ff80ba71f0" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 35740 } } I0703 01:42:10.987840 26812 ts_tablet_manager.cc:1073] T 0a2438297555462aa42cb91089c7a618 P 01f6144ac0484e1d9a6da7ff80ba71f0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s I0703 01:42:10.988541 24940 tablet_copy_service.cc:325] P dfd7ad6293204fc89f9e72436796698b: Request end of tablet copy session 01f6144ac0484e1d9a6da7ff80ba71f0-0a2438297555462aa42cb91089c7a618 received from {username='jenkins'} at 127.5.240.198:49776 I0703 01:42:10.988657 24940 tablet_copy_service.cc:413] P dfd7ad6293204fc89f9e72436796698b: ending tablet copy session 01f6144ac0484e1d9a6da7ff80ba71f0-0a2438297555462aa42cb91089c7a618 on tablet 0a2438297555462aa42cb91089c7a618 with peer 01f6144ac0484e1d9a6da7ff80ba71f0 I0703 01:42:11.334465 26305 raft_consensus.cc:1026] T 0a2438297555462aa42cb91089c7a618 P 01f6144ac0484e1d9a6da7ff80ba71f0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[] I0703 01:42:11.335824 26815 raft_consensus.cc:2684] T 0a2438297555462aa42cb91089c7a618 P dfd7ad6293204fc89f9e72436796698b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 01f6144ac0484e1d9a6da7ff80ba71f0 (127.5.240.198) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } peers { permanent_uuid: "01f6144ac0484e1d9a6da7ff80ba71f0" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 35740 } } } I0703 01:42:11.336278 26305 raft_consensus.cc:2684] T 0a2438297555462aa42cb91089c7a618 P 01f6144ac0484e1d9a6da7ff80ba71f0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 01f6144ac0484e1d9a6da7ff80ba71f0 (127.5.240.198) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } peers { permanent_uuid: "01f6144ac0484e1d9a6da7ff80ba71f0" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 35740 } } } I0703 01:42:11.339287 24793 catalog_manager.cc:3798] T 0a2438297555462aa42cb91089c7a618 P dfd7ad6293204fc89f9e72436796698b reported cstate change: config changed from index -1 to 4, VOTER 01f6144ac0484e1d9a6da7ff80ba71f0 (127.5.240.198) added. New cstate: current_term: 1 leader_uuid: "dfd7ad6293204fc89f9e72436796698b" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } peers { permanent_uuid: "01f6144ac0484e1d9a6da7ff80ba71f0" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 35740 } } } I0703 01:42:11.450050 25043 tablet_service.cc:1140] Received LeaderStepDown RPC: tablet_id: "72ebab6b7393457db959684913708522" dest_uuid: "8ec1760d41f842f398d90e47ab52973b" from {username='jenkins'} at 127.5.240.194:45772 I0703 01:42:11.450192 25043 raft_consensus.cc:552] T 72ebab6b7393457db959684913708522 P 8ec1760d41f842f398d90e47ab52973b [term 1 LEADER]: Received request to step down I0703 01:42:11.450238 25043 raft_consensus.cc:628] T 72ebab6b7393457db959684913708522 P 8ec1760d41f842f398d90e47ab52973b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8ec1760d41f842f398d90e47ab52973b, State: Running, Role: LEADER I0703 01:42:11.450428 25043 consensus_queue.cc:227] T 72ebab6b7393457db959684913708522 P 8ec1760d41f842f398d90e47ab52973b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } peers { permanent_uuid: "ca6ae9994a78460ca6c4ae4ec3f4e9b5" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 41121 } } I0703 01:42:11.450604 25043 consensus_peers.cc:455] T 72ebab6b7393457db959684913708522 P 8ec1760d41f842f398d90e47ab52973b -> Peer ca6ae9994a78460ca6c4ae4ec3f4e9b5 (127.5.240.196:41121): Closing peer: ca6ae9994a78460ca6c4ae4ec3f4e9b5 I0703 01:42:11.450788 25043 raft_consensus.cc:2730] T 72ebab6b7393457db959684913708522 P 8ec1760d41f842f398d90e47ab52973b: Snoozing failure detection for 3.000s (explicit stepdown request) I0703 01:42:11.635542 25043 tablet_service.cc:1140] Received LeaderStepDown RPC: tablet_id: "9a818723ff79493aa17e7434040e8975" dest_uuid: "8ec1760d41f842f398d90e47ab52973b" from {username='jenkins'} at 127.5.240.194:45794 I0703 01:42:11.635668 25043 raft_consensus.cc:552] T 9a818723ff79493aa17e7434040e8975 P 8ec1760d41f842f398d90e47ab52973b [term 1 LEADER]: Received request to step down I0703 01:42:11.635713 25043 raft_consensus.cc:628] T 9a818723ff79493aa17e7434040e8975 P 8ec1760d41f842f398d90e47ab52973b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8ec1760d41f842f398d90e47ab52973b, State: Running, Role: LEADER I0703 01:42:11.635892 25043 consensus_queue.cc:227] T 9a818723ff79493aa17e7434040e8975 P 8ec1760d41f842f398d90e47ab52973b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } peers { permanent_uuid: "2f7e2b9db52f44ba81b7393403f9ed37" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 45230 } } I0703 01:42:11.636015 25043 consensus_peers.cc:455] T 9a818723ff79493aa17e7434040e8975 P 8ec1760d41f842f398d90e47ab52973b -> Peer 2f7e2b9db52f44ba81b7393403f9ed37 (127.5.240.197:45230): Closing peer: 2f7e2b9db52f44ba81b7393403f9ed37 I0703 01:42:11.636112 25043 raft_consensus.cc:2730] T 9a818723ff79493aa17e7434040e8975 P 8ec1760d41f842f398d90e47ab52973b: Snoozing failure detection for 3.000s (explicit stepdown request) I0703 01:42:11.822597 24920 tablet_service.cc:1140] Received LeaderStepDown RPC: tablet_id: "0a2438297555462aa42cb91089c7a618" dest_uuid: "dfd7ad6293204fc89f9e72436796698b" from {username='jenkins'} at 127.5.240.193:50006 I0703 01:42:11.822773 24920 raft_consensus.cc:552] T 0a2438297555462aa42cb91089c7a618 P dfd7ad6293204fc89f9e72436796698b [term 1 LEADER]: Received request to step down I0703 01:42:11.822830 24920 raft_consensus.cc:628] T 0a2438297555462aa42cb91089c7a618 P dfd7ad6293204fc89f9e72436796698b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dfd7ad6293204fc89f9e72436796698b, State: Running, Role: LEADER I0703 01:42:11.823042 24920 consensus_queue.cc:227] T 0a2438297555462aa42cb91089c7a618 P dfd7ad6293204fc89f9e72436796698b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } peers { permanent_uuid: "01f6144ac0484e1d9a6da7ff80ba71f0" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 35740 } } I0703 01:42:11.823175 24920 consensus_peers.cc:455] T 0a2438297555462aa42cb91089c7a618 P dfd7ad6293204fc89f9e72436796698b -> Peer 01f6144ac0484e1d9a6da7ff80ba71f0 (127.5.240.198:35740): Closing peer: 01f6144ac0484e1d9a6da7ff80ba71f0 I0703 01:42:11.823361 24920 raft_consensus.cc:2730] T 0a2438297555462aa42cb91089c7a618 P dfd7ad6293204fc89f9e72436796698b: Snoozing failure detection for 3.000s (explicit stepdown request) I0703 01:42:12.008697 24920 tablet_service.cc:1140] Received LeaderStepDown RPC: tablet_id: "53242d0289d644faa0149e0a54c0984c" dest_uuid: "dfd7ad6293204fc89f9e72436796698b" from {username='jenkins'} at 127.5.240.193:50028 I0703 01:42:12.008841 24920 raft_consensus.cc:552] T 53242d0289d644faa0149e0a54c0984c P dfd7ad6293204fc89f9e72436796698b [term 1 LEADER]: Received request to step down I0703 01:42:12.008890 24920 raft_consensus.cc:628] T 53242d0289d644faa0149e0a54c0984c P dfd7ad6293204fc89f9e72436796698b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dfd7ad6293204fc89f9e72436796698b, State: Running, Role: LEADER I0703 01:42:12.009109 24920 consensus_queue.cc:227] T 53242d0289d644faa0149e0a54c0984c P dfd7ad6293204fc89f9e72436796698b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } peers { permanent_uuid: "cff51d1a4baa4cd0a1625e1d7db94f2c" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 38199 } } I0703 01:42:12.009249 24920 consensus_peers.cc:455] T 53242d0289d644faa0149e0a54c0984c P dfd7ad6293204fc89f9e72436796698b -> Peer cff51d1a4baa4cd0a1625e1d7db94f2c (127.5.240.195:38199): Closing peer: cff51d1a4baa4cd0a1625e1d7db94f2c I0703 01:42:12.009363 24920 raft_consensus.cc:2730] T 53242d0289d644faa0149e0a54c0984c P dfd7ad6293204fc89f9e72436796698b: Snoozing failure detection for 3.000s (explicit stepdown request) I0703 01:42:12.195775 24920 tablet_service.cc:1140] Received LeaderStepDown RPC: tablet_id: "ff7fa826668f419fa3c5225e5d810b9c" dest_uuid: "dfd7ad6293204fc89f9e72436796698b" from {username='jenkins'} at 127.5.240.193:50050 I0703 01:42:12.195911 24920 raft_consensus.cc:552] T ff7fa826668f419fa3c5225e5d810b9c P dfd7ad6293204fc89f9e72436796698b [term 1 LEADER]: Received request to step down I0703 01:42:12.195962 24920 raft_consensus.cc:628] T ff7fa826668f419fa3c5225e5d810b9c P dfd7ad6293204fc89f9e72436796698b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dfd7ad6293204fc89f9e72436796698b, State: Running, Role: LEADER I0703 01:42:12.196169 24920 consensus_queue.cc:227] T ff7fa826668f419fa3c5225e5d810b9c P dfd7ad6293204fc89f9e72436796698b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } peers { permanent_uuid: "ca6ae9994a78460ca6c4ae4ec3f4e9b5" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 41121 } } I0703 01:42:12.196297 24920 consensus_peers.cc:455] T ff7fa826668f419fa3c5225e5d810b9c P dfd7ad6293204fc89f9e72436796698b -> Peer ca6ae9994a78460ca6c4ae4ec3f4e9b5 (127.5.240.196:41121): Closing peer: ca6ae9994a78460ca6c4ae4ec3f4e9b5 I0703 01:42:12.196409 24920 raft_consensus.cc:2730] T ff7fa826668f419fa3c5225e5d810b9c P dfd7ad6293204fc89f9e72436796698b: Snoozing failure detection for 3.000s (explicit stepdown request) I0703 01:42:12.378016 25043 tablet_service.cc:1140] Received LeaderStepDown RPC: tablet_id: "dae8746553524b6a9eb2e3f177f658f1" dest_uuid: "8ec1760d41f842f398d90e47ab52973b" from {username='jenkins'} at 127.5.240.194:45881 I0703 01:42:12.378145 25043 raft_consensus.cc:552] T dae8746553524b6a9eb2e3f177f658f1 P 8ec1760d41f842f398d90e47ab52973b [term 1 LEADER]: Received request to step down I0703 01:42:12.378202 25043 raft_consensus.cc:628] T dae8746553524b6a9eb2e3f177f658f1 P 8ec1760d41f842f398d90e47ab52973b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8ec1760d41f842f398d90e47ab52973b, State: Running, Role: LEADER I0703 01:42:12.378427 25043 consensus_queue.cc:227] T dae8746553524b6a9eb2e3f177f658f1 P 8ec1760d41f842f398d90e47ab52973b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } peers { permanent_uuid: "a344baa95900451dbaf349b68e662b27" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 43710 } } I0703 01:42:12.378638 25043 consensus_peers.cc:455] T dae8746553524b6a9eb2e3f177f658f1 P 8ec1760d41f842f398d90e47ab52973b -> Peer a344baa95900451dbaf349b68e662b27 (127.5.240.199:43710): Closing peer: a344baa95900451dbaf349b68e662b27 I0703 01:42:12.378752 25043 raft_consensus.cc:2730] T dae8746553524b6a9eb2e3f177f658f1 P 8ec1760d41f842f398d90e47ab52973b: Snoozing failure detection for 3.000s (explicit stepdown request) I0703 01:42:12.558274 24920 tablet_service.cc:1140] Received LeaderStepDown RPC: tablet_id: "6359db2dd602463e841abc2a907c49a3" dest_uuid: "dfd7ad6293204fc89f9e72436796698b" from {username='jenkins'} at 127.5.240.193:50093 I0703 01:42:12.558409 24920 raft_consensus.cc:552] T 6359db2dd602463e841abc2a907c49a3 P dfd7ad6293204fc89f9e72436796698b [term 1 LEADER]: Received request to step down I0703 01:42:12.558462 24920 raft_consensus.cc:628] T 6359db2dd602463e841abc2a907c49a3 P dfd7ad6293204fc89f9e72436796698b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dfd7ad6293204fc89f9e72436796698b, State: Running, Role: LEADER I0703 01:42:12.558692 24920 consensus_queue.cc:227] T 6359db2dd602463e841abc2a907c49a3 P dfd7ad6293204fc89f9e72436796698b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } peers { permanent_uuid: "01f6144ac0484e1d9a6da7ff80ba71f0" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 35740 } } I0703 01:42:12.558828 24920 consensus_peers.cc:455] T 6359db2dd602463e841abc2a907c49a3 P dfd7ad6293204fc89f9e72436796698b -> Peer 01f6144ac0484e1d9a6da7ff80ba71f0 (127.5.240.198:35740): Closing peer: 01f6144ac0484e1d9a6da7ff80ba71f0 I0703 01:42:12.558949 24920 raft_consensus.cc:2730] T 6359db2dd602463e841abc2a907c49a3 P dfd7ad6293204fc89f9e72436796698b: Snoozing failure detection for 3.000s (explicit stepdown request) I0703 01:42:12.746834 25043 tablet_service.cc:1140] Received LeaderStepDown RPC: tablet_id: "fd7030608be84fa9b61c3ae90e04e27e" dest_uuid: "8ec1760d41f842f398d90e47ab52973b" from {username='jenkins'} at 127.5.240.194:45925 I0703 01:42:12.746971 25043 raft_consensus.cc:552] T fd7030608be84fa9b61c3ae90e04e27e P 8ec1760d41f842f398d90e47ab52973b [term 1 LEADER]: Received request to step down I0703 01:42:12.747025 25043 raft_consensus.cc:628] T fd7030608be84fa9b61c3ae90e04e27e P 8ec1760d41f842f398d90e47ab52973b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8ec1760d41f842f398d90e47ab52973b, State: Running, Role: LEADER I0703 01:42:12.747207 25043 consensus_queue.cc:227] T fd7030608be84fa9b61c3ae90e04e27e P 8ec1760d41f842f398d90e47ab52973b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } peers { permanent_uuid: "a344baa95900451dbaf349b68e662b27" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 43710 } } I0703 01:42:12.747354 25043 consensus_peers.cc:455] T fd7030608be84fa9b61c3ae90e04e27e P 8ec1760d41f842f398d90e47ab52973b -> Peer a344baa95900451dbaf349b68e662b27 (127.5.240.199:43710): Closing peer: a344baa95900451dbaf349b68e662b27 I0703 01:42:12.747460 25043 raft_consensus.cc:2730] T fd7030608be84fa9b61c3ae90e04e27e P 8ec1760d41f842f398d90e47ab52973b: Snoozing failure detection for 3.000s (explicit stepdown request) I0703 01:42:12.760943 27285 raft_consensus.cc:457] T 72ebab6b7393457db959684913708522 P ca6ae9994a78460ca6c4ae4ec3f4e9b5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8ec1760d41f842f398d90e47ab52973b) I0703 01:42:12.761057 27285 raft_consensus.cc:2730] T 72ebab6b7393457db959684913708522 P ca6ae9994a78460ca6c4ae4ec3f4e9b5: Snoozing failure detection for 2.009s (starting election) I0703 01:42:12.761191 27285 raft_consensus.cc:479] T 72ebab6b7393457db959684913708522 P ca6ae9994a78460ca6c4ae4ec3f4e9b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } peers { permanent_uuid: "ca6ae9994a78460ca6c4ae4ec3f4e9b5" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 41121 } } I0703 01:42:12.761997 27285 leader_election.cc:243] T 72ebab6b7393457db959684913708522 P ca6ae9994a78460ca6c4ae4ec3f4e9b5 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 8ec1760d41f842f398d90e47ab52973b (127.5.240.194:46721) I0703 01:42:12.763070 25043 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "72ebab6b7393457db959684913708522" candidate_uuid: "ca6ae9994a78460ca6c4ae4ec3f4e9b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8ec1760d41f842f398d90e47ab52973b" is_pre_election: true I0703 01:42:12.763269 25043 raft_consensus.cc:2730] T 72ebab6b7393457db959684913708522 P 8ec1760d41f842f398d90e47ab52973b: Snoozing failure detection for 2.091s (vote granted) I0703 01:42:12.763351 25043 raft_consensus.cc:2243] T 72ebab6b7393457db959684913708522 P 8ec1760d41f842f398d90e47ab52973b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca6ae9994a78460ca6c4ae4ec3f4e9b5 in term 1. I0703 01:42:12.764034 25994 leader_election.cc:379] T 72ebab6b7393457db959684913708522 P ca6ae9994a78460ca6c4ae4ec3f4e9b5 [CANDIDATE]: Term 2 pre-election: Vote granted by peer 8ec1760d41f842f398d90e47ab52973b (127.5.240.194:46721) I0703 01:42:12.764107 25994 leader_election.cc:270] T 72ebab6b7393457db959684913708522 P ca6ae9994a78460ca6c4ae4ec3f4e9b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. I0703 01:42:12.764403 27285 raft_consensus.cc:2730] T 72ebab6b7393457db959684913708522 P ca6ae9994a78460ca6c4ae4ec3f4e9b5: Snoozing failure detection for 1.608s (election complete) I0703 01:42:12.764508 27285 raft_consensus.cc:2537] T 72ebab6b7393457db959684913708522 P ca6ae9994a78460ca6c4ae4ec3f4e9b5 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0703 01:42:12.764605 27285 raft_consensus.cc:457] T 72ebab6b7393457db959684913708522 P ca6ae9994a78460ca6c4ae4ec3f4e9b5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8ec1760d41f842f398d90e47ab52973b) I0703 01:42:12.764658 27285 raft_consensus.cc:2730] T 72ebab6b7393457db959684913708522 P ca6ae9994a78460ca6c4ae4ec3f4e9b5: Snoozing failure detection for 1.559s (starting election) I0703 01:42:12.764720 27285 raft_consensus.cc:2782] T 72ebab6b7393457db959684913708522 P ca6ae9994a78460ca6c4ae4ec3f4e9b5 [term 1 FOLLOWER]: Advancing to term 2 I0703 01:42:12.766306 27285 raft_consensus.cc:479] T 72ebab6b7393457db959684913708522 P ca6ae9994a78460ca6c4ae4ec3f4e9b5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } peers { permanent_uuid: "ca6ae9994a78460ca6c4ae4ec3f4e9b5" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 41121 } } I0703 01:42:12.766564 27285 leader_election.cc:243] T 72ebab6b7393457db959684913708522 P ca6ae9994a78460ca6c4ae4ec3f4e9b5 [CANDIDATE]: Term 2 election: Requesting vote from peer 8ec1760d41f842f398d90e47ab52973b (127.5.240.194:46721) I0703 01:42:12.767248 25043 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "72ebab6b7393457db959684913708522" candidate_uuid: "ca6ae9994a78460ca6c4ae4ec3f4e9b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8ec1760d41f842f398d90e47ab52973b" I0703 01:42:12.767357 25043 raft_consensus.cc:2782] T 72ebab6b7393457db959684913708522 P 8ec1760d41f842f398d90e47ab52973b [term 1 FOLLOWER]: Advancing to term 2 I0703 01:42:12.767433 25043 raft_consensus.cc:2730] T 72ebab6b7393457db959684913708522 P 8ec1760d41f842f398d90e47ab52973b: Snoozing failure detection for 1.794s (vote granted) I0703 01:42:12.769199 25043 raft_consensus.cc:2243] T 72ebab6b7393457db959684913708522 P 8ec1760d41f842f398d90e47ab52973b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca6ae9994a78460ca6c4ae4ec3f4e9b5 in term 2. I0703 01:42:12.769642 25994 leader_election.cc:379] T 72ebab6b7393457db959684913708522 P ca6ae9994a78460ca6c4ae4ec3f4e9b5 [CANDIDATE]: Term 2 election: Vote granted by peer 8ec1760d41f842f398d90e47ab52973b (127.5.240.194:46721) I0703 01:42:12.769707 25994 leader_election.cc:270] T 72ebab6b7393457db959684913708522 P ca6ae9994a78460ca6c4ae4ec3f4e9b5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0703 01:42:12.769953 27285 raft_consensus.cc:2730] T 72ebab6b7393457db959684913708522 P ca6ae9994a78460ca6c4ae4ec3f4e9b5: Snoozing failure detection for 1.896s (election complete) I0703 01:42:12.770021 27285 raft_consensus.cc:2537] T 72ebab6b7393457db959684913708522 P ca6ae9994a78460ca6c4ae4ec3f4e9b5 [term 2 FOLLOWER]: Leader election won for term 2 I0703 01:42:12.770505 27285 raft_consensus.cc:593] T 72ebab6b7393457db959684913708522 P ca6ae9994a78460ca6c4ae4ec3f4e9b5 [term 2 LEADER]: Becoming Leader. State: Replica: ca6ae9994a78460ca6c4ae4ec3f4e9b5, State: Running, Role: LEADER I0703 01:42:12.771189 27285 consensus_queue.cc:204] T 72ebab6b7393457db959684913708522 P ca6ae9994a78460ca6c4ae4ec3f4e9b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } peers { permanent_uuid: "ca6ae9994a78460ca6c4ae4ec3f4e9b5" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 41121 } } I0703 01:42:12.774058 24797 catalog_manager.cc:3798] T 72ebab6b7393457db959684913708522 P ca6ae9994a78460ca6c4ae4ec3f4e9b5 reported cstate change: term changed from 1 to 2, leader changed from 8ec1760d41f842f398d90e47ab52973b (127.5.240.194) to ca6ae9994a78460ca6c4ae4ec3f4e9b5 (127.5.240.196). New cstate: current_term: 2 leader_uuid: "ca6ae9994a78460ca6c4ae4ec3f4e9b5" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } peers { permanent_uuid: "ca6ae9994a78460ca6c4ae4ec3f4e9b5" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 41121 } } } I0703 01:42:12.937804 24920 tablet_service.cc:1140] Received LeaderStepDown RPC: tablet_id: "7d1747555a45487e869ed52665686c52" dest_uuid: "dfd7ad6293204fc89f9e72436796698b" from {username='jenkins'} at 127.5.240.193:50137 I0703 01:42:12.937943 24920 raft_consensus.cc:552] T 7d1747555a45487e869ed52665686c52 P dfd7ad6293204fc89f9e72436796698b [term 1 LEADER]: Received request to step down I0703 01:42:12.937991 24920 raft_consensus.cc:628] T 7d1747555a45487e869ed52665686c52 P dfd7ad6293204fc89f9e72436796698b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dfd7ad6293204fc89f9e72436796698b, State: Running, Role: LEADER I0703 01:42:12.938168 24920 consensus_queue.cc:227] T 7d1747555a45487e869ed52665686c52 P dfd7ad6293204fc89f9e72436796698b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } peers { permanent_uuid: "2f7e2b9db52f44ba81b7393403f9ed37" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 45230 } } I0703 01:42:12.938292 24920 consensus_peers.cc:455] T 7d1747555a45487e869ed52665686c52 P dfd7ad6293204fc89f9e72436796698b -> Peer 2f7e2b9db52f44ba81b7393403f9ed37 (127.5.240.197:45230): Closing peer: 2f7e2b9db52f44ba81b7393403f9ed37 I0703 01:42:12.938408 24920 raft_consensus.cc:2730] T 7d1747555a45487e869ed52665686c52 P dfd7ad6293204fc89f9e72436796698b: Snoozing failure detection for 3.000s (explicit stepdown request) I0703 01:42:13.064543 27364 raft_consensus.cc:457] T 9a818723ff79493aa17e7434040e8975 P 2f7e2b9db52f44ba81b7393403f9ed37 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8ec1760d41f842f398d90e47ab52973b) I0703 01:42:13.064687 27364 raft_consensus.cc:2730] T 9a818723ff79493aa17e7434040e8975 P 2f7e2b9db52f44ba81b7393403f9ed37: Snoozing failure detection for 1.780s (starting election) I0703 01:42:13.064791 27364 raft_consensus.cc:479] T 9a818723ff79493aa17e7434040e8975 P 2f7e2b9db52f44ba81b7393403f9ed37 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } peers { permanent_uuid: "2f7e2b9db52f44ba81b7393403f9ed37" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 45230 } } I0703 01:42:13.065421 27364 leader_election.cc:243] T 9a818723ff79493aa17e7434040e8975 P 2f7e2b9db52f44ba81b7393403f9ed37 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 8ec1760d41f842f398d90e47ab52973b (127.5.240.194:46721) I0703 01:42:13.066395 25043 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "9a818723ff79493aa17e7434040e8975" candidate_uuid: "2f7e2b9db52f44ba81b7393403f9ed37" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8ec1760d41f842f398d90e47ab52973b" is_pre_election: true I0703 01:42:13.066568 25043 raft_consensus.cc:2730] T 9a818723ff79493aa17e7434040e8975 P 8ec1760d41f842f398d90e47ab52973b: Snoozing failure detection for 1.948s (vote granted) I0703 01:42:13.066638 25043 raft_consensus.cc:2243] T 9a818723ff79493aa17e7434040e8975 P 8ec1760d41f842f398d90e47ab52973b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f7e2b9db52f44ba81b7393403f9ed37 in term 1. I0703 01:42:13.067066 26116 leader_election.cc:379] T 9a818723ff79493aa17e7434040e8975 P 2f7e2b9db52f44ba81b7393403f9ed37 [CANDIDATE]: Term 2 pre-election: Vote granted by peer 8ec1760d41f842f398d90e47ab52973b (127.5.240.194:46721) I0703 01:42:13.067128 26116 leader_election.cc:270] T 9a818723ff79493aa17e7434040e8975 P 2f7e2b9db52f44ba81b7393403f9ed37 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. I0703 01:42:13.067364 27364 raft_consensus.cc:2730] T 9a818723ff79493aa17e7434040e8975 P 2f7e2b9db52f44ba81b7393403f9ed37: Snoozing failure detection for 2.226s (election complete) I0703 01:42:13.067456 27364 raft_consensus.cc:2537] T 9a818723ff79493aa17e7434040e8975 P 2f7e2b9db52f44ba81b7393403f9ed37 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0703 01:42:13.067523 27364 raft_consensus.cc:457] T 9a818723ff79493aa17e7434040e8975 P 2f7e2b9db52f44ba81b7393403f9ed37 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8ec1760d41f842f398d90e47ab52973b) I0703 01:42:13.067579 27364 raft_consensus.cc:2730] T 9a818723ff79493aa17e7434040e8975 P 2f7e2b9db52f44ba81b7393403f9ed37: Snoozing failure detection for 2.027s (starting election) I0703 01:42:13.067665 27364 raft_consensus.cc:2782] T 9a818723ff79493aa17e7434040e8975 P 2f7e2b9db52f44ba81b7393403f9ed37 [term 1 FOLLOWER]: Advancing to term 2 I0703 01:42:13.069051 27364 raft_consensus.cc:479] T 9a818723ff79493aa17e7434040e8975 P 2f7e2b9db52f44ba81b7393403f9ed37 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } peers { permanent_uuid: "2f7e2b9db52f44ba81b7393403f9ed37" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 45230 } } I0703 01:42:13.069301 27364 leader_election.cc:243] T 9a818723ff79493aa17e7434040e8975 P 2f7e2b9db52f44ba81b7393403f9ed37 [CANDIDATE]: Term 2 election: Requesting vote from peer 8ec1760d41f842f398d90e47ab52973b (127.5.240.194:46721) I0703 01:42:13.069880 25043 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "9a818723ff79493aa17e7434040e8975" candidate_uuid: "2f7e2b9db52f44ba81b7393403f9ed37" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8ec1760d41f842f398d90e47ab52973b" I0703 01:42:13.069988 25043 raft_consensus.cc:2782] T 9a818723ff79493aa17e7434040e8975 P 8ec1760d41f842f398d90e47ab52973b [term 1 FOLLOWER]: Advancing to term 2 I0703 01:42:13.070057 25043 raft_consensus.cc:2730] T 9a818723ff79493aa17e7434040e8975 P 8ec1760d41f842f398d90e47ab52973b: Snoozing failure detection for 1.941s (vote granted) I0703 01:42:13.071429 25043 raft_consensus.cc:2243] T 9a818723ff79493aa17e7434040e8975 P 8ec1760d41f842f398d90e47ab52973b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f7e2b9db52f44ba81b7393403f9ed37 in term 2. I0703 01:42:13.071817 26116 leader_election.cc:379] T 9a818723ff79493aa17e7434040e8975 P 2f7e2b9db52f44ba81b7393403f9ed37 [CANDIDATE]: Term 2 election: Vote granted by peer 8ec1760d41f842f398d90e47ab52973b (127.5.240.194:46721) I0703 01:42:13.071874 26116 leader_election.cc:270] T 9a818723ff79493aa17e7434040e8975 P 2f7e2b9db52f44ba81b7393403f9ed37 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0703 01:42:13.072016 27364 raft_consensus.cc:2730] T 9a818723ff79493aa17e7434040e8975 P 2f7e2b9db52f44ba81b7393403f9ed37: Snoozing failure detection for 1.985s (election complete) I0703 01:42:13.072087 27364 raft_consensus.cc:2537] T 9a818723ff79493aa17e7434040e8975 P 2f7e2b9db52f44ba81b7393403f9ed37 [term 2 FOLLOWER]: Leader election won for term 2 I0703 01:42:13.072365 27364 raft_consensus.cc:593] T 9a818723ff79493aa17e7434040e8975 P 2f7e2b9db52f44ba81b7393403f9ed37 [term 2 LEADER]: Becoming Leader. State: Replica: 2f7e2b9db52f44ba81b7393403f9ed37, State: Running, Role: LEADER I0703 01:42:13.072763 27364 consensus_queue.cc:204] T 9a818723ff79493aa17e7434040e8975 P 2f7e2b9db52f44ba81b7393403f9ed37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } peers { permanent_uuid: "2f7e2b9db52f44ba81b7393403f9ed37" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 45230 } } I0703 01:42:13.075143 24797 catalog_manager.cc:3798] T 9a818723ff79493aa17e7434040e8975 P 2f7e2b9db52f44ba81b7393403f9ed37 reported cstate change: term changed from 1 to 2, leader changed from 8ec1760d41f842f398d90e47ab52973b (127.5.240.194) to 2f7e2b9db52f44ba81b7393403f9ed37 (127.5.240.197). New cstate: current_term: 2 leader_uuid: "2f7e2b9db52f44ba81b7393403f9ed37" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } peers { permanent_uuid: "2f7e2b9db52f44ba81b7393403f9ed37" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 45230 } } } I0703 01:42:13.194164 25043 raft_consensus.cc:1081] T 72ebab6b7393457db959684913708522 P 8ec1760d41f842f398d90e47ab52973b [term 2 FOLLOWER]: Refusing update from remote peer ca6ae9994a78460ca6c4ae4ec3f4e9b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch) I0703 01:42:13.194931 27285 consensus_queue.cc:962] T 72ebab6b7393457db959684913708522 P ca6ae9994a78460ca6c4ae4ec3f4e9b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I0703 01:42:13.240041 27384 raft_consensus.cc:457] T 0a2438297555462aa42cb91089c7a618 P 01f6144ac0484e1d9a6da7ff80ba71f0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dfd7ad6293204fc89f9e72436796698b) I0703 01:42:13.240185 27384 raft_consensus.cc:2730] T 0a2438297555462aa42cb91089c7a618 P 01f6144ac0484e1d9a6da7ff80ba71f0: Snoozing failure detection for 1.973s (starting election) I0703 01:42:13.240281 27384 raft_consensus.cc:479] T 0a2438297555462aa42cb91089c7a618 P 01f6144ac0484e1d9a6da7ff80ba71f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } peers { permanent_uuid: "01f6144ac0484e1d9a6da7ff80ba71f0" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 35740 } } I0703 01:42:13.241171 27384 leader_election.cc:243] T 0a2438297555462aa42cb91089c7a618 P 01f6144ac0484e1d9a6da7ff80ba71f0 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer dfd7ad6293204fc89f9e72436796698b (127.5.240.193:43198) I0703 01:42:13.242355 24920 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "0a2438297555462aa42cb91089c7a618" candidate_uuid: "01f6144ac0484e1d9a6da7ff80ba71f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dfd7ad6293204fc89f9e72436796698b" is_pre_election: true I0703 01:42:13.242597 24920 raft_consensus.cc:2730] T 0a2438297555462aa42cb91089c7a618 P dfd7ad6293204fc89f9e72436796698b: Snoozing failure detection for 1.659s (vote granted) I0703 01:42:13.242669 24920 raft_consensus.cc:2243] T 0a2438297555462aa42cb91089c7a618 P dfd7ad6293204fc89f9e72436796698b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6144ac0484e1d9a6da7ff80ba71f0 in term 1. I0703 01:42:13.243224 26238 leader_election.cc:379] T 0a2438297555462aa42cb91089c7a618 P 01f6144ac0484e1d9a6da7ff80ba71f0 [CANDIDATE]: Term 2 pre-election: Vote granted by peer dfd7ad6293204fc89f9e72436796698b (127.5.240.193:43198) I0703 01:42:13.243296 26238 leader_election.cc:270] T 0a2438297555462aa42cb91089c7a618 P 01f6144ac0484e1d9a6da7ff80ba71f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. I0703 01:42:13.243621 27384 raft_consensus.cc:2730] T 0a2438297555462aa42cb91089c7a618 P 01f6144ac0484e1d9a6da7ff80ba71f0: Snoozing failure detection for 1.742s (election complete) I0703 01:42:13.243719 27384 raft_consensus.cc:2537] T 0a2438297555462aa42cb91089c7a618 P 01f6144ac0484e1d9a6da7ff80ba71f0 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0703 01:42:13.243804 27384 raft_consensus.cc:457] T 0a2438297555462aa42cb91089c7a618 P 01f6144ac0484e1d9a6da7ff80ba71f0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader dfd7ad6293204fc89f9e72436796698b) I0703 01:42:13.243862 27384 raft_consensus.cc:2730] T 0a2438297555462aa42cb91089c7a618 P 01f6144ac0484e1d9a6da7ff80ba71f0: Snoozing failure detection for 1.822s (starting election) I0703 01:42:13.243927 27384 raft_consensus.cc:2782] T 0a2438297555462aa42cb91089c7a618 P 01f6144ac0484e1d9a6da7ff80ba71f0 [term 1 FOLLOWER]: Advancing to term 2 I0703 01:42:13.245736 27384 raft_consensus.cc:479] T 0a2438297555462aa42cb91089c7a618 P 01f6144ac0484e1d9a6da7ff80ba71f0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } peers { permanent_uuid: "01f6144ac0484e1d9a6da7ff80ba71f0" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 35740 } } I0703 01:42:13.246024 27384 leader_election.cc:243] T 0a2438297555462aa42cb91089c7a618 P 01f6144ac0484e1d9a6da7ff80ba71f0 [CANDIDATE]: Term 2 election: Requesting vote from peer dfd7ad6293204fc89f9e72436796698b (127.5.240.193:43198) I0703 01:42:13.246675 24920 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "0a2438297555462aa42cb91089c7a618" candidate_uuid: "01f6144ac0484e1d9a6da7ff80ba71f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dfd7ad6293204fc89f9e72436796698b" I0703 01:42:13.246788 24920 raft_consensus.cc:2782] T 0a2438297555462aa42cb91089c7a618 P dfd7ad6293204fc89f9e72436796698b [term 1 FOLLOWER]: Advancing to term 2 I0703 01:42:13.246873 24920 raft_consensus.cc:2730] T 0a2438297555462aa42cb91089c7a618 P dfd7ad6293204fc89f9e72436796698b: Snoozing failure detection for 1.646s (vote granted) I0703 01:42:13.248620 24920 raft_consensus.cc:2243] T 0a2438297555462aa42cb91089c7a618 P dfd7ad6293204fc89f9e72436796698b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6144ac0484e1d9a6da7ff80ba71f0 in term 2. I0703 01:42:13.249040 26238 leader_election.cc:379] T 0a2438297555462aa42cb91089c7a618 P 01f6144ac0484e1d9a6da7ff80ba71f0 [CANDIDATE]: Term 2 election: Vote granted by peer dfd7ad6293204fc89f9e72436796698b (127.5.240.193:43198) I0703 01:42:13.249107 26238 leader_election.cc:270] T 0a2438297555462aa42cb91089c7a618 P 01f6144ac0484e1d9a6da7ff80ba71f0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0703 01:42:13.249305 27384 raft_consensus.cc:2730] T 0a2438297555462aa42cb91089c7a618 P 01f6144ac0484e1d9a6da7ff80ba71f0: Snoozing failure detection for 1.865s (election complete) I0703 01:42:13.249660 27384 raft_consensus.cc:2537] T 0a2438297555462aa42cb91089c7a618 P 01f6144ac0484e1d9a6da7ff80ba71f0 [term 2 FOLLOWER]: Leader election won for term 2 I0703 01:42:13.250414 27384 raft_consensus.cc:593] T 0a2438297555462aa42cb91089c7a618 P 01f6144ac0484e1d9a6da7ff80ba71f0 [term 2 LEADER]: Becoming Leader. State: Replica: 01f6144ac0484e1d9a6da7ff80ba71f0, State: Running, Role: LEADER I0703 01:42:13.250809 27384 consensus_queue.cc:204] T 0a2438297555462aa42cb91089c7a618 P 01f6144ac0484e1d9a6da7ff80ba71f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } peers { permanent_uuid: "01f6144ac0484e1d9a6da7ff80ba71f0" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 35740 } } I0703 01:42:13.253420 24797 catalog_manager.cc:3798] T 0a2438297555462aa42cb91089c7a618 P 01f6144ac0484e1d9a6da7ff80ba71f0 reported cstate change: term changed from 1 to 2, leader changed from dfd7ad6293204fc89f9e72436796698b (127.5.240.193) to 01f6144ac0484e1d9a6da7ff80ba71f0 (127.5.240.198). New cstate: current_term: 2 leader_uuid: "01f6144ac0484e1d9a6da7ff80ba71f0" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } peers { permanent_uuid: "01f6144ac0484e1d9a6da7ff80ba71f0" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 35740 } } } I0703 01:42:13.334831 27417 raft_consensus.cc:457] T 53242d0289d644faa0149e0a54c0984c P cff51d1a4baa4cd0a1625e1d7db94f2c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dfd7ad6293204fc89f9e72436796698b) I0703 01:42:13.334988 27417 raft_consensus.cc:2730] T 53242d0289d644faa0149e0a54c0984c P cff51d1a4baa4cd0a1625e1d7db94f2c: Snoozing failure detection for 1.706s (starting election) I0703 01:42:13.335069 27417 raft_consensus.cc:479] T 53242d0289d644faa0149e0a54c0984c P cff51d1a4baa4cd0a1625e1d7db94f2c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } peers { permanent_uuid: "cff51d1a4baa4cd0a1625e1d7db94f2c" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 38199 } } I0703 01:42:13.335547 27417 leader_election.cc:243] T 53242d0289d644faa0149e0a54c0984c P cff51d1a4baa4cd0a1625e1d7db94f2c [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer dfd7ad6293204fc89f9e72436796698b (127.5.240.193:43198) I0703 01:42:13.336285 26061 consensus_peers.cc:455] T 72ebab6b7393457db959684913708522 P ca6ae9994a78460ca6c4ae4ec3f4e9b5 -> Peer 8ec1760d41f842f398d90e47ab52973b (127.5.240.194:46721): Closing peer: 8ec1760d41f842f398d90e47ab52973b I0703 01:42:13.336494 24920 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "53242d0289d644faa0149e0a54c0984c" candidate_uuid: "cff51d1a4baa4cd0a1625e1d7db94f2c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dfd7ad6293204fc89f9e72436796698b" is_pre_election: true I0703 01:42:13.336635 24920 raft_consensus.cc:2730] T 53242d0289d644faa0149e0a54c0984c P dfd7ad6293204fc89f9e72436796698b: Snoozing failure detection for 1.699s (vote granted) I0703 01:42:13.336570 26061 consensus_queue.cc:204] T 72ebab6b7393457db959684913708522 P ca6ae9994a78460ca6c4ae4ec3f4e9b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ca6ae9994a78460ca6c4ae4ec3f4e9b5" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 41121 } } I0703 01:42:13.336688 24920 raft_consensus.cc:2243] T 53242d0289d644faa0149e0a54c0984c P dfd7ad6293204fc89f9e72436796698b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cff51d1a4baa4cd0a1625e1d7db94f2c in term 1. I0703 01:42:13.337198 25862 leader_election.cc:379] T 53242d0289d644faa0149e0a54c0984c P cff51d1a4baa4cd0a1625e1d7db94f2c [CANDIDATE]: Term 2 pre-election: Vote granted by peer dfd7ad6293204fc89f9e72436796698b (127.5.240.193:43198) I0703 01:42:13.337265 25862 leader_election.cc:270] T 53242d0289d644faa0149e0a54c0984c P cff51d1a4baa4cd0a1625e1d7db94f2c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. I0703 01:42:13.337203 27291 raft_consensus.cc:2684] T 72ebab6b7393457db959684913708522 P ca6ae9994a78460ca6c4ae4ec3f4e9b5 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 8ec1760d41f842f398d90e47ab52973b (127.5.240.194) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ca6ae9994a78460ca6c4ae4ec3f4e9b5" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 41121 } } } I0703 01:42:13.337471 27417 raft_consensus.cc:2730] T 53242d0289d644faa0149e0a54c0984c P cff51d1a4baa4cd0a1625e1d7db94f2c: Snoozing failure detection for 2.095s (election complete) I0703 01:42:13.337568 27417 raft_consensus.cc:2537] T 53242d0289d644faa0149e0a54c0984c P cff51d1a4baa4cd0a1625e1d7db94f2c [term 1 FOLLOWER]: Leader pre-election won for term 2 I0703 01:42:13.337615 27417 raft_consensus.cc:457] T 53242d0289d644faa0149e0a54c0984c P cff51d1a4baa4cd0a1625e1d7db94f2c [term 1 FOLLOWER]: Starting leader election (detected failure of leader dfd7ad6293204fc89f9e72436796698b) I0703 01:42:13.337664 27417 raft_consensus.cc:2730] T 53242d0289d644faa0149e0a54c0984c P cff51d1a4baa4cd0a1625e1d7db94f2c: Snoozing failure detection for 1.761s (starting election) I0703 01:42:13.337710 27417 raft_consensus.cc:2782] T 53242d0289d644faa0149e0a54c0984c P cff51d1a4baa4cd0a1625e1d7db94f2c [term 1 FOLLOWER]: Advancing to term 2 I0703 01:42:13.339025 27417 raft_consensus.cc:479] T 53242d0289d644faa0149e0a54c0984c P cff51d1a4baa4cd0a1625e1d7db94f2c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } peers { permanent_uuid: "cff51d1a4baa4cd0a1625e1d7db94f2c" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 38199 } } I0703 01:42:13.339268 27417 leader_election.cc:243] T 53242d0289d644faa0149e0a54c0984c P cff51d1a4baa4cd0a1625e1d7db94f2c [CANDIDATE]: Term 2 election: Requesting vote from peer dfd7ad6293204fc89f9e72436796698b (127.5.240.193:43198) I0703 01:42:13.339912 24920 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "53242d0289d644faa0149e0a54c0984c" candidate_uuid: "cff51d1a4baa4cd0a1625e1d7db94f2c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dfd7ad6293204fc89f9e72436796698b" I0703 01:42:13.340040 24920 raft_consensus.cc:2782] T 53242d0289d644faa0149e0a54c0984c P dfd7ad6293204fc89f9e72436796698b [term 1 FOLLOWER]: Advancing to term 2 I0703 01:42:13.340106 24920 raft_consensus.cc:2730] T 53242d0289d644faa0149e0a54c0984c P dfd7ad6293204fc89f9e72436796698b: Snoozing failure detection for 1.967s (vote granted) I0703 01:42:13.340517 24797 catalog_manager.cc:3798] T 72ebab6b7393457db959684913708522 P ca6ae9994a78460ca6c4ae4ec3f4e9b5 reported cstate change: config changed from index 4 to 6, VOTER 8ec1760d41f842f398d90e47ab52973b (127.5.240.194) evicted. New cstate: current_term: 2 leader_uuid: "ca6ae9994a78460ca6c4ae4ec3f4e9b5" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ca6ae9994a78460ca6c4ae4ec3f4e9b5" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 41121 } } } I0703 01:42:13.341622 24920 raft_consensus.cc:2243] T 53242d0289d644faa0149e0a54c0984c P dfd7ad6293204fc89f9e72436796698b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cff51d1a4baa4cd0a1625e1d7db94f2c in term 2. I0703 01:42:13.342043 25862 leader_election.cc:379] T 53242d0289d644faa0149e0a54c0984c P cff51d1a4baa4cd0a1625e1d7db94f2c [CANDIDATE]: Term 2 election: Vote granted by peer dfd7ad6293204fc89f9e72436796698b (127.5.240.193:43198) I0703 01:42:13.342104 25862 leader_election.cc:270] T 53242d0289d644faa0149e0a54c0984c P cff51d1a4baa4cd0a1625e1d7db94f2c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0703 01:42:13.342279 27417 raft_consensus.cc:2730] T 53242d0289d644faa0149e0a54c0984c P cff51d1a4baa4cd0a1625e1d7db94f2c: Snoozing failure detection for 2.129s (election complete) I0703 01:42:13.342345 27417 raft_consensus.cc:2537] T 53242d0289d644faa0149e0a54c0984c P cff51d1a4baa4cd0a1625e1d7db94f2c [term 2 FOLLOWER]: Leader election won for term 2 I0703 01:42:13.343488 24797 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 72ebab6b7393457db959684913708522 on 8ec1760d41f842f398d90e47ab52973b (127.5.240.194:46721) (TS 8ec1760d41f842f398d90e47ab52973b not found in new config with opid_index 6) I0703 01:42:13.343879 27417 raft_consensus.cc:593] T 53242d0289d644faa0149e0a54c0984c P cff51d1a4baa4cd0a1625e1d7db94f2c [term 2 LEADER]: Becoming Leader. State: Replica: cff51d1a4baa4cd0a1625e1d7db94f2c, State: Running, Role: LEADER I0703 01:42:13.344046 27417 consensus_queue.cc:204] T 53242d0289d644faa0149e0a54c0984c P cff51d1a4baa4cd0a1625e1d7db94f2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } peers { permanent_uuid: "cff51d1a4baa4cd0a1625e1d7db94f2c" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 38199 } } I0703 01:42:13.344125 25022 tablet_service.cc:800] Processing DeleteTablet for tablet 72ebab6b7393457db959684913708522 with delete_type TABLET_DATA_TOMBSTONED (TS 8ec1760d41f842f398d90e47ab52973b not found in new config with opid_index 6) from {username='jenkins'} at 127.5.240.194:45697 I0703 01:42:13.345163 27419 tablet_replica.cc:261] T 72ebab6b7393457db959684913708522 P 8ec1760d41f842f398d90e47ab52973b: stopping tablet replica I0703 01:42:13.345325 27419 maintenance_manager.cc:237] P 8ec1760d41f842f398d90e47ab52973b: Unregistered op CompactRowSetsOp(72ebab6b7393457db959684913708522) I0703 01:42:13.345413 27419 maintenance_manager.cc:237] P 8ec1760d41f842f398d90e47ab52973b: Unregistered op MinorDeltaCompactionOp(72ebab6b7393457db959684913708522) I0703 01:42:13.345468 27419 maintenance_manager.cc:237] P 8ec1760d41f842f398d90e47ab52973b: Unregistered op MajorDeltaCompactionOp(72ebab6b7393457db959684913708522) I0703 01:42:13.345535 27419 maintenance_manager.cc:237] P 8ec1760d41f842f398d90e47ab52973b: Unregistered op UndoDeltaBlockGCOp(72ebab6b7393457db959684913708522) I0703 01:42:13.345655 27419 maintenance_manager.cc:237] P 8ec1760d41f842f398d90e47ab52973b: Unregistered op FlushMRSOp(72ebab6b7393457db959684913708522) I0703 01:42:13.345706 27419 maintenance_manager.cc:237] P 8ec1760d41f842f398d90e47ab52973b: Unregistered op FlushDeltaMemStoresOp(72ebab6b7393457db959684913708522) I0703 01:42:13.345748 27419 maintenance_manager.cc:237] P 8ec1760d41f842f398d90e47ab52973b: Unregistered op LogGCOp(72ebab6b7393457db959684913708522) I0703 01:42:13.345816 27419 raft_consensus.cc:2040] T 72ebab6b7393457db959684913708522 P 8ec1760d41f842f398d90e47ab52973b [term 2 FOLLOWER]: Raft consensus shutting down. I0703 01:42:13.346011 27419 raft_consensus.cc:2067] T 72ebab6b7393457db959684913708522 P 8ec1760d41f842f398d90e47ab52973b [term 2 FOLLOWER]: Raft consensus is shut down! I0703 01:42:13.347765 24797 catalog_manager.cc:3798] T 53242d0289d644faa0149e0a54c0984c P cff51d1a4baa4cd0a1625e1d7db94f2c reported cstate change: term changed from 1 to 2, leader changed from dfd7ad6293204fc89f9e72436796698b (127.5.240.193) to cff51d1a4baa4cd0a1625e1d7db94f2c (127.5.240.195). New cstate: current_term: 2 leader_uuid: "cff51d1a4baa4cd0a1625e1d7db94f2c" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } peers { permanent_uuid: "cff51d1a4baa4cd0a1625e1d7db94f2c" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 38199 } } } I0703 01:42:13.348094 27419 ts_tablet_manager.cc:1369] T 72ebab6b7393457db959684913708522 P 8ec1760d41f842f398d90e47ab52973b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:42:13.351455 27419 ts_tablet_manager.cc:1382] T 72ebab6b7393457db959684913708522 P 8ec1760d41f842f398d90e47ab52973b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5 I0703 01:42:13.351544 27419 log.cc:986] T 72ebab6b7393457db959684913708522 P 8ec1760d41f842f398d90e47ab52973b: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal/wals/72ebab6b7393457db959684913708522 I0703 01:42:13.352202 24783 catalog_manager.cc:3103] TS 8ec1760d41f842f398d90e47ab52973b (127.5.240.194:46721): tablet 72ebab6b7393457db959684913708522 (table rebalance_test_table_3 [id=72ffb8cd105e4058a3b2caf04d27ee7f]) successfully deleted I0703 01:42:13.453878 27452 raft_consensus.cc:457] T dae8746553524b6a9eb2e3f177f658f1 P a344baa95900451dbaf349b68e662b27 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8ec1760d41f842f398d90e47ab52973b) I0703 01:42:13.454025 27452 raft_consensus.cc:2730] T dae8746553524b6a9eb2e3f177f658f1 P a344baa95900451dbaf349b68e662b27: Snoozing failure detection for 2.042s (starting election) I0703 01:42:13.454123 27452 raft_consensus.cc:479] T dae8746553524b6a9eb2e3f177f658f1 P a344baa95900451dbaf349b68e662b27 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } peers { permanent_uuid: "a344baa95900451dbaf349b68e662b27" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 43710 } } I0703 01:42:13.454764 27452 leader_election.cc:243] T dae8746553524b6a9eb2e3f177f658f1 P a344baa95900451dbaf349b68e662b27 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 8ec1760d41f842f398d90e47ab52973b (127.5.240.194:46721) I0703 01:42:13.455816 25043 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "dae8746553524b6a9eb2e3f177f658f1" candidate_uuid: "a344baa95900451dbaf349b68e662b27" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8ec1760d41f842f398d90e47ab52973b" is_pre_election: true I0703 01:42:13.455955 25043 raft_consensus.cc:2730] T dae8746553524b6a9eb2e3f177f658f1 P 8ec1760d41f842f398d90e47ab52973b: Snoozing failure detection for 1.865s (vote granted) I0703 01:42:13.456033 25043 raft_consensus.cc:2243] T dae8746553524b6a9eb2e3f177f658f1 P 8ec1760d41f842f398d90e47ab52973b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a344baa95900451dbaf349b68e662b27 in term 1. I0703 01:42:13.456470 26360 leader_election.cc:379] T dae8746553524b6a9eb2e3f177f658f1 P a344baa95900451dbaf349b68e662b27 [CANDIDATE]: Term 2 pre-election: Vote granted by peer 8ec1760d41f842f398d90e47ab52973b (127.5.240.194:46721) I0703 01:42:13.456554 26360 leader_election.cc:270] T dae8746553524b6a9eb2e3f177f658f1 P a344baa95900451dbaf349b68e662b27 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. I0703 01:42:13.456830 27452 raft_consensus.cc:2730] T dae8746553524b6a9eb2e3f177f658f1 P a344baa95900451dbaf349b68e662b27: Snoozing failure detection for 1.569s (election complete) I0703 01:42:13.456943 27452 raft_consensus.cc:2537] T dae8746553524b6a9eb2e3f177f658f1 P a344baa95900451dbaf349b68e662b27 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0703 01:42:13.457010 27452 raft_consensus.cc:457] T dae8746553524b6a9eb2e3f177f658f1 P a344baa95900451dbaf349b68e662b27 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8ec1760d41f842f398d90e47ab52973b) I0703 01:42:13.457058 27452 raft_consensus.cc:2730] T dae8746553524b6a9eb2e3f177f658f1 P a344baa95900451dbaf349b68e662b27: Snoozing failure detection for 1.811s (starting election) I0703 01:42:13.457119 27452 raft_consensus.cc:2782] T dae8746553524b6a9eb2e3f177f658f1 P a344baa95900451dbaf349b68e662b27 [term 1 FOLLOWER]: Advancing to term 2 I0703 01:42:13.458755 27452 raft_consensus.cc:479] T dae8746553524b6a9eb2e3f177f658f1 P a344baa95900451dbaf349b68e662b27 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } peers { permanent_uuid: "a344baa95900451dbaf349b68e662b27" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 43710 } } I0703 01:42:13.459035 27452 leader_election.cc:243] T dae8746553524b6a9eb2e3f177f658f1 P a344baa95900451dbaf349b68e662b27 [CANDIDATE]: Term 2 election: Requesting vote from peer 8ec1760d41f842f398d90e47ab52973b (127.5.240.194:46721) I0703 01:42:13.459769 25043 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "dae8746553524b6a9eb2e3f177f658f1" candidate_uuid: "a344baa95900451dbaf349b68e662b27" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8ec1760d41f842f398d90e47ab52973b" I0703 01:42:13.459890 25043 raft_consensus.cc:2782] T dae8746553524b6a9eb2e3f177f658f1 P 8ec1760d41f842f398d90e47ab52973b [term 1 FOLLOWER]: Advancing to term 2 I0703 01:42:13.459964 25043 raft_consensus.cc:2730] T dae8746553524b6a9eb2e3f177f658f1 P 8ec1760d41f842f398d90e47ab52973b: Snoozing failure detection for 2.191s (vote granted) I0703 01:42:13.461738 25043 raft_consensus.cc:2243] T dae8746553524b6a9eb2e3f177f658f1 P 8ec1760d41f842f398d90e47ab52973b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a344baa95900451dbaf349b68e662b27 in term 2. I0703 01:42:13.462231 26360 leader_election.cc:379] T dae8746553524b6a9eb2e3f177f658f1 P a344baa95900451dbaf349b68e662b27 [CANDIDATE]: Term 2 election: Vote granted by peer 8ec1760d41f842f398d90e47ab52973b (127.5.240.194:46721) I0703 01:42:13.462294 26360 leader_election.cc:270] T dae8746553524b6a9eb2e3f177f658f1 P a344baa95900451dbaf349b68e662b27 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0703 01:42:13.462467 27452 raft_consensus.cc:2730] T dae8746553524b6a9eb2e3f177f658f1 P a344baa95900451dbaf349b68e662b27: Snoozing failure detection for 1.520s (election complete) I0703 01:42:13.462535 27452 raft_consensus.cc:2537] T dae8746553524b6a9eb2e3f177f658f1 P a344baa95900451dbaf349b68e662b27 [term 2 FOLLOWER]: Leader election won for term 2 I0703 01:42:13.462862 27452 raft_consensus.cc:593] T dae8746553524b6a9eb2e3f177f658f1 P a344baa95900451dbaf349b68e662b27 [term 2 LEADER]: Becoming Leader. State: Replica: a344baa95900451dbaf349b68e662b27, State: Running, Role: LEADER I0703 01:42:13.463244 27452 consensus_queue.cc:204] T dae8746553524b6a9eb2e3f177f658f1 P a344baa95900451dbaf349b68e662b27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } peers { permanent_uuid: "a344baa95900451dbaf349b68e662b27" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 43710 } } I0703 01:42:13.465514 24797 catalog_manager.cc:3798] T dae8746553524b6a9eb2e3f177f658f1 P a344baa95900451dbaf349b68e662b27 reported cstate change: term changed from 1 to 2, leader changed from 8ec1760d41f842f398d90e47ab52973b (127.5.240.194) to a344baa95900451dbaf349b68e662b27 (127.5.240.199). New cstate: current_term: 2 leader_uuid: "a344baa95900451dbaf349b68e662b27" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } peers { permanent_uuid: "a344baa95900451dbaf349b68e662b27" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 43710 } } } I0703 01:42:13.570303 27291 raft_consensus.cc:457] T ff7fa826668f419fa3c5225e5d810b9c P ca6ae9994a78460ca6c4ae4ec3f4e9b5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dfd7ad6293204fc89f9e72436796698b) I0703 01:42:13.570385 27291 raft_consensus.cc:2730] T ff7fa826668f419fa3c5225e5d810b9c P ca6ae9994a78460ca6c4ae4ec3f4e9b5: Snoozing failure detection for 1.652s (starting election) I0703 01:42:13.570461 27291 raft_consensus.cc:479] T ff7fa826668f419fa3c5225e5d810b9c P ca6ae9994a78460ca6c4ae4ec3f4e9b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } peers { permanent_uuid: "ca6ae9994a78460ca6c4ae4ec3f4e9b5" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 41121 } } I0703 01:42:13.570930 27291 leader_election.cc:243] T ff7fa826668f419fa3c5225e5d810b9c P ca6ae9994a78460ca6c4ae4ec3f4e9b5 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer dfd7ad6293204fc89f9e72436796698b (127.5.240.193:43198) I0703 01:42:13.571619 24920 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "ff7fa826668f419fa3c5225e5d810b9c" candidate_uuid: "ca6ae9994a78460ca6c4ae4ec3f4e9b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dfd7ad6293204fc89f9e72436796698b" is_pre_election: true I0703 01:42:13.571775 24920 raft_consensus.cc:2730] T ff7fa826668f419fa3c5225e5d810b9c P dfd7ad6293204fc89f9e72436796698b: Snoozing failure detection for 1.870s (vote granted) I0703 01:42:13.571841 24920 raft_consensus.cc:2243] T ff7fa826668f419fa3c5225e5d810b9c P dfd7ad6293204fc89f9e72436796698b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca6ae9994a78460ca6c4ae4ec3f4e9b5 in term 1. I0703 01:42:13.572238 25994 leader_election.cc:379] T ff7fa826668f419fa3c5225e5d810b9c P ca6ae9994a78460ca6c4ae4ec3f4e9b5 [CANDIDATE]: Term 2 pre-election: Vote granted by peer dfd7ad6293204fc89f9e72436796698b (127.5.240.193:43198) I0703 01:42:13.572299 25994 leader_election.cc:270] T ff7fa826668f419fa3c5225e5d810b9c P ca6ae9994a78460ca6c4ae4ec3f4e9b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. I0703 01:42:13.572480 27291 raft_consensus.cc:2730] T ff7fa826668f419fa3c5225e5d810b9c P ca6ae9994a78460ca6c4ae4ec3f4e9b5: Snoozing failure detection for 1.540s (election complete) I0703 01:42:13.572554 27291 raft_consensus.cc:2537] T ff7fa826668f419fa3c5225e5d810b9c P ca6ae9994a78460ca6c4ae4ec3f4e9b5 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0703 01:42:13.572620 27291 raft_consensus.cc:457] T ff7fa826668f419fa3c5225e5d810b9c P ca6ae9994a78460ca6c4ae4ec3f4e9b5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader dfd7ad6293204fc89f9e72436796698b) I0703 01:42:13.572674 27291 raft_consensus.cc:2730] T ff7fa826668f419fa3c5225e5d810b9c P ca6ae9994a78460ca6c4ae4ec3f4e9b5: Snoozing failure detection for 2.008s (starting election) I0703 01:42:13.572728 27291 raft_consensus.cc:2782] T ff7fa826668f419fa3c5225e5d810b9c P ca6ae9994a78460ca6c4ae4ec3f4e9b5 [term 1 FOLLOWER]: Advancing to term 2 I0703 01:42:13.574237 27291 raft_consensus.cc:479] T ff7fa826668f419fa3c5225e5d810b9c P ca6ae9994a78460ca6c4ae4ec3f4e9b5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } peers { permanent_uuid: "ca6ae9994a78460ca6c4ae4ec3f4e9b5" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 41121 } } I0703 01:42:13.574518 27291 leader_election.cc:243] T ff7fa826668f419fa3c5225e5d810b9c P ca6ae9994a78460ca6c4ae4ec3f4e9b5 [CANDIDATE]: Term 2 election: Requesting vote from peer dfd7ad6293204fc89f9e72436796698b (127.5.240.193:43198) I0703 01:42:13.575212 24920 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "ff7fa826668f419fa3c5225e5d810b9c" candidate_uuid: "ca6ae9994a78460ca6c4ae4ec3f4e9b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dfd7ad6293204fc89f9e72436796698b" I0703 01:42:13.575342 24920 raft_consensus.cc:2782] T ff7fa826668f419fa3c5225e5d810b9c P dfd7ad6293204fc89f9e72436796698b [term 1 FOLLOWER]: Advancing to term 2 I0703 01:42:13.575415 24920 raft_consensus.cc:2730] T ff7fa826668f419fa3c5225e5d810b9c P dfd7ad6293204fc89f9e72436796698b: Snoozing failure detection for 1.551s (vote granted) I0703 01:42:13.576778 24920 raft_consensus.cc:2243] T ff7fa826668f419fa3c5225e5d810b9c P dfd7ad6293204fc89f9e72436796698b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca6ae9994a78460ca6c4ae4ec3f4e9b5 in term 2. I0703 01:42:13.577181 25994 leader_election.cc:379] T ff7fa826668f419fa3c5225e5d810b9c P ca6ae9994a78460ca6c4ae4ec3f4e9b5 [CANDIDATE]: Term 2 election: Vote granted by peer dfd7ad6293204fc89f9e72436796698b (127.5.240.193:43198) I0703 01:42:13.577239 25994 leader_election.cc:270] T ff7fa826668f419fa3c5225e5d810b9c P ca6ae9994a78460ca6c4ae4ec3f4e9b5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0703 01:42:13.577412 27291 raft_consensus.cc:2730] T ff7fa826668f419fa3c5225e5d810b9c P ca6ae9994a78460ca6c4ae4ec3f4e9b5: Snoozing failure detection for 1.935s (election complete) I0703 01:42:13.577476 27291 raft_consensus.cc:2537] T ff7fa826668f419fa3c5225e5d810b9c P ca6ae9994a78460ca6c4ae4ec3f4e9b5 [term 2 FOLLOWER]: Leader election won for term 2 I0703 01:42:13.577636 27291 raft_consensus.cc:593] T ff7fa826668f419fa3c5225e5d810b9c P ca6ae9994a78460ca6c4ae4ec3f4e9b5 [term 2 LEADER]: Becoming Leader. State: Replica: ca6ae9994a78460ca6c4ae4ec3f4e9b5, State: Running, Role: LEADER I0703 01:42:13.577807 27291 consensus_queue.cc:204] T ff7fa826668f419fa3c5225e5d810b9c P ca6ae9994a78460ca6c4ae4ec3f4e9b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } peers { permanent_uuid: "ca6ae9994a78460ca6c4ae4ec3f4e9b5" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 41121 } } I0703 01:42:13.579619 24793 catalog_manager.cc:3798] T ff7fa826668f419fa3c5225e5d810b9c P ca6ae9994a78460ca6c4ae4ec3f4e9b5 reported cstate change: term changed from 1 to 2, leader changed from dfd7ad6293204fc89f9e72436796698b (127.5.240.193) to ca6ae9994a78460ca6c4ae4ec3f4e9b5 (127.5.240.196). New cstate: current_term: 2 leader_uuid: "ca6ae9994a78460ca6c4ae4ec3f4e9b5" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } peers { permanent_uuid: "ca6ae9994a78460ca6c4ae4ec3f4e9b5" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 41121 } } } I0703 01:42:13.620996 25043 raft_consensus.cc:1081] T 9a818723ff79493aa17e7434040e8975 P 8ec1760d41f842f398d90e47ab52973b [term 2 FOLLOWER]: Refusing update from remote peer 2f7e2b9db52f44ba81b7393403f9ed37: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch) I0703 01:42:13.622210 27513 consensus_queue.cc:962] T 9a818723ff79493aa17e7434040e8975 P 2f7e2b9db52f44ba81b7393403f9ed37 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I0703 01:42:13.694365 24920 raft_consensus.cc:1081] T 0a2438297555462aa42cb91089c7a618 P dfd7ad6293204fc89f9e72436796698b [term 2 FOLLOWER]: Refusing update from remote peer 01f6144ac0484e1d9a6da7ff80ba71f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch) I0703 01:42:13.695623 27384 consensus_queue.cc:962] T 0a2438297555462aa42cb91089c7a618 P 01f6144ac0484e1d9a6da7ff80ba71f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I0703 01:42:13.794895 27452 raft_consensus.cc:457] T fd7030608be84fa9b61c3ae90e04e27e P a344baa95900451dbaf349b68e662b27 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8ec1760d41f842f398d90e47ab52973b) I0703 01:42:13.794987 27452 raft_consensus.cc:2730] T fd7030608be84fa9b61c3ae90e04e27e P a344baa95900451dbaf349b68e662b27: Snoozing failure detection for 1.998s (starting election) I0703 01:42:13.795060 27452 raft_consensus.cc:479] T fd7030608be84fa9b61c3ae90e04e27e P a344baa95900451dbaf349b68e662b27 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } peers { permanent_uuid: "a344baa95900451dbaf349b68e662b27" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 43710 } } I0703 01:42:13.795392 27452 leader_election.cc:243] T fd7030608be84fa9b61c3ae90e04e27e P a344baa95900451dbaf349b68e662b27 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 8ec1760d41f842f398d90e47ab52973b (127.5.240.194:46721) I0703 01:42:13.796334 25042 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "fd7030608be84fa9b61c3ae90e04e27e" candidate_uuid: "a344baa95900451dbaf349b68e662b27" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8ec1760d41f842f398d90e47ab52973b" is_pre_election: true I0703 01:42:13.796543 25042 raft_consensus.cc:2730] T fd7030608be84fa9b61c3ae90e04e27e P 8ec1760d41f842f398d90e47ab52973b: Snoozing failure detection for 1.539s (vote granted) I0703 01:42:13.796640 25042 raft_consensus.cc:2243] T fd7030608be84fa9b61c3ae90e04e27e P 8ec1760d41f842f398d90e47ab52973b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a344baa95900451dbaf349b68e662b27 in term 1. I0703 01:42:13.797564 26360 leader_election.cc:379] T fd7030608be84fa9b61c3ae90e04e27e P a344baa95900451dbaf349b68e662b27 [CANDIDATE]: Term 2 pre-election: Vote granted by peer 8ec1760d41f842f398d90e47ab52973b (127.5.240.194:46721) I0703 01:42:13.797644 26360 leader_election.cc:270] T fd7030608be84fa9b61c3ae90e04e27e P a344baa95900451dbaf349b68e662b27 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. I0703 01:42:13.797832 27452 raft_consensus.cc:2730] T fd7030608be84fa9b61c3ae90e04e27e P a344baa95900451dbaf349b68e662b27: Snoozing failure detection for 1.688s (election complete) I0703 01:42:13.797907 27452 raft_consensus.cc:2537] T fd7030608be84fa9b61c3ae90e04e27e P a344baa95900451dbaf349b68e662b27 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0703 01:42:13.797960 27452 raft_consensus.cc:457] T fd7030608be84fa9b61c3ae90e04e27e P a344baa95900451dbaf349b68e662b27 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8ec1760d41f842f398d90e47ab52973b) I0703 01:42:13.798015 27452 raft_consensus.cc:2730] T fd7030608be84fa9b61c3ae90e04e27e P a344baa95900451dbaf349b68e662b27: Snoozing failure detection for 1.699s (starting election) I0703 01:42:13.798075 27452 raft_consensus.cc:2782] T fd7030608be84fa9b61c3ae90e04e27e P a344baa95900451dbaf349b68e662b27 [term 1 FOLLOWER]: Advancing to term 2 I0703 01:42:13.800130 27452 raft_consensus.cc:479] T fd7030608be84fa9b61c3ae90e04e27e P a344baa95900451dbaf349b68e662b27 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } peers { permanent_uuid: "a344baa95900451dbaf349b68e662b27" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 43710 } } I0703 01:42:13.800388 27452 leader_election.cc:243] T fd7030608be84fa9b61c3ae90e04e27e P a344baa95900451dbaf349b68e662b27 [CANDIDATE]: Term 2 election: Requesting vote from peer 8ec1760d41f842f398d90e47ab52973b (127.5.240.194:46721) I0703 01:42:13.801101 25042 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "fd7030608be84fa9b61c3ae90e04e27e" candidate_uuid: "a344baa95900451dbaf349b68e662b27" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8ec1760d41f842f398d90e47ab52973b" I0703 01:42:13.801209 25042 raft_consensus.cc:2782] T fd7030608be84fa9b61c3ae90e04e27e P 8ec1760d41f842f398d90e47ab52973b [term 1 FOLLOWER]: Advancing to term 2 I0703 01:42:13.801273 25042 raft_consensus.cc:2730] T fd7030608be84fa9b61c3ae90e04e27e P 8ec1760d41f842f398d90e47ab52973b: Snoozing failure detection for 1.905s (vote granted) I0703 01:42:13.802641 25042 raft_consensus.cc:2243] T fd7030608be84fa9b61c3ae90e04e27e P 8ec1760d41f842f398d90e47ab52973b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a344baa95900451dbaf349b68e662b27 in term 2. I0703 01:42:13.803048 26360 leader_election.cc:379] T fd7030608be84fa9b61c3ae90e04e27e P a344baa95900451dbaf349b68e662b27 [CANDIDATE]: Term 2 election: Vote granted by peer 8ec1760d41f842f398d90e47ab52973b (127.5.240.194:46721) I0703 01:42:13.803103 26360 leader_election.cc:270] T fd7030608be84fa9b61c3ae90e04e27e P a344baa95900451dbaf349b68e662b27 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0703 01:42:13.803290 27452 raft_consensus.cc:2730] T fd7030608be84fa9b61c3ae90e04e27e P a344baa95900451dbaf349b68e662b27: Snoozing failure detection for 1.796s (election complete) I0703 01:42:13.803354 27452 raft_consensus.cc:2537] T fd7030608be84fa9b61c3ae90e04e27e P a344baa95900451dbaf349b68e662b27 [term 2 FOLLOWER]: Leader election won for term 2 I0703 01:42:13.803447 27452 raft_consensus.cc:593] T fd7030608be84fa9b61c3ae90e04e27e P a344baa95900451dbaf349b68e662b27 [term 2 LEADER]: Becoming Leader. State: Replica: a344baa95900451dbaf349b68e662b27, State: Running, Role: LEADER I0703 01:42:13.803654 27452 consensus_queue.cc:204] T fd7030608be84fa9b61c3ae90e04e27e P a344baa95900451dbaf349b68e662b27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } peers { permanent_uuid: "a344baa95900451dbaf349b68e662b27" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 43710 } } I0703 01:42:13.805946 24797 catalog_manager.cc:3798] T fd7030608be84fa9b61c3ae90e04e27e P a344baa95900451dbaf349b68e662b27 reported cstate change: term changed from 1 to 2, leader changed from 8ec1760d41f842f398d90e47ab52973b (127.5.240.194) to a344baa95900451dbaf349b68e662b27 (127.5.240.199). New cstate: current_term: 2 leader_uuid: "a344baa95900451dbaf349b68e662b27" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 } } peers { permanent_uuid: "a344baa95900451dbaf349b68e662b27" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 43710 } } } I0703 01:42:13.869077 27390 raft_consensus.cc:457] T 6359db2dd602463e841abc2a907c49a3 P 01f6144ac0484e1d9a6da7ff80ba71f0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dfd7ad6293204fc89f9e72436796698b) I0703 01:42:13.869186 27390 raft_consensus.cc:2730] T 6359db2dd602463e841abc2a907c49a3 P 01f6144ac0484e1d9a6da7ff80ba71f0: Snoozing failure detection for 2.093s (starting election) I0703 01:42:13.869283 27390 raft_consensus.cc:479] T 6359db2dd602463e841abc2a907c49a3 P 01f6144ac0484e1d9a6da7ff80ba71f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } peers { permanent_uuid: "01f6144ac0484e1d9a6da7ff80ba71f0" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 35740 } } I0703 01:42:13.869666 27390 leader_election.cc:243] T 6359db2dd602463e841abc2a907c49a3 P 01f6144ac0484e1d9a6da7ff80ba71f0 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer dfd7ad6293204fc89f9e72436796698b (127.5.240.193:43198) I0703 01:42:13.870347 24919 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "6359db2dd602463e841abc2a907c49a3" candidate_uuid: "01f6144ac0484e1d9a6da7ff80ba71f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dfd7ad6293204fc89f9e72436796698b" is_pre_election: true I0703 01:42:13.870540 24919 raft_consensus.cc:2730] T 6359db2dd602463e841abc2a907c49a3 P dfd7ad6293204fc89f9e72436796698b: Snoozing failure detection for 2.142s (vote granted) I0703 01:42:13.870615 24919 raft_consensus.cc:2243] T 6359db2dd602463e841abc2a907c49a3 P dfd7ad6293204fc89f9e72436796698b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01f6144ac0484e1d9a6da7ff80ba71f0 in term 1. I0703 01:42:13.871130 26238 leader_election.cc:379] T 6359db2dd602463e841abc2a907c49a3 P 01f6144ac0484e1d9a6da7ff80ba71f0 [CANDIDATE]: Term 2 pre-election: Vote granted by peer dfd7ad6293204fc89f9e72436796698b (127.5.240.193:43198) I0703 01:42:13.871198 26238 leader_election.cc:270] T 6359db2dd602463e841abc2a907c49a3 P 01f6144ac0484e1d9a6da7ff80ba71f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. I0703 01:42:13.871451 27390 raft_consensus.cc:2730] T 6359db2dd602463e841abc2a907c49a3 P 01f6144ac0484e1d9a6da7ff80ba71f0: Snoozing failure detection for 1.543s (election complete) I0703 01:42:13.871515 27390 raft_consensus.cc:2537] T 6359db2dd602463e841abc2a907c49a3 P 01f6144ac0484e1d9a6da7ff80ba71f0 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0703 01:42:13.871573 27390 raft_consensus.cc:457] T 6359db2dd602463e841abc2a907c49a3 P 01f6144ac0484e1d9a6da7ff80ba71f0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader dfd7ad6293204fc89f9e72436796698b) I0703 01:42:13.871651 27390 raft_consensus.cc:2730] T 6359db2dd602463e841abc2a907c49a3 P 01f6144ac0484e1d9a6da7ff80ba71f0: Snoozing failure detection for 1.738s (starting election) I0703 01:42:13.871716 27390 raft_consensus.cc:2782] T 6359db2dd602463e841abc2a907c49a3 P 01f6144ac0484e1d9a6da7ff80ba71f0 [term 1 FOLLOWER]: Advancing to term 2 I0703 01:42:13.873106 27390 raft_consensus.cc:479] T 6359db2dd602463e841abc2a907c49a3 P 01f6144ac0484e1d9a6da7ff80ba71f0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } peers { permanent_uuid: "01f6144ac0484e1d9a6da7ff80ba71f0" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 35740 } } I0703 01:42:13.873399 27390 leader_election.cc:243] T 6359db2dd602463e841abc2a907c49a3 P 01f6144ac0484e1d9a6da7ff80ba71f0 [CANDIDATE]: Term 2 election: Requesting vote from peer dfd7ad6293204fc89f9e72436796698b (127.5.240.193:43198) I0703 01:42:13.874225 24919 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "6359db2dd602463e841abc2a907c49a3" candidate_uuid: "01f6144ac0484e1d9a6da7ff80ba71f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dfd7ad6293204fc89f9e72436796698b" I0703 01:42:13.874351 24919 raft_consensus.cc:2782] T 6359db2dd602463e841abc2a907c49a3 P dfd7ad6293204fc89f9e72436796698b [term 1 FOLLOWER]: Advancing to term 2 I0703 01:42:13.874414 24919 raft_consensus.cc:2730] T 6359db2dd602463e841abc2a907c49a3 P dfd7ad6293204fc89f9e72436796698b: Snoozing failure detection for 1.849s (vote granted) I0703 01:42:13.875743 24919 raft_consensus.cc:2243] T 6359db2dd602463e841abc2a907c49a3 P dfd7ad6293204fc89f9e72436796698b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01f6144ac0484e1d9a6da7ff80ba71f0 in term 2. I0703 01:42:13.876204 26238 leader_election.cc:379] T 6359db2dd602463e841abc2a907c49a3 P 01f6144ac0484e1d9a6da7ff80ba71f0 [CANDIDATE]: Term 2 election: Vote granted by peer dfd7ad6293204fc89f9e72436796698b (127.5.240.193:43198) I0703 01:42:13.876260 26238 leader_election.cc:270] T 6359db2dd602463e841abc2a907c49a3 P 01f6144ac0484e1d9a6da7ff80ba71f0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0703 01:42:13.876466 27390 raft_consensus.cc:2730] T 6359db2dd602463e841abc2a907c49a3 P 01f6144ac0484e1d9a6da7ff80ba71f0: Snoozing failure detection for 2.102s (election complete) I0703 01:42:13.876538 27390 raft_consensus.cc:2537] T 6359db2dd602463e841abc2a907c49a3 P 01f6144ac0484e1d9a6da7ff80ba71f0 [term 2 FOLLOWER]: Leader election won for term 2 I0703 01:42:13.876886 27390 raft_consensus.cc:593] T 6359db2dd602463e841abc2a907c49a3 P 01f6144ac0484e1d9a6da7ff80ba71f0 [term 2 LEADER]: Becoming Leader. State: Replica: 01f6144ac0484e1d9a6da7ff80ba71f0, State: Running, Role: LEADER I0703 01:42:13.877082 27390 consensus_queue.cc:204] T 6359db2dd602463e841abc2a907c49a3 P 01f6144ac0484e1d9a6da7ff80ba71f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } peers { permanent_uuid: "01f6144ac0484e1d9a6da7ff80ba71f0" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 35740 } } I0703 01:42:13.877236 24919 raft_consensus.cc:1081] T 53242d0289d644faa0149e0a54c0984c P dfd7ad6293204fc89f9e72436796698b [term 2 FOLLOWER]: Refusing update from remote peer cff51d1a4baa4cd0a1625e1d7db94f2c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch) I0703 01:42:13.877916 27598 consensus_queue.cc:962] T 53242d0289d644faa0149e0a54c0984c P cff51d1a4baa4cd0a1625e1d7db94f2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I0703 01:42:13.879112 24797 catalog_manager.cc:3798] T 6359db2dd602463e841abc2a907c49a3 P 01f6144ac0484e1d9a6da7ff80ba71f0 reported cstate change: term changed from 1 to 2, leader changed from dfd7ad6293204fc89f9e72436796698b (127.5.240.193) to 01f6144ac0484e1d9a6da7ff80ba71f0 (127.5.240.198). New cstate: current_term: 2 leader_uuid: "01f6144ac0484e1d9a6da7ff80ba71f0" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } peers { permanent_uuid: "01f6144ac0484e1d9a6da7ff80ba71f0" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 35740 } } } I0703 01:42:14.006990 25042 raft_consensus.cc:1081] T dae8746553524b6a9eb2e3f177f658f1 P 8ec1760d41f842f398d90e47ab52973b [term 2 FOLLOWER]: Refusing update from remote peer a344baa95900451dbaf349b68e662b27: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch) I0703 01:42:14.007761 27452 consensus_queue.cc:962] T dae8746553524b6a9eb2e3f177f658f1 P a344baa95900451dbaf349b68e662b27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I0703 01:42:14.062258 27513 raft_consensus.cc:457] T 7d1747555a45487e869ed52665686c52 P 2f7e2b9db52f44ba81b7393403f9ed37 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dfd7ad6293204fc89f9e72436796698b) I0703 01:42:14.062340 27513 raft_consensus.cc:2730] T 7d1747555a45487e869ed52665686c52 P 2f7e2b9db52f44ba81b7393403f9ed37: Snoozing failure detection for 1.638s (starting election) I0703 01:42:14.062419 27513 raft_consensus.cc:479] T 7d1747555a45487e869ed52665686c52 P 2f7e2b9db52f44ba81b7393403f9ed37 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } peers { permanent_uuid: "2f7e2b9db52f44ba81b7393403f9ed37" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 45230 } } I0703 01:42:14.062754 27513 leader_election.cc:243] T 7d1747555a45487e869ed52665686c52 P 2f7e2b9db52f44ba81b7393403f9ed37 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer dfd7ad6293204fc89f9e72436796698b (127.5.240.193:43198) I0703 01:42:14.063453 24919 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "7d1747555a45487e869ed52665686c52" candidate_uuid: "2f7e2b9db52f44ba81b7393403f9ed37" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dfd7ad6293204fc89f9e72436796698b" is_pre_election: true I0703 01:42:14.063616 24919 raft_consensus.cc:2730] T 7d1747555a45487e869ed52665686c52 P dfd7ad6293204fc89f9e72436796698b: Snoozing failure detection for 1.968s (vote granted) I0703 01:42:14.063693 24919 raft_consensus.cc:2243] T 7d1747555a45487e869ed52665686c52 P dfd7ad6293204fc89f9e72436796698b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f7e2b9db52f44ba81b7393403f9ed37 in term 1. I0703 01:42:14.064184 26116 leader_election.cc:379] T 7d1747555a45487e869ed52665686c52 P 2f7e2b9db52f44ba81b7393403f9ed37 [CANDIDATE]: Term 2 pre-election: Vote granted by peer dfd7ad6293204fc89f9e72436796698b (127.5.240.193:43198) I0703 01:42:14.064241 26116 leader_election.cc:270] T 7d1747555a45487e869ed52665686c52 P 2f7e2b9db52f44ba81b7393403f9ed37 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. I0703 01:42:14.064433 27513 raft_consensus.cc:2730] T 7d1747555a45487e869ed52665686c52 P 2f7e2b9db52f44ba81b7393403f9ed37: Snoozing failure detection for 1.677s (election complete) I0703 01:42:14.064512 27513 raft_consensus.cc:2537] T 7d1747555a45487e869ed52665686c52 P 2f7e2b9db52f44ba81b7393403f9ed37 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0703 01:42:14.064563 27513 raft_consensus.cc:457] T 7d1747555a45487e869ed52665686c52 P 2f7e2b9db52f44ba81b7393403f9ed37 [term 1 FOLLOWER]: Starting leader election (detected failure of leader dfd7ad6293204fc89f9e72436796698b) I0703 01:42:14.064643 27513 raft_consensus.cc:2730] T 7d1747555a45487e869ed52665686c52 P 2f7e2b9db52f44ba81b7393403f9ed37: Snoozing failure detection for 1.872s (starting election) I0703 01:42:14.064703 27513 raft_consensus.cc:2782] T 7d1747555a45487e869ed52665686c52 P 2f7e2b9db52f44ba81b7393403f9ed37 [term 1 FOLLOWER]: Advancing to term 2 I0703 01:42:14.066102 27513 raft_consensus.cc:479] T 7d1747555a45487e869ed52665686c52 P 2f7e2b9db52f44ba81b7393403f9ed37 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } peers { permanent_uuid: "2f7e2b9db52f44ba81b7393403f9ed37" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 45230 } } I0703 01:42:14.066339 27513 leader_election.cc:243] T 7d1747555a45487e869ed52665686c52 P 2f7e2b9db52f44ba81b7393403f9ed37 [CANDIDATE]: Term 2 election: Requesting vote from peer dfd7ad6293204fc89f9e72436796698b (127.5.240.193:43198) I0703 01:42:14.066977 24919 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "7d1747555a45487e869ed52665686c52" candidate_uuid: "2f7e2b9db52f44ba81b7393403f9ed37" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dfd7ad6293204fc89f9e72436796698b" I0703 01:42:14.067081 24919 raft_consensus.cc:2782] T 7d1747555a45487e869ed52665686c52 P dfd7ad6293204fc89f9e72436796698b [term 1 FOLLOWER]: Advancing to term 2 I0703 01:42:14.067157 24919 raft_consensus.cc:2730] T 7d1747555a45487e869ed52665686c52 P dfd7ad6293204fc89f9e72436796698b: Snoozing failure detection for 1.700s (vote granted) I0703 01:42:14.068709 24919 raft_consensus.cc:2243] T 7d1747555a45487e869ed52665686c52 P dfd7ad6293204fc89f9e72436796698b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f7e2b9db52f44ba81b7393403f9ed37 in term 2. I0703 01:42:14.068967 24920 raft_consensus.cc:1081] T ff7fa826668f419fa3c5225e5d810b9c P dfd7ad6293204fc89f9e72436796698b [term 2 FOLLOWER]: Refusing update from remote peer ca6ae9994a78460ca6c4ae4ec3f4e9b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch) I0703 01:42:14.069319 26116 leader_election.cc:379] T 7d1747555a45487e869ed52665686c52 P 2f7e2b9db52f44ba81b7393403f9ed37 [CANDIDATE]: Term 2 election: Vote granted by peer dfd7ad6293204fc89f9e72436796698b (127.5.240.193:43198) I0703 01:42:14.069377 26116 leader_election.cc:270] T 7d1747555a45487e869ed52665686c52 P 2f7e2b9db52f44ba81b7393403f9ed37 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0703 01:42:14.069566 27513 raft_consensus.cc:2730] T 7d1747555a45487e869ed52665686c52 P 2f7e2b9db52f44ba81b7393403f9ed37: Snoozing failure detection for 1.690s (election complete) I0703 01:42:14.069641 27513 raft_consensus.cc:2537] T 7d1747555a45487e869ed52665686c52 P 2f7e2b9db52f44ba81b7393403f9ed37 [term 2 FOLLOWER]: Leader election won for term 2 I0703 01:42:14.069753 27513 raft_consensus.cc:593] T 7d1747555a45487e869ed52665686c52 P 2f7e2b9db52f44ba81b7393403f9ed37 [term 2 LEADER]: Becoming Leader. State: Replica: 2f7e2b9db52f44ba81b7393403f9ed37, State: Running, Role: LEADER I0703 01:42:14.069716 27291 consensus_queue.cc:962] T ff7fa826668f419fa3c5225e5d810b9c P ca6ae9994a78460ca6c4ae4ec3f4e9b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I0703 01:42:14.069957 27513 consensus_queue.cc:204] T 7d1747555a45487e869ed52665686c52 P 2f7e2b9db52f44ba81b7393403f9ed37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } peers { permanent_uuid: "2f7e2b9db52f44ba81b7393403f9ed37" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 45230 } } I0703 01:42:14.072547 24797 catalog_manager.cc:3798] T 7d1747555a45487e869ed52665686c52 P 2f7e2b9db52f44ba81b7393403f9ed37 reported cstate change: term changed from 1 to 2, leader changed from dfd7ad6293204fc89f9e72436796698b (127.5.240.193) to 2f7e2b9db52f44ba81b7393403f9ed37 (127.5.240.197). New cstate: current_term: 2 leader_uuid: "2f7e2b9db52f44ba81b7393403f9ed37" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 } } peers { permanent_uuid: "2f7e2b9db52f44ba81b7393403f9ed37" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 45230 } } } I0703 01:42:14.341423 25042 raft_consensus.cc:1081] T fd7030608be84fa9b61c3ae90e04e27e P 8ec1760d41f842f398d90e47ab52973b [term 2 FOLLOWER]: Refusing update from remote peer a344baa95900451dbaf349b68e662b27: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch) I0703 01:42:14.341980 27461 consensus_queue.cc:962] T fd7030608be84fa9b61c3ae90e04e27e P a344baa95900451dbaf349b68e662b27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ec1760d41f842f398d90e47ab52973b" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 46721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I0703 01:42:14.382488 24920 raft_consensus.cc:1081] T 6359db2dd602463e841abc2a907c49a3 P dfd7ad6293204fc89f9e72436796698b [term 2 FOLLOWER]: Refusing update from remote peer 01f6144ac0484e1d9a6da7ff80ba71f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch) I0703 01:42:14.383106 27390 consensus_queue.cc:962] T 6359db2dd602463e841abc2a907c49a3 P 01f6144ac0484e1d9a6da7ff80ba71f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I0703 01:42:14.473191 26183 consensus_peers.cc:455] T 9a818723ff79493aa17e7434040e8975 P 2f7e2b9db52f44ba81b7393403f9ed37 -> Peer 8ec1760d41f842f398d90e47ab52973b (127.5.240.194:46721): Closing peer: 8ec1760d41f842f398d90e47ab52973b I0703 01:42:14.473448 26183 consensus_queue.cc:204] T 9a818723ff79493aa17e7434040e8975 P 2f7e2b9db52f44ba81b7393403f9ed37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2f7e2b9db52f44ba81b7393403f9ed37" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 45230 } } I0703 01:42:14.474133 27513 raft_consensus.cc:2684] T 9a818723ff79493aa17e7434040e8975 P 2f7e2b9db52f44ba81b7393403f9ed37 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 8ec1760d41f842f398d90e47ab52973b (127.5.240.194) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2f7e2b9db52f44ba81b7393403f9ed37" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 45230 } } } I0703 01:42:14.477032 24793 catalog_manager.cc:3798] T 9a818723ff79493aa17e7434040e8975 P 2f7e2b9db52f44ba81b7393403f9ed37 reported cstate change: config changed from index 4 to 6, VOTER 8ec1760d41f842f398d90e47ab52973b (127.5.240.194) evicted. New cstate: current_term: 2 leader_uuid: "2f7e2b9db52f44ba81b7393403f9ed37" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2f7e2b9db52f44ba81b7393403f9ed37" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 45230 } } } I0703 01:42:14.479372 24793 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 9a818723ff79493aa17e7434040e8975 on 8ec1760d41f842f398d90e47ab52973b (127.5.240.194:46721) (TS 8ec1760d41f842f398d90e47ab52973b not found in new config with opid_index 6) I0703 01:42:14.479949 25022 tablet_service.cc:800] Processing DeleteTablet for tablet 9a818723ff79493aa17e7434040e8975 with delete_type TABLET_DATA_TOMBSTONED (TS 8ec1760d41f842f398d90e47ab52973b not found in new config with opid_index 6) from {username='jenkins'} at 127.5.240.194:45697 I0703 01:42:14.480468 27726 tablet_replica.cc:261] T 9a818723ff79493aa17e7434040e8975 P 8ec1760d41f842f398d90e47ab52973b: stopping tablet replica I0703 01:42:14.480572 27726 maintenance_manager.cc:237] P 8ec1760d41f842f398d90e47ab52973b: Unregistered op CompactRowSetsOp(9a818723ff79493aa17e7434040e8975) I0703 01:42:14.480659 27726 maintenance_manager.cc:237] P 8ec1760d41f842f398d90e47ab52973b: Unregistered op MinorDeltaCompactionOp(9a818723ff79493aa17e7434040e8975) I0703 01:42:14.480712 27726 maintenance_manager.cc:237] P 8ec1760d41f842f398d90e47ab52973b: Unregistered op MajorDeltaCompactionOp(9a818723ff79493aa17e7434040e8975) I0703 01:42:14.480798 27726 maintenance_manager.cc:237] P 8ec1760d41f842f398d90e47ab52973b: Unregistered op UndoDeltaBlockGCOp(9a818723ff79493aa17e7434040e8975) I0703 01:42:14.480876 27726 maintenance_manager.cc:237] P 8ec1760d41f842f398d90e47ab52973b: Unregistered op FlushMRSOp(9a818723ff79493aa17e7434040e8975) I0703 01:42:14.480926 27726 maintenance_manager.cc:237] P 8ec1760d41f842f398d90e47ab52973b: Unregistered op FlushDeltaMemStoresOp(9a818723ff79493aa17e7434040e8975) I0703 01:42:14.480971 27726 maintenance_manager.cc:237] P 8ec1760d41f842f398d90e47ab52973b: Unregistered op LogGCOp(9a818723ff79493aa17e7434040e8975) I0703 01:42:14.481025 27726 raft_consensus.cc:2040] T 9a818723ff79493aa17e7434040e8975 P 8ec1760d41f842f398d90e47ab52973b [term 2 FOLLOWER]: Raft consensus shutting down. I0703 01:42:14.481137 27726 raft_consensus.cc:2067] T 9a818723ff79493aa17e7434040e8975 P 8ec1760d41f842f398d90e47ab52973b [term 2 FOLLOWER]: Raft consensus is shut down! I0703 01:42:14.481961 27726 ts_tablet_manager.cc:1369] T 9a818723ff79493aa17e7434040e8975 P 8ec1760d41f842f398d90e47ab52973b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:42:14.484676 27726 ts_tablet_manager.cc:1382] T 9a818723ff79493aa17e7434040e8975 P 8ec1760d41f842f398d90e47ab52973b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5 I0703 01:42:14.484752 27726 log.cc:986] T 9a818723ff79493aa17e7434040e8975 P 8ec1760d41f842f398d90e47ab52973b: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal/wals/9a818723ff79493aa17e7434040e8975 I0703 01:42:14.485322 24783 catalog_manager.cc:3103] TS 8ec1760d41f842f398d90e47ab52973b (127.5.240.194:46721): tablet 9a818723ff79493aa17e7434040e8975 (table rebalance_test_table_1 [id=fce1efea49b34f7fa21533a1026074fb]) successfully deleted I0703 01:42:14.597916 26305 consensus_peers.cc:455] T 0a2438297555462aa42cb91089c7a618 P 01f6144ac0484e1d9a6da7ff80ba71f0 -> Peer dfd7ad6293204fc89f9e72436796698b (127.5.240.193:43198): Closing peer: dfd7ad6293204fc89f9e72436796698b I0703 01:42:14.598172 26305 consensus_queue.cc:204] T 0a2438297555462aa42cb91089c7a618 P 01f6144ac0484e1d9a6da7ff80ba71f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "01f6144ac0484e1d9a6da7ff80ba71f0" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 35740 } } I0703 01:42:14.598856 27390 raft_consensus.cc:2684] T 0a2438297555462aa42cb91089c7a618 P 01f6144ac0484e1d9a6da7ff80ba71f0 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER dfd7ad6293204fc89f9e72436796698b (127.5.240.193) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "01f6144ac0484e1d9a6da7ff80ba71f0" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 35740 } } } I0703 01:42:14.601595 24793 catalog_manager.cc:3798] T 0a2438297555462aa42cb91089c7a618 P 01f6144ac0484e1d9a6da7ff80ba71f0 reported cstate change: config changed from index 4 to 6, VOTER dfd7ad6293204fc89f9e72436796698b (127.5.240.193) evicted. New cstate: current_term: 2 leader_uuid: "01f6144ac0484e1d9a6da7ff80ba71f0" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "01f6144ac0484e1d9a6da7ff80ba71f0" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 35740 } } } I0703 01:42:14.603762 24793 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 0a2438297555462aa42cb91089c7a618 on dfd7ad6293204fc89f9e72436796698b (127.5.240.193:43198) (TS dfd7ad6293204fc89f9e72436796698b not found in new config with opid_index 6) I0703 01:42:14.604347 24899 tablet_service.cc:800] Processing DeleteTablet for tablet 0a2438297555462aa42cb91089c7a618 with delete_type TABLET_DATA_TOMBSTONED (TS dfd7ad6293204fc89f9e72436796698b not found in new config with opid_index 6) from {username='jenkins'} at 127.5.240.193:49886 I0703 01:42:14.604993 27773 tablet_replica.cc:261] T 0a2438297555462aa42cb91089c7a618 P dfd7ad6293204fc89f9e72436796698b: stopping tablet replica I0703 01:42:14.605125 27773 maintenance_manager.cc:237] P dfd7ad6293204fc89f9e72436796698b: Unregistered op CompactRowSetsOp(0a2438297555462aa42cb91089c7a618) I0703 01:42:14.605199 27773 maintenance_manager.cc:237] P dfd7ad6293204fc89f9e72436796698b: Unregistered op MinorDeltaCompactionOp(0a2438297555462aa42cb91089c7a618) I0703 01:42:14.605252 27773 maintenance_manager.cc:237] P dfd7ad6293204fc89f9e72436796698b: Unregistered op MajorDeltaCompactionOp(0a2438297555462aa42cb91089c7a618) I0703 01:42:14.605312 27773 maintenance_manager.cc:237] P dfd7ad6293204fc89f9e72436796698b: Unregistered op UndoDeltaBlockGCOp(0a2438297555462aa42cb91089c7a618) I0703 01:42:14.605387 27773 maintenance_manager.cc:237] P dfd7ad6293204fc89f9e72436796698b: Unregistered op FlushMRSOp(0a2438297555462aa42cb91089c7a618) I0703 01:42:14.605437 27773 maintenance_manager.cc:237] P dfd7ad6293204fc89f9e72436796698b: Unregistered op FlushDeltaMemStoresOp(0a2438297555462aa42cb91089c7a618) I0703 01:42:14.605480 27773 maintenance_manager.cc:237] P dfd7ad6293204fc89f9e72436796698b: Unregistered op LogGCOp(0a2438297555462aa42cb91089c7a618) I0703 01:42:14.605556 27773 raft_consensus.cc:2040] T 0a2438297555462aa42cb91089c7a618 P dfd7ad6293204fc89f9e72436796698b [term 2 FOLLOWER]: Raft consensus shutting down. I0703 01:42:14.605743 27773 raft_consensus.cc:2067] T 0a2438297555462aa42cb91089c7a618 P dfd7ad6293204fc89f9e72436796698b [term 2 FOLLOWER]: Raft consensus is shut down! I0703 01:42:14.607497 27773 ts_tablet_manager.cc:1369] T 0a2438297555462aa42cb91089c7a618 P dfd7ad6293204fc89f9e72436796698b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:42:14.610883 27773 ts_tablet_manager.cc:1382] T 0a2438297555462aa42cb91089c7a618 P dfd7ad6293204fc89f9e72436796698b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5 I0703 01:42:14.610976 27773 log.cc:986] T 0a2438297555462aa42cb91089c7a618 P dfd7ad6293204fc89f9e72436796698b: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal/wals/0a2438297555462aa42cb91089c7a618 I0703 01:42:14.611552 24783 catalog_manager.cc:3103] TS dfd7ad6293204fc89f9e72436796698b (127.5.240.193:43198): tablet 0a2438297555462aa42cb91089c7a618 (table rebalance_test_table_2 [id=92effdbc87a0486eab9ab6de8407a54f]) successfully deleted I0703 01:42:14.650837 24920 raft_consensus.cc:1081] T 7d1747555a45487e869ed52665686c52 P dfd7ad6293204fc89f9e72436796698b [term 2 FOLLOWER]: Refusing update from remote peer 2f7e2b9db52f44ba81b7393403f9ed37: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch) I0703 01:42:14.651420 27513 consensus_queue.cc:962] T 7d1747555a45487e869ed52665686c52 P 2f7e2b9db52f44ba81b7393403f9ed37 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfd7ad6293204fc89f9e72436796698b" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 43198 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I0703 01:42:14.710099 25931 consensus_peers.cc:455] T 53242d0289d644faa0149e0a54c0984c P cff51d1a4baa4cd0a1625e1d7db94f2c -> Peer dfd7ad6293204fc89f9e72436796698b (127.5.240.193:43198): Closing peer: dfd7ad6293204fc89f9e72436796698b I0703 01:42:14.710346 25931 consensus_queue.cc:204] T 53242d0289d644faa0149e0a54c0984c P cff51d1a4baa4cd0a1625e1d7db94f2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cff51d1a4baa4cd0a1625e1d7db94f2c" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 38199 } } I0703 01:42:14.711019 27601 raft_consensus.cc:2684] T 53242d0289d644faa0149e0a54c0984c P cff51d1a4baa4cd0a1625e1d7db94f2c [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER dfd7ad6293204fc89f9e72436796698b (127.5.240.193) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cff51d1a4baa4cd0a1625e1d7db94f2c" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 38199 } } } I0703 01:42:14.714630 24793 catalog_manager.cc:3798] T 53242d0289d644faa0149e0a54c0984c P cff51d1a4baa4cd0a1625e1d7db94f2c reported cstate change: config changed from index 4 to 6, VOTER dfd7ad6293204fc89f9e72436796698b (127.5.240.193) evicted. New cstate: current_term: 2 leader_uuid: "cff51d1a4baa4cd0a1625e1d7db94f2c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cff51d1a4baa4cd0a1625e1d7db94f2c" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 38199 } } } I0703 01:42:14.716981 24793 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 53242d0289d644faa0149e0a54c0984c on dfd7ad6293204fc89f9e72436796698b (127.5.240.193:43198) (TS dfd7ad6293204fc89f9e72436796698b not found in new config with opid_index 6) I0703 01:42:14.717506 24899 tablet_service.cc:800] Processing DeleteTablet for tablet 53242d0289d644faa0149e0a54c0984c with delete_type TABLET_DATA_TOMBSTONED (TS dfd7ad6293204fc89f9e72436796698b not found in new config with opid_index 6) from {username='jenkins'} at 127.5.240.193:49886 I0703 01:42:14.717739 27773 tablet_replica.cc:261] T 53242d0289d644faa0149e0a54c0984c P dfd7ad6293204fc89f9e72436796698b: stopping tablet replica I0703 01:42:14.717835 27773 maintenance_manager.cc:237] P dfd7ad6293204fc89f9e72436796698b: Unregistered op CompactRowSetsOp(53242d0289d644faa0149e0a54c0984c) I0703 01:42:14.717900 27773 maintenance_manager.cc:237] P dfd7ad6293204fc89f9e72436796698b: Unregistered op MinorDeltaCompactionOp(53242d0289d644faa0149e0a54c0984c) I0703 01:42:14.717964 27773 maintenance_manager.cc:237] P dfd7ad6293204fc89f9e72436796698b: Unregistered op MajorDeltaCompactionOp(53242d0289d644faa0149e0a54c0984c) I0703 01:42:14.718013 27773 maintenance_manager.cc:237] P dfd7ad6293204fc89f9e72436796698b: Unregistered op UndoDeltaBlockGCOp(53242d0289d644faa0149e0a54c0984c) I0703 01:42:14.718072 27773 maintenance_manager.cc:237] P dfd7ad6293204fc89f9e72436796698b: Unregistered op FlushMRSOp(53242d0289d644faa0149e0a54c0984c) I0703 01:42:14.718107 27773 maintenance_manager.cc:237] P dfd7ad6293204fc89f9e72436796698b: Unregistered op FlushDeltaMemStoresOp(53242d0289d644faa0149e0a54c0984c) I0703 01:42:14.718138 27773 maintenance_manager.cc:237] P dfd7ad6293204fc89f9e72436796698b: Unregistered op LogGCOp(53242d0289d644faa0149e0a54c0984c) I0703 01:42:14.718169 27773 raft_consensus.cc:2040] T 53242d0289d644faa0149e0a54c0984c P dfd7ad6293204fc89f9e72436796698b [term 2 FOLLOWER]: Raft consensus shutting down. I0703 01:42:14.718255 27773 raft_consensus.cc:2067] T 53242d0289d644faa0149e0a54c0984c P dfd7ad6293204fc89f9e72436796698b [term 2 FOLLOWER]: Raft consensus is shut down! I0703 01:42:14.719180 27773 ts_tablet_manager.cc:1369] T 53242d0289d644faa0149e0a54c0984c P dfd7ad6293204fc89f9e72436796698b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:42:14.721962 27773 ts_tablet_manager.cc:1382] T 53242d0289d644faa0149e0a54c0984c P dfd7ad6293204fc89f9e72436796698b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5 I0703 01:42:14.722316 27773 log.cc:986] T 53242d0289d644faa0149e0a54c0984c P dfd7ad6293204fc89f9e72436796698b: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal/wals/53242d0289d644faa0149e0a54c0984c I0703 01:42:14.722836 24783 catalog_manager.cc:3103] TS dfd7ad6293204fc89f9e72436796698b (127.5.240.193:43198): tablet 53242d0289d644faa0149e0a54c0984c (table rebalance_test_table_4 [id=199091349c4b44e3a05e6e999c26968e]) successfully deleted I0703 01:42:14.827313 26061 consensus_peers.cc:455] T ff7fa826668f419fa3c5225e5d810b9c P ca6ae9994a78460ca6c4ae4ec3f4e9b5 -> Peer dfd7ad6293204fc89f9e72436796698b (127.5.240.193:43198): Closing peer: dfd7ad6293204fc89f9e72436796698b I0703 01:42:14.827491 26061 consensus_queue.cc:204] T ff7fa826668f419fa3c5225e5d810b9c P ca6ae9994a78460ca6c4ae4ec3f4e9b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ca6ae9994a78460ca6c4ae4ec3f4e9b5" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 41121 } } I0703 01:42:14.828122 27757 raft_consensus.cc:2684] T ff7fa826668f419fa3c5225e5d810b9c P ca6ae9994a78460ca6c4ae4ec3f4e9b5 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER dfd7ad6293204fc89f9e72436796698b (127.5.240.193) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ca6ae9994a78460ca6c4ae4ec3f4e9b5" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 41121 } } } I0703 01:42:14.831390 24793 catalog_manager.cc:3798] T ff7fa826668f419fa3c5225e5d810b9c P ca6ae9994a78460ca6c4ae4ec3f4e9b5 reported cstate change: config changed from index 4 to 6, VOTER dfd7ad6293204fc89f9e72436796698b (127.5.240.193) evicted. New cstate: current_term: 2 leader_uuid: "ca6ae9994a78460ca6c4ae4ec3f4e9b5" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ca6ae9994a78460ca6c4ae4ec3f4e9b5" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 41121 } } } I0703 01:42:14.833531 24793 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet ff7fa826668f419fa3c5225e5d810b9c on dfd7ad6293204fc89f9e72436796698b (127.5.240.193:43198) (TS dfd7ad6293204fc89f9e72436796698b not found in new config with opid_index 6) I0703 01:42:14.833979 24899 tablet_service.cc:800] Processing DeleteTablet for tablet ff7fa826668f419fa3c5225e5d810b9c with delete_type TABLET_DATA_TOMBSTONED (TS dfd7ad6293204fc89f9e72436796698b not found in new config with opid_index 6) from {username='jenkins'} at 127.5.240.193:49886 I0703 01:42:14.834223 27773 tablet_replica.cc:261] T ff7fa826668f419fa3c5225e5d810b9c P dfd7ad6293204fc89f9e72436796698b: stopping tablet replica I0703 01:42:14.834300 27773 maintenance_manager.cc:237] P dfd7ad6293204fc89f9e72436796698b: Unregistered op CompactRowSetsOp(ff7fa826668f419fa3c5225e5d810b9c) I0703 01:42:14.834367 27773 maintenance_manager.cc:237] P dfd7ad6293204fc89f9e72436796698b: Unregistered op MinorDeltaCompactionOp(ff7fa826668f419fa3c5225e5d810b9c) I0703 01:42:14.834419 27773 maintenance_manager.cc:237] P dfd7ad6293204fc89f9e72436796698b: Unregistered op MajorDeltaCompactionOp(ff7fa826668f419fa3c5225e5d810b9c) I0703 01:42:14.834473 27773 maintenance_manager.cc:237] P dfd7ad6293204fc89f9e72436796698b: Unregistered op UndoDeltaBlockGCOp(ff7fa826668f419fa3c5225e5d810b9c) I0703 01:42:14.834535 27773 maintenance_manager.cc:237] P dfd7ad6293204fc89f9e72436796698b: Unregistered op FlushMRSOp(ff7fa826668f419fa3c5225e5d810b9c) I0703 01:42:14.834602 27773 maintenance_manager.cc:237] P dfd7ad6293204fc89f9e72436796698b: Unregistered op FlushDeltaMemStoresOp(ff7fa826668f419fa3c5225e5d810b9c) I0703 01:42:14.834647 27773 maintenance_manager.cc:237] P dfd7ad6293204fc89f9e72436796698b: Unregistered op LogGCOp(ff7fa826668f419fa3c5225e5d810b9c) I0703 01:42:14.834697 27773 raft_consensus.cc:2040] T ff7fa826668f419fa3c5225e5d810b9c P dfd7ad6293204fc89f9e72436796698b [term 2 FOLLOWER]: Raft consensus shutting down. I0703 01:42:14.834789 27773 raft_consensus.cc:2067] T ff7fa826668f419fa3c5225e5d810b9c P dfd7ad6293204fc89f9e72436796698b [term 2 FOLLOWER]: Raft consensus is shut down! I0703 01:42:14.835536 27773 ts_tablet_manager.cc:1369] T ff7fa826668f419fa3c5225e5d810b9c P dfd7ad6293204fc89f9e72436796698b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:42:14.838873 27773 ts_tablet_manager.cc:1382] T ff7fa826668f419fa3c5225e5d810b9c P dfd7ad6293204fc89f9e72436796698b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5 I0703 01:42:14.838948 27773 log.cc:986] T ff7fa826668f419fa3c5225e5d810b9c P dfd7ad6293204fc89f9e72436796698b: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal/wals/ff7fa826668f419fa3c5225e5d810b9c I0703 01:42:14.839431 24783 catalog_manager.cc:3103] TS dfd7ad6293204fc89f9e72436796698b (127.5.240.193:43198): tablet ff7fa826668f419fa3c5225e5d810b9c (table rebalance_test_table_0 [id=fa6dde76da3d4d7e904d1ea3e3be2d1d]) successfully deleted I0703 01:42:14.949230 26426 consensus_peers.cc:455] T dae8746553524b6a9eb2e3f177f658f1 P a344baa95900451dbaf349b68e662b27 -> Peer 8ec1760d41f842f398d90e47ab52973b (127.5.240.194:46721): Closing peer: 8ec1760d41f842f398d90e47ab52973b I0703 01:42:14.949489 26426 consensus_queue.cc:204] T dae8746553524b6a9eb2e3f177f658f1 P a344baa95900451dbaf349b68e662b27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a344baa95900451dbaf349b68e662b27" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 43710 } } I0703 01:42:14.950179 27452 raft_consensus.cc:2684] T dae8746553524b6a9eb2e3f177f658f1 P a344baa95900451dbaf349b68e662b27 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 8ec1760d41f842f398d90e47ab52973b (127.5.240.194) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a344baa95900451dbaf349b68e662b27" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 43710 } } } I0703 01:42:14.953435 24793 catalog_manager.cc:3798] T dae8746553524b6a9eb2e3f177f658f1 P a344baa95900451dbaf349b68e662b27 reported cstate change: config changed from index 4 to 6, VOTER 8ec1760d41f842f398d90e47ab52973b (127.5.240.194) evicted. New cstate: current_term: 2 leader_uuid: "a344baa95900451dbaf349b68e662b27" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a344baa95900451dbaf349b68e662b27" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 43710 } } } I0703 01:42:14.955775 24793 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet dae8746553524b6a9eb2e3f177f658f1 on 8ec1760d41f842f398d90e47ab52973b (127.5.240.194:46721) (TS 8ec1760d41f842f398d90e47ab52973b not found in new config with opid_index 6) I0703 01:42:14.956354 25022 tablet_service.cc:800] Processing DeleteTablet for tablet dae8746553524b6a9eb2e3f177f658f1 with delete_type TABLET_DATA_TOMBSTONED (TS 8ec1760d41f842f398d90e47ab52973b not found in new config with opid_index 6) from {username='jenkins'} at 127.5.240.194:45697 I0703 01:42:14.956624 27726 tablet_replica.cc:261] T dae8746553524b6a9eb2e3f177f658f1 P 8ec1760d41f842f398d90e47ab52973b: stopping tablet replica I0703 01:42:14.956717 27726 maintenance_manager.cc:237] P 8ec1760d41f842f398d90e47ab52973b: Unregistered op CompactRowSetsOp(dae8746553524b6a9eb2e3f177f658f1) I0703 01:42:14.956773 27726 maintenance_manager.cc:237] P 8ec1760d41f842f398d90e47ab52973b: Unregistered op MinorDeltaCompactionOp(dae8746553524b6a9eb2e3f177f658f1) I0703 01:42:14.956835 27726 maintenance_manager.cc:237] P 8ec1760d41f842f398d90e47ab52973b: Unregistered op MajorDeltaCompactionOp(dae8746553524b6a9eb2e3f177f658f1) I0703 01:42:14.956897 27726 maintenance_manager.cc:237] P 8ec1760d41f842f398d90e47ab52973b: Unregistered op UndoDeltaBlockGCOp(dae8746553524b6a9eb2e3f177f658f1) I0703 01:42:14.956962 27726 maintenance_manager.cc:237] P 8ec1760d41f842f398d90e47ab52973b: Unregistered op FlushMRSOp(dae8746553524b6a9eb2e3f177f658f1) I0703 01:42:14.957010 27726 maintenance_manager.cc:237] P 8ec1760d41f842f398d90e47ab52973b: Unregistered op FlushDeltaMemStoresOp(dae8746553524b6a9eb2e3f177f658f1) I0703 01:42:14.957059 27726 maintenance_manager.cc:237] P 8ec1760d41f842f398d90e47ab52973b: Unregistered op LogGCOp(dae8746553524b6a9eb2e3f177f658f1) I0703 01:42:14.957110 27726 raft_consensus.cc:2040] T dae8746553524b6a9eb2e3f177f658f1 P 8ec1760d41f842f398d90e47ab52973b [term 2 FOLLOWER]: Raft consensus shutting down. I0703 01:42:14.957212 27726 raft_consensus.cc:2067] T dae8746553524b6a9eb2e3f177f658f1 P 8ec1760d41f842f398d90e47ab52973b [term 2 FOLLOWER]: Raft consensus is shut down! I0703 01:42:14.957999 27726 ts_tablet_manager.cc:1369] T dae8746553524b6a9eb2e3f177f658f1 P 8ec1760d41f842f398d90e47ab52973b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:42:14.961035 27726 ts_tablet_manager.cc:1382] T dae8746553524b6a9eb2e3f177f658f1 P 8ec1760d41f842f398d90e47ab52973b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5 I0703 01:42:14.961086 27726 log.cc:986] T dae8746553524b6a9eb2e3f177f658f1 P 8ec1760d41f842f398d90e47ab52973b: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal/wals/dae8746553524b6a9eb2e3f177f658f1 I0703 01:42:14.961551 24783 catalog_manager.cc:3103] TS 8ec1760d41f842f398d90e47ab52973b (127.5.240.194:46721): tablet dae8746553524b6a9eb2e3f177f658f1 (table rebalance_test_table_3 [id=72ffb8cd105e4058a3b2caf04d27ee7f]) successfully deleted I0703 01:42:15.072883 26305 consensus_peers.cc:455] T 6359db2dd602463e841abc2a907c49a3 P 01f6144ac0484e1d9a6da7ff80ba71f0 -> Peer dfd7ad6293204fc89f9e72436796698b (127.5.240.193:43198): Closing peer: dfd7ad6293204fc89f9e72436796698b I0703 01:42:15.073084 26305 consensus_queue.cc:204] T 6359db2dd602463e841abc2a907c49a3 P 01f6144ac0484e1d9a6da7ff80ba71f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "01f6144ac0484e1d9a6da7ff80ba71f0" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 35740 } } I0703 01:42:15.073734 27390 raft_consensus.cc:2684] T 6359db2dd602463e841abc2a907c49a3 P 01f6144ac0484e1d9a6da7ff80ba71f0 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER dfd7ad6293204fc89f9e72436796698b (127.5.240.193) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "01f6144ac0484e1d9a6da7ff80ba71f0" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 35740 } } } I0703 01:42:15.076709 24793 catalog_manager.cc:3798] T 6359db2dd602463e841abc2a907c49a3 P 01f6144ac0484e1d9a6da7ff80ba71f0 reported cstate change: config changed from index 4 to 6, VOTER dfd7ad6293204fc89f9e72436796698b (127.5.240.193) evicted. New cstate: current_term: 2 leader_uuid: "01f6144ac0484e1d9a6da7ff80ba71f0" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "01f6144ac0484e1d9a6da7ff80ba71f0" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 35740 } } } I0703 01:42:15.078907 24793 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 6359db2dd602463e841abc2a907c49a3 on dfd7ad6293204fc89f9e72436796698b (127.5.240.193:43198) (TS dfd7ad6293204fc89f9e72436796698b not found in new config with opid_index 6) I0703 01:42:15.079581 24899 tablet_service.cc:800] Processing DeleteTablet for tablet 6359db2dd602463e841abc2a907c49a3 with delete_type TABLET_DATA_TOMBSTONED (TS dfd7ad6293204fc89f9e72436796698b not found in new config with opid_index 6) from {username='jenkins'} at 127.5.240.193:49886 I0703 01:42:15.079797 27773 tablet_replica.cc:261] T 6359db2dd602463e841abc2a907c49a3 P dfd7ad6293204fc89f9e72436796698b: stopping tablet replica I0703 01:42:15.079879 27773 maintenance_manager.cc:237] P dfd7ad6293204fc89f9e72436796698b: Unregistered op CompactRowSetsOp(6359db2dd602463e841abc2a907c49a3) I0703 01:42:15.079941 27773 maintenance_manager.cc:237] P dfd7ad6293204fc89f9e72436796698b: Unregistered op MinorDeltaCompactionOp(6359db2dd602463e841abc2a907c49a3) I0703 01:42:15.079996 27773 maintenance_manager.cc:237] P dfd7ad6293204fc89f9e72436796698b: Unregistered op MajorDeltaCompactionOp(6359db2dd602463e841abc2a907c49a3) I0703 01:42:15.080049 27773 maintenance_manager.cc:237] P dfd7ad6293204fc89f9e72436796698b: Unregistered op UndoDeltaBlockGCOp(6359db2dd602463e841abc2a907c49a3) I0703 01:42:15.080108 27773 maintenance_manager.cc:237] P dfd7ad6293204fc89f9e72436796698b: Unregistered op FlushMRSOp(6359db2dd602463e841abc2a907c49a3) I0703 01:42:15.080154 27773 maintenance_manager.cc:237] P dfd7ad6293204fc89f9e72436796698b: Unregistered op FlushDeltaMemStoresOp(6359db2dd602463e841abc2a907c49a3) I0703 01:42:15.080202 27773 maintenance_manager.cc:237] P dfd7ad6293204fc89f9e72436796698b: Unregistered op LogGCOp(6359db2dd602463e841abc2a907c49a3) I0703 01:42:15.080250 27773 raft_consensus.cc:2040] T 6359db2dd602463e841abc2a907c49a3 P dfd7ad6293204fc89f9e72436796698b [term 2 FOLLOWER]: Raft consensus shutting down. I0703 01:42:15.080348 27773 raft_consensus.cc:2067] T 6359db2dd602463e841abc2a907c49a3 P dfd7ad6293204fc89f9e72436796698b [term 2 FOLLOWER]: Raft consensus is shut down! I0703 01:42:15.081151 27773 ts_tablet_manager.cc:1369] T 6359db2dd602463e841abc2a907c49a3 P dfd7ad6293204fc89f9e72436796698b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:42:15.083942 27773 ts_tablet_manager.cc:1382] T 6359db2dd602463e841abc2a907c49a3 P dfd7ad6293204fc89f9e72436796698b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5 I0703 01:42:15.084003 27773 log.cc:986] T 6359db2dd602463e841abc2a907c49a3 P dfd7ad6293204fc89f9e72436796698b: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal/wals/6359db2dd602463e841abc2a907c49a3 I0703 01:42:15.084625 24783 catalog_manager.cc:3103] TS dfd7ad6293204fc89f9e72436796698b (127.5.240.193:43198): tablet 6359db2dd602463e841abc2a907c49a3 (table rebalance_test_table_0 [id=fa6dde76da3d4d7e904d1ea3e3be2d1d]) successfully deleted I0703 01:42:15.201949 26426 consensus_peers.cc:455] T fd7030608be84fa9b61c3ae90e04e27e P a344baa95900451dbaf349b68e662b27 -> Peer 8ec1760d41f842f398d90e47ab52973b (127.5.240.194:46721): Closing peer: 8ec1760d41f842f398d90e47ab52973b I0703 01:42:15.202157 26426 consensus_queue.cc:204] T fd7030608be84fa9b61c3ae90e04e27e P a344baa95900451dbaf349b68e662b27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a344baa95900451dbaf349b68e662b27" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 43710 } } I0703 01:42:15.202793 27452 raft_consensus.cc:2684] T fd7030608be84fa9b61c3ae90e04e27e P a344baa95900451dbaf349b68e662b27 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 8ec1760d41f842f398d90e47ab52973b (127.5.240.194) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a344baa95900451dbaf349b68e662b27" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 43710 } } } I0703 01:42:15.206095 24797 catalog_manager.cc:3798] T fd7030608be84fa9b61c3ae90e04e27e P a344baa95900451dbaf349b68e662b27 reported cstate change: config changed from index 4 to 6, VOTER 8ec1760d41f842f398d90e47ab52973b (127.5.240.194) evicted. New cstate: current_term: 2 leader_uuid: "a344baa95900451dbaf349b68e662b27" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a344baa95900451dbaf349b68e662b27" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 43710 } } } I0703 01:42:15.208408 24797 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet fd7030608be84fa9b61c3ae90e04e27e on 8ec1760d41f842f398d90e47ab52973b (127.5.240.194:46721) (TS 8ec1760d41f842f398d90e47ab52973b not found in new config with opid_index 6) I0703 01:42:15.208914 25022 tablet_service.cc:800] Processing DeleteTablet for tablet fd7030608be84fa9b61c3ae90e04e27e with delete_type TABLET_DATA_TOMBSTONED (TS 8ec1760d41f842f398d90e47ab52973b not found in new config with opid_index 6) from {username='jenkins'} at 127.5.240.194:45697 I0703 01:42:15.209182 27726 tablet_replica.cc:261] T fd7030608be84fa9b61c3ae90e04e27e P 8ec1760d41f842f398d90e47ab52973b: stopping tablet replica I0703 01:42:15.209264 27726 maintenance_manager.cc:237] P 8ec1760d41f842f398d90e47ab52973b: Unregistered op CompactRowSetsOp(fd7030608be84fa9b61c3ae90e04e27e) I0703 01:42:15.209334 27726 maintenance_manager.cc:237] P 8ec1760d41f842f398d90e47ab52973b: Unregistered op MinorDeltaCompactionOp(fd7030608be84fa9b61c3ae90e04e27e) I0703 01:42:15.209385 27726 maintenance_manager.cc:237] P 8ec1760d41f842f398d90e47ab52973b: Unregistered op MajorDeltaCompactionOp(fd7030608be84fa9b61c3ae90e04e27e) I0703 01:42:15.209440 27726 maintenance_manager.cc:237] P 8ec1760d41f842f398d90e47ab52973b: Unregistered op UndoDeltaBlockGCOp(fd7030608be84fa9b61c3ae90e04e27e) I0703 01:42:15.209511 27726 maintenance_manager.cc:237] P 8ec1760d41f842f398d90e47ab52973b: Unregistered op FlushMRSOp(fd7030608be84fa9b61c3ae90e04e27e) I0703 01:42:15.209561 27726 maintenance_manager.cc:237] P 8ec1760d41f842f398d90e47ab52973b: Unregistered op FlushDeltaMemStoresOp(fd7030608be84fa9b61c3ae90e04e27e) I0703 01:42:15.209623 27726 maintenance_manager.cc:237] P 8ec1760d41f842f398d90e47ab52973b: Unregistered op LogGCOp(fd7030608be84fa9b61c3ae90e04e27e) I0703 01:42:15.209676 27726 raft_consensus.cc:2040] T fd7030608be84fa9b61c3ae90e04e27e P 8ec1760d41f842f398d90e47ab52973b [term 2 FOLLOWER]: Raft consensus shutting down. I0703 01:42:15.209779 27726 raft_consensus.cc:2067] T fd7030608be84fa9b61c3ae90e04e27e P 8ec1760d41f842f398d90e47ab52973b [term 2 FOLLOWER]: Raft consensus is shut down! I0703 01:42:15.210466 27726 ts_tablet_manager.cc:1369] T fd7030608be84fa9b61c3ae90e04e27e P 8ec1760d41f842f398d90e47ab52973b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:42:15.213667 27726 ts_tablet_manager.cc:1382] T fd7030608be84fa9b61c3ae90e04e27e P 8ec1760d41f842f398d90e47ab52973b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5 I0703 01:42:15.213774 27726 log.cc:986] T fd7030608be84fa9b61c3ae90e04e27e P 8ec1760d41f842f398d90e47ab52973b: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal/wals/fd7030608be84fa9b61c3ae90e04e27e I0703 01:42:15.214320 24783 catalog_manager.cc:3103] TS 8ec1760d41f842f398d90e47ab52973b (127.5.240.194:46721): tablet fd7030608be84fa9b61c3ae90e04e27e (table rebalance_test_table_1 [id=fce1efea49b34f7fa21533a1026074fb]) successfully deleted I0703 01:42:15.320116 26183 consensus_peers.cc:455] T 7d1747555a45487e869ed52665686c52 P 2f7e2b9db52f44ba81b7393403f9ed37 -> Peer dfd7ad6293204fc89f9e72436796698b (127.5.240.193:43198): Closing peer: dfd7ad6293204fc89f9e72436796698b I0703 01:42:15.320312 26183 consensus_queue.cc:204] T 7d1747555a45487e869ed52665686c52 P 2f7e2b9db52f44ba81b7393403f9ed37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2f7e2b9db52f44ba81b7393403f9ed37" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 45230 } } I0703 01:42:15.320950 27515 raft_consensus.cc:2684] T 7d1747555a45487e869ed52665686c52 P 2f7e2b9db52f44ba81b7393403f9ed37 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER dfd7ad6293204fc89f9e72436796698b (127.5.240.193) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2f7e2b9db52f44ba81b7393403f9ed37" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 45230 } } } I0703 01:42:15.324010 24797 catalog_manager.cc:3798] T 7d1747555a45487e869ed52665686c52 P 2f7e2b9db52f44ba81b7393403f9ed37 reported cstate change: config changed from index 4 to 6, VOTER dfd7ad6293204fc89f9e72436796698b (127.5.240.193) evicted. New cstate: current_term: 2 leader_uuid: "2f7e2b9db52f44ba81b7393403f9ed37" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2f7e2b9db52f44ba81b7393403f9ed37" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 45230 } } } I0703 01:42:15.326112 24797 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 7d1747555a45487e869ed52665686c52 on dfd7ad6293204fc89f9e72436796698b (127.5.240.193:43198) (TS dfd7ad6293204fc89f9e72436796698b not found in new config with opid_index 6) I0703 01:42:15.326573 24899 tablet_service.cc:800] Processing DeleteTablet for tablet 7d1747555a45487e869ed52665686c52 with delete_type TABLET_DATA_TOMBSTONED (TS dfd7ad6293204fc89f9e72436796698b not found in new config with opid_index 6) from {username='jenkins'} at 127.5.240.193:49886 I0703 01:42:15.326789 27773 tablet_replica.cc:261] T 7d1747555a45487e869ed52665686c52 P dfd7ad6293204fc89f9e72436796698b: stopping tablet replica I0703 01:42:15.326858 27773 maintenance_manager.cc:237] P dfd7ad6293204fc89f9e72436796698b: Unregistered op CompactRowSetsOp(7d1747555a45487e869ed52665686c52) I0703 01:42:15.326910 27773 maintenance_manager.cc:237] P dfd7ad6293204fc89f9e72436796698b: Unregistered op MinorDeltaCompactionOp(7d1747555a45487e869ed52665686c52) I0703 01:42:15.326951 27773 maintenance_manager.cc:237] P dfd7ad6293204fc89f9e72436796698b: Unregistered op MajorDeltaCompactionOp(7d1747555a45487e869ed52665686c52) I0703 01:42:15.326987 27773 maintenance_manager.cc:237] P dfd7ad6293204fc89f9e72436796698b: Unregistered op UndoDeltaBlockGCOp(7d1747555a45487e869ed52665686c52) I0703 01:42:15.327028 27773 maintenance_manager.cc:237] P dfd7ad6293204fc89f9e72436796698b: Unregistered op FlushMRSOp(7d1747555a45487e869ed52665686c52) I0703 01:42:15.327062 27773 maintenance_manager.cc:237] P dfd7ad6293204fc89f9e72436796698b: Unregistered op FlushDeltaMemStoresOp(7d1747555a45487e869ed52665686c52) I0703 01:42:15.327092 27773 maintenance_manager.cc:237] P dfd7ad6293204fc89f9e72436796698b: Unregistered op LogGCOp(7d1747555a45487e869ed52665686c52) I0703 01:42:15.327131 27773 raft_consensus.cc:2040] T 7d1747555a45487e869ed52665686c52 P dfd7ad6293204fc89f9e72436796698b [term 2 FOLLOWER]: Raft consensus shutting down. I0703 01:42:15.327214 27773 raft_consensus.cc:2067] T 7d1747555a45487e869ed52665686c52 P dfd7ad6293204fc89f9e72436796698b [term 2 FOLLOWER]: Raft consensus is shut down! I0703 01:42:15.327953 27773 ts_tablet_manager.cc:1369] T 7d1747555a45487e869ed52665686c52 P dfd7ad6293204fc89f9e72436796698b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:42:15.330899 27773 ts_tablet_manager.cc:1382] T 7d1747555a45487e869ed52665686c52 P dfd7ad6293204fc89f9e72436796698b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5 I0703 01:42:15.330973 27773 log.cc:986] T 7d1747555a45487e869ed52665686c52 P dfd7ad6293204fc89f9e72436796698b: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal/wals/7d1747555a45487e869ed52665686c52 I0703 01:42:15.331529 24783 catalog_manager.cc:3103] TS dfd7ad6293204fc89f9e72436796698b (127.5.240.193:43198): tablet 7d1747555a45487e869ed52665686c52 (table rebalance_test_table_2 [id=92effdbc87a0486eab9ab6de8407a54f]) successfully deleted Master Summary UUID | Address | Status ----------------------------------+---------------------+--------- 0ed28bce615e467daedd680797177631 | 127.5.240.254:57131 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- allow_unsafe_replication_factor | true | unsafe | all 1 server(s) checked ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status ----------------------------------+---------------------+--------- 01f6144ac0484e1d9a6da7ff80ba71f0 | 127.5.240.198:35740 | HEALTHY 2f7e2b9db52f44ba81b7393403f9ed37 | 127.5.240.197:45230 | HEALTHY 8ec1760d41f842f398d90e47ab52973b | 127.5.240.194:46721 | HEALTHY a344baa95900451dbaf349b68e662b27 | 127.5.240.199:43710 | HEALTHY ca6ae9994a78460ca6c4ae4ec3f4e9b5 | 127.5.240.196:41121 | HEALTHY cff51d1a4baa4cd0a1625e1d7db94f2c | 127.5.240.195:38199 | HEALTHY dfd7ad6293204fc89f9e72436796698b | 127.5.240.193:43198 | HEALTHY Flag | Value | Tags | Tablet Server ------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_server_key_size | 1024 | experimental | all 7 server(s) checked local_ip_for_outbound_sockets | 127.5.240.193 | experimental | 127.5.240.193:43198 local_ip_for_outbound_sockets | 127.5.240.194 | experimental | 127.5.240.194:46721 local_ip_for_outbound_sockets | 127.5.240.195 | experimental | 127.5.240.195:38199 local_ip_for_outbound_sockets | 127.5.240.196 | experimental | 127.5.240.196:41121 local_ip_for_outbound_sockets | 127.5.240.197 | experimental | 127.5.240.197:45230 local_ip_for_outbound_sockets | 127.5.240.198 | experimental | 127.5.240.198:35740 local_ip_for_outbound_sockets | 127.5.240.199 | experimental | 127.5.240.199:43710 never_fsync | true | unsafe,advanced | all 7 server(s) checked raft_prepare_replacement_before_eviction | false | experimental,advanced | all 7 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked server_dump_info_format | pb | hidden | all 7 server(s) checked server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.5.240.193:43198 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.240.194:46721 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.240.195:38199 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.5.240.196:41121 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.240.197:45230 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.240.198:35740 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_0.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.5.240.199:43710 Version Summary Version | Servers ----------------+------------------------- 1.8.0-SNAPSHOT | all 8 server(s) checked Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ------------------------+----+---------+---------------+---------+------------+------------------+------------- TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0 rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0 rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0 rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0 rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0 rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0 | Total Count ----------------+------------- Masters | 1 Tablet Servers | 7 Tables | 6 Tablets | 16 Replicas | 16 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set OK I0703 01:42:16.721122 6083 log_verifier.cc:121] Checking tablet 0a2438297555462aa42cb91089c7a618 I0703 01:42:16.732014 6083 log_verifier.cc:172] Verified matching terms for 6 ops in tablet 0a2438297555462aa42cb91089c7a618 I0703 01:42:16.732079 6083 log_verifier.cc:121] Checking tablet 46469043bcb248f6a4505bc705edcda2 I0703 01:42:16.743057 6083 log_verifier.cc:172] Verified matching terms for 3 ops in tablet 46469043bcb248f6a4505bc705edcda2 I0703 01:42:16.743106 6083 log_verifier.cc:121] Checking tablet 4eea38c54a6b491081040e6059ca592e I0703 01:42:16.753105 6083 log_verifier.cc:172] Verified matching terms for 3 ops in tablet 4eea38c54a6b491081040e6059ca592e I0703 01:42:16.753154 6083 log_verifier.cc:121] Checking tablet 53242d0289d644faa0149e0a54c0984c I0703 01:42:16.764032 6083 log_verifier.cc:172] Verified matching terms for 6 ops in tablet 53242d0289d644faa0149e0a54c0984c I0703 01:42:16.764088 6083 log_verifier.cc:121] Checking tablet 6359db2dd602463e841abc2a907c49a3 I0703 01:42:16.775049 6083 log_verifier.cc:172] Verified matching terms for 6 ops in tablet 6359db2dd602463e841abc2a907c49a3 I0703 01:42:16.775105 6083 log_verifier.cc:121] Checking tablet 72ebab6b7393457db959684913708522 I0703 01:42:16.785315 6083 log_verifier.cc:172] Verified matching terms for 6 ops in tablet 72ebab6b7393457db959684913708522 I0703 01:42:16.785368 6083 log_verifier.cc:121] Checking tablet 7d1747555a45487e869ed52665686c52 I0703 01:42:16.796058 6083 log_verifier.cc:172] Verified matching terms for 6 ops in tablet 7d1747555a45487e869ed52665686c52 I0703 01:42:16.796110 6083 log_verifier.cc:121] Checking tablet 93faa898182e46b2a6aab9df75230c0b I0703 01:42:16.805934 6083 log_verifier.cc:172] Verified matching terms for 3 ops in tablet 93faa898182e46b2a6aab9df75230c0b I0703 01:42:16.805980 6083 log_verifier.cc:121] Checking tablet 9a818723ff79493aa17e7434040e8975 I0703 01:42:16.816442 6083 log_verifier.cc:172] Verified matching terms for 6 ops in tablet 9a818723ff79493aa17e7434040e8975 I0703 01:42:16.816496 6083 log_verifier.cc:121] Checking tablet c7f0c202bc0c4351afc57e7efbbd5e02 I0703 01:42:16.826439 6083 log_verifier.cc:172] Verified matching terms for 3 ops in tablet c7f0c202bc0c4351afc57e7efbbd5e02 I0703 01:42:16.826490 6083 log_verifier.cc:121] Checking tablet dae8746553524b6a9eb2e3f177f658f1 I0703 01:42:16.836944 6083 log_verifier.cc:172] Verified matching terms for 6 ops in tablet dae8746553524b6a9eb2e3f177f658f1 I0703 01:42:16.836997 6083 log_verifier.cc:121] Checking tablet dcabbf8c71c14f248522b11a1426555d I0703 01:42:16.847013 6083 log_verifier.cc:172] Verified matching terms for 3 ops in tablet dcabbf8c71c14f248522b11a1426555d I0703 01:42:16.847065 6083 log_verifier.cc:121] Checking tablet ee4327166db34f66baa031d7e2da4999 I0703 01:42:16.858041 6083 log_verifier.cc:172] Verified matching terms for 3 ops in tablet ee4327166db34f66baa031d7e2da4999 I0703 01:42:16.858130 6083 log_verifier.cc:121] Checking tablet fc9b869cd219497896d16275d18c81b9 I0703 01:42:16.865993 6083 log_verifier.cc:172] Verified matching terms for 2 ops in tablet fc9b869cd219497896d16275d18c81b9 I0703 01:42:16.866041 6083 log_verifier.cc:121] Checking tablet fd7030608be84fa9b61c3ae90e04e27e I0703 01:42:16.876379 6083 log_verifier.cc:172] Verified matching terms for 6 ops in tablet fd7030608be84fa9b61c3ae90e04e27e I0703 01:42:16.876432 6083 log_verifier.cc:121] Checking tablet ff7fa826668f419fa3c5225e5d810b9c I0703 01:42:16.886829 6083 log_verifier.cc:172] Verified matching terms for 6 ops in tablet ff7fa826668f419fa3c5225e5d810b9c I0703 01:42:16.887722 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 24834 I0703 01:42:16.889948 24960 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:42:16.967650 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 24834 W0703 01:42:16.981659 24782 connection.cc:512] server connection from 127.5.240.193:41273 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:41273): Connection reset by peer (error 104) W0703 01:42:16.982095 26116 connection.cc:512] client connection to 127.5.240.193:43198 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:43198): Connection reset by peer (error 104) W0703 01:42:16.982154 25994 connection.cc:512] client connection to 127.5.240.193:43198 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:43198): Connection reset by peer (error 104) W0703 01:42:16.982650 26117 connection.cc:512] server connection from 127.5.240.193:55047 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:55047): Connection reset by peer (error 104) W0703 01:42:16.982683 25993 connection.cc:512] server connection from 127.5.240.193:34408 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:34408): Connection reset by peer (error 104) W0703 01:42:16.982751 25861 connection.cc:512] server connection from 127.5.240.193:52720 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:52720): Connection reset by peer (error 104) I0703 01:42:16.982826 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 24967 W0703 01:42:16.983067 25862 connection.cc:512] client connection to 127.5.240.193:43198 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:43198): Connection reset by peer (error 104) W0703 01:42:16.983141 26238 connection.cc:512] client connection to 127.5.240.193:43198 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:43198): Connection reset by peer (error 104) W0703 01:42:16.983216 26236 connection.cc:512] server connection from 127.5.240.193:40467 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:40467): Connection reset by peer (error 104) I0703 01:42:16.983774 25083 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:42:17.058337 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 24967 W0703 01:42:17.071808 26116 connection.cc:512] client connection to 127.5.240.194:46721 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:46721): Connection reset by peer (error 104) W0703 01:42:17.071918 24782 connection.cc:512] server connection from 127.5.240.194:33164 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:33164): Connection reset by peer (error 104) W0703 01:42:17.072268 26114 connection.cc:512] server connection from 127.5.240.194:36996 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:36996): Connection reset by peer (error 104) W0703 01:42:17.072399 26358 connection.cc:512] server connection from 127.5.240.194:46116 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:46116): Connection reset by peer (error 104) W0703 01:42:17.072399 26360 connection.cc:512] client connection to 127.5.240.194:46721 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:46721): Connection reset by peer (error 104) I0703 01:42:17.072656 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 25853 W0703 01:42:17.072914 25994 connection.cc:512] client connection to 127.5.240.194:46721 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:46721): Connection reset by peer (error 104) W0703 01:42:17.072914 25995 connection.cc:512] server connection from 127.5.240.194:60308 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:60308): Connection reset by peer (error 104) I0703 01:42:17.076529 25975 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:42:17.135272 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 25853 W0703 01:42:17.146440 24782 connection.cc:512] server connection from 127.5.240.195:42980 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.195:42980): Connection reset by peer (error 104) I0703 01:42:17.146687 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 25985 I0703 01:42:17.150310 26101 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:42:17.210541 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 25985 W0703 01:42:17.221078 24783 connection.cc:512] server connection from 127.5.240.196:38048 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.196:38048): Connection reset by peer (error 104) I0703 01:42:17.221254 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 26107 I0703 01:42:17.224939 26223 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:42:17.284463 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 26107 W0703 01:42:17.295497 24782 connection.cc:512] server connection from 127.5.240.197:43609 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.197:43609): Connection reset by peer (error 104) I0703 01:42:17.295714 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 26229 I0703 01:42:17.299500 26345 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:42:17.358655 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 26229 W0703 01:42:17.369261 24784 connection.cc:512] server connection from 127.5.240.198:56782 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.198:56782): Connection reset by peer (error 104) I0703 01:42:17.369771 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 26351 I0703 01:42:17.373972 26467 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:42:17.433460 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 26351 W0703 01:42:17.443784 24784 connection.cc:512] server connection from 127.5.240.199:36877 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.199:36877): Connection reset by peer (error 104) I0703 01:42:17.443841 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master pid 24774 I0703 01:42:17.444566 24830 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:42:17.542418 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master with pid 24774 [ OK ] RebalanceParamTest.Rebalance/0 (27468 ms) [ RUN ] RebalanceParamTest.Rebalance/1 I0703 01:42:17.586263 6083 test_util.cc:208] Using random seed: 1252376132 I0703 01:42:17.588011 6083 ts_itest-base.cc:113] Starting cluster with: I0703 01:42:17.588064 6083 ts_itest-base.cc:114] -------------- I0703 01:42:17.588090 6083 ts_itest-base.cc:115] 7 tablet servers I0703 01:42:17.588131 6083 ts_itest-base.cc:116] 1 replicas per TS I0703 01:42:17.588155 6083 ts_itest-base.cc:117] -------------- I0703 01:42:17.589300 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:47851 --webserver_interface=127.5.240.254 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --allow_unsafe_replication_factor --raft_prepare_replacement_before_eviction=true --tserver_unresponsive_timeout_ms=3000 --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:42:17.775039 28231 flags.cc:406] Enabled unsafe flag: --allow_unsafe_replication_factor=true W0703 01:42:17.776053 28231 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:42:17.776135 28231 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:42:17.778391 28231 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0703 01:42:17.778468 28231 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0703 01:42:17.778517 28231 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:42:17.778563 28231 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0703 01:42:17.778626 28231 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0703 01:42:17.786859 28231 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --allow_unsafe_replication_factor=true --ipki_ca_key_size=1024 --tserver_unresponsive_timeout_ms=3000 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:47851 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.5.240.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:42:17.792675 28231 master_main.cc:80] Initializing master server... I0703 01:42:17.793447 28231 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 222208us I0703 01:42:17.794574 28231 fs_manager.cc:261] Metadata directory not provided I0703 01:42:17.794656 28231 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:42:17.794917 28231 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:42:17.794982 28231 server_base.cc:433] Attempting to create new FS layout instead I0703 01:42:17.799228 28231 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/instance: uuid: "651f357379414cf8959c30179d420b62" format_stamp: "Formatted at 2018-07-03 08:42:17 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:42:17.800165 28231 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "651f357379414cf8959c30179d420b62" format_stamp: "Formatted at 2018-07-03 08:42:17 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:42:17.808612 28231 fs_manager.cc:501] Time spent creating directory manager: real 0.008s user 0.004s sys 0.003s I0703 01:42:17.809844 28231 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:42:17.810262 28231 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:42:17.813992 28231 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0703 01:42:17.814052 28231 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal uuid: "651f357379414cf8959c30179d420b62" format_stamp: "Formatted at 2018-07-03 08:42:17 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:42:17.814172 28231 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:42:17.843814 28231 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:42:17.844804 28231 master_main.cc:83] Starting Master server... I0703 01:42:17.881896 28231 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.254:47851 I0703 01:42:17.882344 28231 webserver.cc:175] Starting webserver on 127.5.240.254:0 I0703 01:42:17.882391 28231 webserver.cc:186] Document root disabled I0703 01:42:17.883636 28231 webserver.cc:313] Webserver started. Bound to: http://127.5.240.254:52890/ I0703 01:42:17.884218 28231 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb I0703 01:42:17.885243 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master as pid 28231 I0703 01:42:17.885962 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.5.240.193:0 --local_ip_for_outbound_sockets=127.5.240.193 --webserver_interface=127.5.240.193 --webserver_port=0 --tserver_master_addrs=127.5.240.254:47851 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0703 01:42:17.888074 28290 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:17.899966 28290 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 651f357379414cf8959c30179d420b62: Bootstrap starting. I0703 01:42:17.902096 28290 tablet_bootstrap.cc:578] T 00000000000000000000000000000000 P 651f357379414cf8959c30179d420b62: No blocks or log segments found. Creating new log. I0703 01:42:17.902863 28290 log.cc:525] T 00000000000000000000000000000000 P 651f357379414cf8959c30179d420b62: Log is configured to *not* fsync() on all Append() calls I0703 01:42:17.904642 28290 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 651f357379414cf8959c30179d420b62: No bootstrap required, opened a new log I0703 01:42:17.910881 28290 raft_consensus.cc:325] T 00000000000000000000000000000000 P 651f357379414cf8959c30179d420b62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "651f357379414cf8959c30179d420b62" member_type: VOTER } I0703 01:42:17.911072 28290 raft_consensus.cc:351] T 00000000000000000000000000000000 P 651f357379414cf8959c30179d420b62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:42:17.911131 28290 raft_consensus.cc:628] T 00000000000000000000000000000000 P 651f357379414cf8959c30179d420b62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 651f357379414cf8959c30179d420b62, State: Initialized, Role: FOLLOWER I0703 01:42:17.911592 28290 consensus_queue.cc:227] T 00000000000000000000000000000000 P 651f357379414cf8959c30179d420b62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "651f357379414cf8959c30179d420b62" member_type: VOTER } I0703 01:42:17.911748 28290 raft_consensus.cc:365] T 00000000000000000000000000000000 P 651f357379414cf8959c30179d420b62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:42:17.911830 28290 raft_consensus.cc:457] T 00000000000000000000000000000000 P 651f357379414cf8959c30179d420b62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:42:17.911916 28290 raft_consensus.cc:2730] T 00000000000000000000000000000000 P 651f357379414cf8959c30179d420b62: Snoozing failure detection for 1.750s (starting election) I0703 01:42:17.912006 28290 raft_consensus.cc:2782] T 00000000000000000000000000000000 P 651f357379414cf8959c30179d420b62 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:42:17.913347 28290 raft_consensus.cc:479] T 00000000000000000000000000000000 P 651f357379414cf8959c30179d420b62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "651f357379414cf8959c30179d420b62" member_type: VOTER } I0703 01:42:17.913730 28290 leader_election.cc:270] T 00000000000000000000000000000000 P 651f357379414cf8959c30179d420b62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:42:17.915213 28293 raft_consensus.cc:2730] T 00000000000000000000000000000000 P 651f357379414cf8959c30179d420b62: Snoozing failure detection for 1.748s (election complete) I0703 01:42:17.915474 28293 raft_consensus.cc:2537] T 00000000000000000000000000000000 P 651f357379414cf8959c30179d420b62 [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:42:17.921363 28293 raft_consensus.cc:593] T 00000000000000000000000000000000 P 651f357379414cf8959c30179d420b62 [term 1 LEADER]: Becoming Leader. State: Replica: 651f357379414cf8959c30179d420b62, State: Running, Role: LEADER I0703 01:42:17.921753 28293 consensus_queue.cc:204] T 00000000000000000000000000000000 P 651f357379414cf8959c30179d420b62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "651f357379414cf8959c30179d420b62" member_type: VOTER } I0703 01:42:17.924793 28290 sys_catalog.cc:337] T 00000000000000000000000000000000 P 651f357379414cf8959c30179d420b62 [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "651f357379414cf8959c30179d420b62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "651f357379414cf8959c30179d420b62" member_type: VOTER } } I0703 01:42:17.924686 28294 sys_catalog.cc:337] T 00000000000000000000000000000000 P 651f357379414cf8959c30179d420b62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "651f357379414cf8959c30179d420b62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "651f357379414cf8959c30179d420b62" member_type: VOTER } } I0703 01:42:17.925264 28290 sys_catalog.cc:340] T 00000000000000000000000000000000 P 651f357379414cf8959c30179d420b62 [sys.catalog]: This master's current role is: LEADER I0703 01:42:17.925413 28294 sys_catalog.cc:340] T 00000000000000000000000000000000 P 651f357379414cf8959c30179d420b62 [sys.catalog]: This master's current role is: LEADER I0703 01:42:17.925098 28295 sys_catalog.cc:337] T 00000000000000000000000000000000 P 651f357379414cf8959c30179d420b62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 651f357379414cf8959c30179d420b62. Latest consensus state: current_term: 1 leader_uuid: "651f357379414cf8959c30179d420b62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "651f357379414cf8959c30179d420b62" member_type: VOTER } } I0703 01:42:17.926460 28295 sys_catalog.cc:340] T 00000000000000000000000000000000 P 651f357379414cf8959c30179d420b62 [sys.catalog]: This master's current role is: LEADER I0703 01:42:17.928323 28290 sys_catalog.cc:424] T 00000000000000000000000000000000 P 651f357379414cf8959c30179d420b62 [sys.catalog]: configured and running, proceeding with master startup. I0703 01:42:17.929769 28231 master_main.cc:86] Master server successfully started. W0703 01:42:17.933961 28298 catalog_manager.cc:1064] T 00000000000000000000000000000000 P 651f357379414cf8959c30179d420b62: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0703 01:42:17.934068 28298 catalog_manager.cc:563] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0703 01:42:17.934212 28297 catalog_manager.cc:1006] Loading table and tablet metadata into memory... I0703 01:42:17.937940 28297 catalog_manager.cc:1017] Initializing Kudu internal certificate authority... I0703 01:42:17.970472 28297 catalog_manager.cc:904] Generated new certificate authority record I0703 01:42:17.972533 28297 catalog_manager.cc:1026] Loading token signing keys... I0703 01:42:17.990191 28297 catalog_manager.cc:4102] T 00000000000000000000000000000000 P 651f357379414cf8959c30179d420b62: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:42:18.064731 28291 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:42:18.065755 28291 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:42:18.067752 28291 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0703 01:42:18.067816 28291 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:42:18.067922 28291 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.193 I0703 01:42:18.075914 28291 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.5.240.193 --webserver_port=0 --tserver_master_addrs=127.5.240.254:47851 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.193 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:42:18.082597 28291 tablet_server_main.cc:85] Initializing tablet server... I0703 01:42:18.084137 28291 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 222708us I0703 01:42:18.084749 28291 fs_manager.cc:261] Metadata directory not provided I0703 01:42:18.084818 28291 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:42:18.085045 28291 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:42:18.085115 28291 server_base.cc:433] Attempting to create new FS layout instead I0703 01:42:18.088511 28291 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "5ee4739615124e308720b4aeddfe29cb" format_stamp: "Formatted at 2018-07-03 08:42:18 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:42:18.089041 28291 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "5ee4739615124e308720b4aeddfe29cb" format_stamp: "Formatted at 2018-07-03 08:42:18 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:42:18.096657 28291 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.003s sys 0.003s I0703 01:42:18.097864 28291 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:42:18.098237 28291 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:42:18.101760 28291 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0703 01:42:18.101814 28291 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal uuid: "5ee4739615124e308720b4aeddfe29cb" format_stamp: "Formatted at 2018-07-03 08:42:18 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:42:18.101943 28291 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:42:18.175335 28291 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:42:18.177040 28291 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:42:18.177402 28291 tablet_server_main.cc:90] Starting tablet server... I0703 01:42:18.273813 28291 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.193:54206 I0703 01:42:18.274302 28291 webserver.cc:175] Starting webserver on 127.5.240.193:0 I0703 01:42:18.274348 28291 webserver.cc:186] Document root disabled I0703 01:42:18.275544 28291 webserver.cc:313] Webserver started. Bound to: http://127.5.240.193:46058/ I0703 01:42:18.276080 28291 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb I0703 01:42:18.277762 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 28291 I0703 01:42:18.278571 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.5.240.194:0 --local_ip_for_outbound_sockets=127.5.240.194 --webserver_interface=127.5.240.194 --webserver_port=0 --tserver_master_addrs=127.5.240.254:47851 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0703 01:42:18.284461 28291 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:42:18.298933 28420 heartbeater.cc:346] Connected to a master server at 127.5.240.254:47851 I0703 01:42:18.299068 28420 heartbeater.cc:426] Registering TS with master... I0703 01:42:18.300246 28420 heartbeater.cc:475] Master 127.5.240.254:47851 requested a full tablet report, sending... I0703 01:42:18.301800 28257 ts_manager.cc:98] Registered new tserver with Master: 5ee4739615124e308720b4aeddfe29cb (127.5.240.193:54206) I0703 01:42:18.304311 28257 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.193:54456 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:42:18.447518 28424 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:42:18.448559 28424 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:42:18.450691 28424 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0703 01:42:18.450757 28424 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:42:18.450861 28424 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.194 I0703 01:42:18.458307 28424 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.194:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.5.240.194 --webserver_port=0 --tserver_master_addrs=127.5.240.254:47851 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.194 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:42:18.464985 28424 tablet_server_main.cc:85] Initializing tablet server... I0703 01:42:18.466498 28424 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 222708us I0703 01:42:18.467074 28424 fs_manager.cc:261] Metadata directory not provided I0703 01:42:18.467144 28424 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:42:18.467762 28424 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:42:18.467831 28424 server_base.cc:433] Attempting to create new FS layout instead I0703 01:42:18.471137 28424 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "31c252b54241470ba1115181c3672664" format_stamp: "Formatted at 2018-07-03 08:42:18 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:42:18.471694 28424 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "31c252b54241470ba1115181c3672664" format_stamp: "Formatted at 2018-07-03 08:42:18 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:42:18.479432 28424 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.002s sys 0.004s I0703 01:42:18.480667 28424 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:42:18.481034 28424 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:42:18.484136 28424 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0703 01:42:18.484187 28424 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal uuid: "31c252b54241470ba1115181c3672664" format_stamp: "Formatted at 2018-07-03 08:42:18 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:42:18.484314 28424 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:42:18.508635 28424 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:42:18.510565 28424 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:42:18.510888 28424 tablet_server_main.cc:90] Starting tablet server... I0703 01:42:18.595196 28424 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.194:38625 I0703 01:42:18.595650 28424 webserver.cc:175] Starting webserver on 127.5.240.194:0 I0703 01:42:18.595695 28424 webserver.cc:186] Document root disabled I0703 01:42:18.596909 28424 webserver.cc:313] Webserver started. Bound to: http://127.5.240.194:43657/ I0703 01:42:18.597435 28424 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb I0703 01:42:18.599735 28424 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:42:18.600656 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 28424 I0703 01:42:18.601275 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.5.240.195:0 --local_ip_for_outbound_sockets=127.5.240.195 --webserver_interface=127.5.240.195 --webserver_port=0 --tserver_master_addrs=127.5.240.254:47851 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0703 01:42:18.611591 28543 heartbeater.cc:346] Connected to a master server at 127.5.240.254:47851 I0703 01:42:18.611737 28543 heartbeater.cc:426] Registering TS with master... I0703 01:42:18.612118 28543 heartbeater.cc:475] Master 127.5.240.254:47851 requested a full tablet report, sending... I0703 01:42:18.613116 28257 ts_manager.cc:98] Registered new tserver with Master: 31c252b54241470ba1115181c3672664 (127.5.240.194:38625) I0703 01:42:18.614334 28257 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.194:55278 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:42:18.764029 28546 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:42:18.765053 28546 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:42:18.767030 28546 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0703 01:42:18.767094 28546 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:42:18.767208 28546 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.195 I0703 01:42:18.774566 28546 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.195:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.5.240.195 --webserver_port=0 --tserver_master_addrs=127.5.240.254:47851 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.195 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:42:18.781177 28546 tablet_server_main.cc:85] Initializing tablet server... I0703 01:42:18.782696 28546 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 222708us I0703 01:42:18.783255 28546 fs_manager.cc:261] Metadata directory not provided I0703 01:42:18.783325 28546 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:42:18.783552 28546 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:42:18.783638 28546 server_base.cc:433] Attempting to create new FS layout instead I0703 01:42:18.786922 28546 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "0ae780b86cc348d3a55bbaf5cf5a6217" format_stamp: "Formatted at 2018-07-03 08:42:18 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:42:18.787451 28546 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "0ae780b86cc348d3a55bbaf5cf5a6217" format_stamp: "Formatted at 2018-07-03 08:42:18 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:42:18.794813 28546 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.004s sys 0.002s I0703 01:42:18.796000 28546 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:42:18.796365 28546 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:42:18.799939 28546 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s I0703 01:42:18.799989 28546 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal uuid: "0ae780b86cc348d3a55bbaf5cf5a6217" format_stamp: "Formatted at 2018-07-03 08:42:18 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:42:18.800112 28546 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:42:18.826978 28546 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:42:18.828907 28546 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:42:18.829246 28546 tablet_server_main.cc:90] Starting tablet server... I0703 01:42:18.918727 28546 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.195:46219 I0703 01:42:18.919143 28546 webserver.cc:175] Starting webserver on 127.5.240.195:0 I0703 01:42:18.919189 28546 webserver.cc:186] Document root disabled I0703 01:42:18.919904 28546 webserver.cc:313] Webserver started. Bound to: http://127.5.240.195:43479/ I0703 01:42:18.920439 28546 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb I0703 01:42:18.922883 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 28546 I0703 01:42:18.923776 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.5.240.196:0 --local_ip_for_outbound_sockets=127.5.240.196 --webserver_interface=127.5.240.196 --webserver_port=0 --tserver_master_addrs=127.5.240.254:47851 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0703 01:42:18.928808 28546 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:42:18.934991 28665 heartbeater.cc:346] Connected to a master server at 127.5.240.254:47851 I0703 01:42:18.935160 28665 heartbeater.cc:426] Registering TS with master... I0703 01:42:18.935678 28665 heartbeater.cc:475] Master 127.5.240.254:47851 requested a full tablet report, sending... I0703 01:42:18.936730 28257 ts_manager.cc:98] Registered new tserver with Master: 0ae780b86cc348d3a55bbaf5cf5a6217 (127.5.240.195:46219) I0703 01:42:18.937548 28257 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.195:40937 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:42:19.089257 28668 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:42:19.090276 28668 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:42:19.092300 28668 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0703 01:42:19.092350 28668 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:42:19.092433 28668 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.196 I0703 01:42:19.099736 28668 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.196:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb --webserver_interface=127.5.240.196 --webserver_port=0 --tserver_master_addrs=127.5.240.254:47851 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.196 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:42:19.106252 28668 tablet_server_main.cc:85] Initializing tablet server... I0703 01:42:19.107786 28668 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 223208us I0703 01:42:19.108322 28668 fs_manager.cc:261] Metadata directory not provided I0703 01:42:19.108381 28668 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:42:19.108608 28668 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:42:19.108669 28668 server_base.cc:433] Attempting to create new FS layout instead I0703 01:42:19.112021 28668 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/instance: uuid: "35f0266742fc4707a8cc10c6437b84ed" format_stamp: "Formatted at 2018-07-03 08:42:19 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:42:19.112525 28668 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal/instance: uuid: "35f0266742fc4707a8cc10c6437b84ed" format_stamp: "Formatted at 2018-07-03 08:42:19 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:42:19.119724 28668 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s I0703 01:42:19.120944 28668 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:42:19.121289 28668 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:42:19.124847 28668 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0703 01:42:19.124892 28668 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal uuid: "35f0266742fc4707a8cc10c6437b84ed" format_stamp: "Formatted at 2018-07-03 08:42:19 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:42:19.125006 28668 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:42:19.147845 28668 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:42:19.149705 28668 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:42:19.150040 28668 tablet_server_main.cc:90] Starting tablet server... I0703 01:42:19.236138 28668 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.196:50368 I0703 01:42:19.236524 28668 webserver.cc:175] Starting webserver on 127.5.240.196:0 I0703 01:42:19.236567 28668 webserver.cc:186] Document root disabled I0703 01:42:19.237762 28668 webserver.cc:313] Webserver started. Bound to: http://127.5.240.196:37872/ I0703 01:42:19.238286 28668 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb I0703 01:42:19.240154 28668 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:42:19.242120 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 28668 I0703 01:42:19.242980 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data --block_manager=log --rpc_bind_addresses=127.5.240.197:0 --local_ip_for_outbound_sockets=127.5.240.197 --webserver_interface=127.5.240.197 --webserver_port=0 --tserver_master_addrs=127.5.240.254:47851 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0703 01:42:19.253690 28787 heartbeater.cc:346] Connected to a master server at 127.5.240.254:47851 I0703 01:42:19.253872 28787 heartbeater.cc:426] Registering TS with master... I0703 01:42:19.254364 28787 heartbeater.cc:475] Master 127.5.240.254:47851 requested a full tablet report, sending... I0703 01:42:19.255439 28257 ts_manager.cc:98] Registered new tserver with Master: 35f0266742fc4707a8cc10c6437b84ed (127.5.240.196:50368) I0703 01:42:19.256716 28257 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.196:49683 I0703 01:42:19.306447 28420 heartbeater.cc:467] Master 127.5.240.254:47851 was elected leader, sending a full tablet report... WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:42:19.409447 28790 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:42:19.410464 28790 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:42:19.412540 28790 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0703 01:42:19.412606 28790 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:42:19.412711 28790 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.197 I0703 01:42:19.420071 28790 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.197:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb --webserver_interface=127.5.240.197 --webserver_port=0 --tserver_master_addrs=127.5.240.254:47851 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.197 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:42:19.426762 28790 tablet_server_main.cc:85] Initializing tablet server... I0703 01:42:19.428280 28790 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 223208us I0703 01:42:19.428850 28790 fs_manager.cc:261] Metadata directory not provided I0703 01:42:19.428918 28790 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:42:19.429147 28790 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:42:19.429214 28790 server_base.cc:433] Attempting to create new FS layout instead I0703 01:42:19.432477 28790 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/instance: uuid: "e98d29f36f2347f29170e987cb83ced4" format_stamp: "Formatted at 2018-07-03 08:42:19 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:42:19.433059 28790 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal/instance: uuid: "e98d29f36f2347f29170e987cb83ced4" format_stamp: "Formatted at 2018-07-03 08:42:19 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:42:19.440420 28790 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.004s sys 0.002s I0703 01:42:19.441659 28790 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:42:19.442109 28790 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:42:19.445732 28790 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0703 01:42:19.445791 28790 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal uuid: "e98d29f36f2347f29170e987cb83ced4" format_stamp: "Formatted at 2018-07-03 08:42:19 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:42:19.445922 28790 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:42:19.510957 28790 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:42:19.512373 28790 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:42:19.512691 28790 tablet_server_main.cc:90] Starting tablet server... I0703 01:42:19.597784 28790 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.197:41466 I0703 01:42:19.598202 28790 webserver.cc:175] Starting webserver on 127.5.240.197:0 I0703 01:42:19.598248 28790 webserver.cc:186] Document root disabled I0703 01:42:19.599447 28790 webserver.cc:313] Webserver started. Bound to: http://127.5.240.197:33425/ I0703 01:42:19.599994 28790 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb I0703 01:42:19.601893 28790 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:42:19.604785 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 28790 I0703 01:42:19.605649 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data --block_manager=log --rpc_bind_addresses=127.5.240.198:0 --local_ip_for_outbound_sockets=127.5.240.198 --webserver_interface=127.5.240.198 --webserver_port=0 --tserver_master_addrs=127.5.240.254:47851 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0703 01:42:19.615638 28909 heartbeater.cc:346] Connected to a master server at 127.5.240.254:47851 I0703 01:42:19.615784 28909 heartbeater.cc:426] Registering TS with master... I0703 01:42:19.616235 28909 heartbeater.cc:475] Master 127.5.240.254:47851 requested a full tablet report, sending... I0703 01:42:19.616255 28543 heartbeater.cc:467] Master 127.5.240.254:47851 was elected leader, sending a full tablet report... I0703 01:42:19.617365 28256 ts_manager.cc:98] Registered new tserver with Master: e98d29f36f2347f29170e987cb83ced4 (127.5.240.197:41466) I0703 01:42:19.618391 28256 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.197:49372 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:42:19.769377 28912 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:42:19.770367 28912 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:42:19.772338 28912 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0703 01:42:19.772387 28912 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:42:19.772469 28912 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.198 I0703 01:42:19.779803 28912 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.198:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data/info.pb --webserver_interface=127.5.240.198 --webserver_port=0 --tserver_master_addrs=127.5.240.254:47851 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.198 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:42:19.786345 28912 tablet_server_main.cc:85] Initializing tablet server... I0703 01:42:19.787866 28912 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 223208us I0703 01:42:19.788396 28912 fs_manager.cc:261] Metadata directory not provided I0703 01:42:19.788455 28912 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:42:19.788689 28912 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:42:19.788750 28912 server_base.cc:433] Attempting to create new FS layout instead I0703 01:42:19.791940 28912 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data/instance: uuid: "575620f4889c4704b7a7c00da34b6a53" format_stamp: "Formatted at 2018-07-03 08:42:19 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:42:19.792487 28912 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/wal/instance: uuid: "575620f4889c4704b7a7c00da34b6a53" format_stamp: "Formatted at 2018-07-03 08:42:19 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:42:19.799794 28912 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s I0703 01:42:19.800997 28912 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:42:19.801374 28912 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:42:19.805042 28912 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0703 01:42:19.805096 28912 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/wal uuid: "575620f4889c4704b7a7c00da34b6a53" format_stamp: "Formatted at 2018-07-03 08:42:19 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:42:19.805230 28912 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:42:19.888691 28912 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:42:19.890622 28912 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:42:19.890970 28912 tablet_server_main.cc:90] Starting tablet server... I0703 01:42:19.939749 28665 heartbeater.cc:467] Master 127.5.240.254:47851 was elected leader, sending a full tablet report... I0703 01:42:19.976652 28912 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.198:39152 I0703 01:42:19.977083 28912 webserver.cc:175] Starting webserver on 127.5.240.198:0 I0703 01:42:19.977129 28912 webserver.cc:186] Document root disabled I0703 01:42:19.977890 28912 webserver.cc:313] Webserver started. Bound to: http://127.5.240.198:40981/ I0703 01:42:19.978435 28912 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data/info.pb I0703 01:42:19.980839 28912 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:42:19.988250 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 28912 I0703 01:42:19.988868 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data --block_manager=log --rpc_bind_addresses=127.5.240.199:0 --local_ip_for_outbound_sockets=127.5.240.199 --webserver_interface=127.5.240.199 --webserver_port=0 --tserver_master_addrs=127.5.240.254:47851 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0703 01:42:19.992539 29031 heartbeater.cc:346] Connected to a master server at 127.5.240.254:47851 I0703 01:42:19.992699 29031 heartbeater.cc:426] Registering TS with master... I0703 01:42:19.993120 29031 heartbeater.cc:475] Master 127.5.240.254:47851 requested a full tablet report, sending... I0703 01:42:19.994081 28256 ts_manager.cc:98] Registered new tserver with Master: 575620f4889c4704b7a7c00da34b6a53 (127.5.240.198:39152) I0703 01:42:19.994946 28256 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.198:38227 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:42:20.150858 29034 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:42:20.151901 29034 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:42:20.153990 29034 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0703 01:42:20.154057 29034 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:42:20.154155 29034 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.199 I0703 01:42:20.161378 29034 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.199:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data/info.pb --webserver_interface=127.5.240.199 --webserver_port=0 --tserver_master_addrs=127.5.240.254:47851 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.199 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:42:20.168093 29034 tablet_server_main.cc:85] Initializing tablet server... I0703 01:42:20.169628 29034 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 223708us I0703 01:42:20.170193 29034 fs_manager.cc:261] Metadata directory not provided I0703 01:42:20.170259 29034 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:42:20.170877 29034 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:42:20.170948 29034 server_base.cc:433] Attempting to create new FS layout instead I0703 01:42:20.174607 29034 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data/instance: uuid: "6505fcf874c440788e377dac9aa99209" format_stamp: "Formatted at 2018-07-03 08:42:20 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:42:20.175283 29034 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/wal/instance: uuid: "6505fcf874c440788e377dac9aa99209" format_stamp: "Formatted at 2018-07-03 08:42:20 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:42:20.183034 29034 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s I0703 01:42:20.184242 29034 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:42:20.184639 29034 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:42:20.187832 29034 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0703 01:42:20.187885 29034 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/wal uuid: "6505fcf874c440788e377dac9aa99209" format_stamp: "Formatted at 2018-07-03 08:42:20 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:42:20.188016 29034 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:42:20.237720 29034 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:42:20.239636 29034 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:42:20.239943 29034 tablet_server_main.cc:90] Starting tablet server... I0703 01:42:20.259008 28787 heartbeater.cc:467] Master 127.5.240.254:47851 was elected leader, sending a full tablet report... I0703 01:42:20.325709 29034 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.199:33903 I0703 01:42:20.326128 29034 webserver.cc:175] Starting webserver on 127.5.240.199:0 I0703 01:42:20.326172 29034 webserver.cc:186] Document root disabled I0703 01:42:20.326894 29034 webserver.cc:313] Webserver started. Bound to: http://127.5.240.199:47396/ I0703 01:42:20.327414 29034 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data/info.pb I0703 01:42:20.331229 29034 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:42:20.337790 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 29034 I0703 01:42:20.340924 29153 heartbeater.cc:346] Connected to a master server at 127.5.240.254:47851 I0703 01:42:20.341100 29153 heartbeater.cc:426] Registering TS with master... I0703 01:42:20.341642 29153 heartbeater.cc:475] Master 127.5.240.254:47851 requested a full tablet report, sending... I0703 01:42:20.342689 28256 ts_manager.cc:98] Registered new tserver with Master: 6505fcf874c440788e377dac9aa99209 (127.5.240.199:33903) I0703 01:42:20.343736 28256 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.199:34127 I0703 01:42:20.344244 6083 external_mini_cluster.cc:490] 7 TS(s) registered with all masters I0703 01:42:20.355116 28257 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.5.240.254:44777: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 1 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0703 01:42:20.374258 28480 tablet_service.cc:759] Processing CreateTablet for tablet 7f2afbb6827246e1be0babd14a9d3cbd (table=TestTable [id=c6cd7be86a5540f888ec47e64a82f275]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:42:20.375075 28480 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 7f2afbb6827246e1be0babd14a9d3cbd. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:20.385751 28480 ts_tablet_manager.cc:1173] T 7f2afbb6827246e1be0babd14a9d3cbd P 31c252b54241470ba1115181c3672664: Registered tablet (data state: TABLET_DATA_READY) I0703 01:42:20.386461 29169 ts_tablet_manager.cc:1028] T 7f2afbb6827246e1be0babd14a9d3cbd P 31c252b54241470ba1115181c3672664: Bootstrapping tablet I0703 01:42:20.387181 29169 tablet_bootstrap.cc:427] T 7f2afbb6827246e1be0babd14a9d3cbd P 31c252b54241470ba1115181c3672664: Bootstrap starting. I0703 01:42:20.389719 29169 tablet_bootstrap.cc:578] T 7f2afbb6827246e1be0babd14a9d3cbd P 31c252b54241470ba1115181c3672664: No blocks or log segments found. Creating new log. I0703 01:42:20.390435 29169 log.cc:525] T 7f2afbb6827246e1be0babd14a9d3cbd P 31c252b54241470ba1115181c3672664: Log is configured to *not* fsync() on all Append() calls I0703 01:42:20.392441 29169 tablet_bootstrap.cc:427] T 7f2afbb6827246e1be0babd14a9d3cbd P 31c252b54241470ba1115181c3672664: No bootstrap required, opened a new log I0703 01:42:20.392630 29169 ts_tablet_manager.cc:1045] T 7f2afbb6827246e1be0babd14a9d3cbd P 31c252b54241470ba1115181c3672664: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s I0703 01:42:20.398648 29169 raft_consensus.cc:325] T 7f2afbb6827246e1be0babd14a9d3cbd P 31c252b54241470ba1115181c3672664 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } } I0703 01:42:20.398891 29169 raft_consensus.cc:351] T 7f2afbb6827246e1be0babd14a9d3cbd P 31c252b54241470ba1115181c3672664 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:42:20.398952 29169 raft_consensus.cc:628] T 7f2afbb6827246e1be0babd14a9d3cbd P 31c252b54241470ba1115181c3672664 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31c252b54241470ba1115181c3672664, State: Initialized, Role: FOLLOWER I0703 01:42:20.399344 29169 consensus_queue.cc:227] T 7f2afbb6827246e1be0babd14a9d3cbd P 31c252b54241470ba1115181c3672664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } } I0703 01:42:20.399514 29169 raft_consensus.cc:365] T 7f2afbb6827246e1be0babd14a9d3cbd P 31c252b54241470ba1115181c3672664 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:42:20.399588 29169 raft_consensus.cc:457] T 7f2afbb6827246e1be0babd14a9d3cbd P 31c252b54241470ba1115181c3672664 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:42:20.399682 29169 raft_consensus.cc:2730] T 7f2afbb6827246e1be0babd14a9d3cbd P 31c252b54241470ba1115181c3672664: Snoozing failure detection for 1.854s (starting election) I0703 01:42:20.399754 29169 raft_consensus.cc:2782] T 7f2afbb6827246e1be0babd14a9d3cbd P 31c252b54241470ba1115181c3672664 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:42:20.401247 29169 raft_consensus.cc:479] T 7f2afbb6827246e1be0babd14a9d3cbd P 31c252b54241470ba1115181c3672664 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } } I0703 01:42:20.401573 29169 leader_election.cc:270] T 7f2afbb6827246e1be0babd14a9d3cbd P 31c252b54241470ba1115181c3672664 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:42:20.402812 29171 raft_consensus.cc:2730] T 7f2afbb6827246e1be0babd14a9d3cbd P 31c252b54241470ba1115181c3672664: Snoozing failure detection for 1.516s (election complete) I0703 01:42:20.403223 29171 raft_consensus.cc:2537] T 7f2afbb6827246e1be0babd14a9d3cbd P 31c252b54241470ba1115181c3672664 [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:42:20.405489 29169 ts_tablet_manager.cc:1073] T 7f2afbb6827246e1be0babd14a9d3cbd P 31c252b54241470ba1115181c3672664: Time spent starting tablet: real 0.012s user 0.008s sys 0.003s I0703 01:42:20.405565 29171 raft_consensus.cc:593] T 7f2afbb6827246e1be0babd14a9d3cbd P 31c252b54241470ba1115181c3672664 [term 1 LEADER]: Becoming Leader. State: Replica: 31c252b54241470ba1115181c3672664, State: Running, Role: LEADER I0703 01:42:20.405949 29171 consensus_queue.cc:204] T 7f2afbb6827246e1be0babd14a9d3cbd P 31c252b54241470ba1115181c3672664 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } } I0703 01:42:20.410008 28257 catalog_manager.cc:3798] T 7f2afbb6827246e1be0babd14a9d3cbd P 31c252b54241470ba1115181c3672664 reported cstate change: term changed from 0 to 1, leader changed from to 31c252b54241470ba1115181c3672664 (127.5.240.194). New cstate: current_term: 1 leader_uuid: "31c252b54241470ba1115181c3672664" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } health_report { overall_health: HEALTHY } } } I0703 01:42:20.416831 6083 external_mini_cluster.cc:490] 7 TS(s) registered with all masters I0703 01:42:20.417891 6083 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 5ee4739615124e308720b4aeddfe29cb to finish bootstrapping I0703 01:42:20.424268 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 31c252b54241470ba1115181c3672664 to finish bootstrapping I0703 01:42:20.428980 6083 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 0ae780b86cc348d3a55bbaf5cf5a6217 to finish bootstrapping I0703 01:42:20.435307 6083 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 35f0266742fc4707a8cc10c6437b84ed to finish bootstrapping I0703 01:42:20.441568 6083 ts_itest-base.cc:233] Waiting for 0 tablets on tserver e98d29f36f2347f29170e987cb83ced4 to finish bootstrapping I0703 01:42:20.447062 6083 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 575620f4889c4704b7a7c00da34b6a53 to finish bootstrapping I0703 01:42:20.453030 6083 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 6505fcf874c440788e377dac9aa99209 to finish bootstrapping I0703 01:42:20.460441 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 28546 I0703 01:42:20.461006 28662 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:42:20.499689 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 28546 W0703 01:42:20.507372 28239 connection.cc:512] server connection from 127.5.240.195:40937 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.195:40937): Connection reset by peer (error 104) I0703 01:42:20.507491 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 28668 I0703 01:42:20.509222 28784 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:42:20.552295 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 28668 W0703 01:42:20.559998 28240 connection.cc:512] server connection from 127.5.240.196:49683 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.196:49683): Connection reset by peer (error 104) I0703 01:42:20.560341 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 28790 I0703 01:42:20.561107 28906 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:42:20.598784 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 28790 W0703 01:42:20.601001 28544 tablet.cc:1678] T 7f2afbb6827246e1be0babd14a9d3cbd P 31c252b54241470ba1115181c3672664: Can't schedule compaction. Clean time has not been advanced past its initial value. W0703 01:42:20.607336 28239 connection.cc:512] server connection from 127.5.240.197:49372 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.197:49372): Connection reset by peer (error 104) I0703 01:42:20.607427 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 28912 I0703 01:42:20.608180 29028 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:42:20.645934 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 28912 W0703 01:42:20.653038 28238 connection.cc:512] server connection from 127.5.240.198:38227 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.198:38227): Connection reset by peer (error 104) I0703 01:42:20.653367 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 29034 I0703 01:42:20.654106 29150 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:42:20.692644 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 29034 W0703 01:42:20.699800 28238 connection.cc:512] server connection from 127.5.240.199:34127 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.199:34127): Connection reset by peer (error 104) I0703 01:42:24.450814 28257 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.5.240.254:44777: name: "rebalance_test_table_0" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 1 split_rows_range_bounds { } partition_schema { hash_bucket_schemas { columns { name: "key" } num_buckets: 3 seed: 0 } } I0703 01:42:24.464267 28356 tablet_service.cc:759] Processing CreateTablet for tablet 45b3833ac0664a8d9889ab14b3d0dd69 (table=rebalance_test_table_0 [id=2fa00f90e0e84298b5fa673ff8ab7e37]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I0703 01:42:24.464334 28355 tablet_service.cc:759] Processing CreateTablet for tablet ca3c350378324abcb8b68a2aaa4ba5a7 (table=rebalance_test_table_0 [id=2fa00f90e0e84298b5fa673ff8ab7e37]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I0703 01:42:24.464334 28357 tablet_service.cc:759] Processing CreateTablet for tablet b4acd291b108434da9b8791c61fd1dc2 (table=rebalance_test_table_0 [id=2fa00f90e0e84298b5fa673ff8ab7e37]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I0703 01:42:24.465539 28355 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet ca3c350378324abcb8b68a2aaa4ba5a7. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:24.465864 28357 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet b4acd291b108434da9b8791c61fd1dc2. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:24.466132 28356 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 45b3833ac0664a8d9889ab14b3d0dd69. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:24.479454 28355 ts_tablet_manager.cc:1173] T ca3c350378324abcb8b68a2aaa4ba5a7 P 5ee4739615124e308720b4aeddfe29cb: Registered tablet (data state: TABLET_DATA_READY) I0703 01:42:24.479993 28357 ts_tablet_manager.cc:1173] T b4acd291b108434da9b8791c61fd1dc2 P 5ee4739615124e308720b4aeddfe29cb: Registered tablet (data state: TABLET_DATA_READY) I0703 01:42:24.480201 28356 ts_tablet_manager.cc:1173] T 45b3833ac0664a8d9889ab14b3d0dd69 P 5ee4739615124e308720b4aeddfe29cb: Registered tablet (data state: TABLET_DATA_READY) I0703 01:42:24.480484 29192 ts_tablet_manager.cc:1028] T ca3c350378324abcb8b68a2aaa4ba5a7 P 5ee4739615124e308720b4aeddfe29cb: Bootstrapping tablet I0703 01:42:24.480813 29192 tablet_bootstrap.cc:427] T ca3c350378324abcb8b68a2aaa4ba5a7 P 5ee4739615124e308720b4aeddfe29cb: Bootstrap starting. I0703 01:42:24.483253 29192 tablet_bootstrap.cc:578] T ca3c350378324abcb8b68a2aaa4ba5a7 P 5ee4739615124e308720b4aeddfe29cb: No blocks or log segments found. Creating new log. I0703 01:42:24.483978 29192 log.cc:525] T ca3c350378324abcb8b68a2aaa4ba5a7 P 5ee4739615124e308720b4aeddfe29cb: Log is configured to *not* fsync() on all Append() calls I0703 01:42:24.485433 29192 tablet_bootstrap.cc:427] T ca3c350378324abcb8b68a2aaa4ba5a7 P 5ee4739615124e308720b4aeddfe29cb: No bootstrap required, opened a new log I0703 01:42:24.485608 29192 ts_tablet_manager.cc:1045] T ca3c350378324abcb8b68a2aaa4ba5a7 P 5ee4739615124e308720b4aeddfe29cb: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.002s I0703 01:42:24.491427 29192 raft_consensus.cc:325] T ca3c350378324abcb8b68a2aaa4ba5a7 P 5ee4739615124e308720b4aeddfe29cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } } I0703 01:42:24.491691 29192 raft_consensus.cc:351] T ca3c350378324abcb8b68a2aaa4ba5a7 P 5ee4739615124e308720b4aeddfe29cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:42:24.491746 29192 raft_consensus.cc:628] T ca3c350378324abcb8b68a2aaa4ba5a7 P 5ee4739615124e308720b4aeddfe29cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ee4739615124e308720b4aeddfe29cb, State: Initialized, Role: FOLLOWER I0703 01:42:24.492130 29192 consensus_queue.cc:227] T ca3c350378324abcb8b68a2aaa4ba5a7 P 5ee4739615124e308720b4aeddfe29cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } } I0703 01:42:24.492287 29192 raft_consensus.cc:365] T ca3c350378324abcb8b68a2aaa4ba5a7 P 5ee4739615124e308720b4aeddfe29cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:42:24.492344 29192 raft_consensus.cc:457] T ca3c350378324abcb8b68a2aaa4ba5a7 P 5ee4739615124e308720b4aeddfe29cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:42:24.492408 29192 raft_consensus.cc:2730] T ca3c350378324abcb8b68a2aaa4ba5a7 P 5ee4739615124e308720b4aeddfe29cb: Snoozing failure detection for 1.920s (starting election) I0703 01:42:24.492468 29192 raft_consensus.cc:2782] T ca3c350378324abcb8b68a2aaa4ba5a7 P 5ee4739615124e308720b4aeddfe29cb [term 0 FOLLOWER]: Advancing to term 1 I0703 01:42:24.493765 29192 raft_consensus.cc:479] T ca3c350378324abcb8b68a2aaa4ba5a7 P 5ee4739615124e308720b4aeddfe29cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } } I0703 01:42:24.494074 29192 leader_election.cc:270] T ca3c350378324abcb8b68a2aaa4ba5a7 P 5ee4739615124e308720b4aeddfe29cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:42:24.495235 29194 raft_consensus.cc:2730] T ca3c350378324abcb8b68a2aaa4ba5a7 P 5ee4739615124e308720b4aeddfe29cb: Snoozing failure detection for 1.872s (election complete) I0703 01:42:24.495424 29194 raft_consensus.cc:2537] T ca3c350378324abcb8b68a2aaa4ba5a7 P 5ee4739615124e308720b4aeddfe29cb [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:42:24.497304 29192 ts_tablet_manager.cc:1073] T ca3c350378324abcb8b68a2aaa4ba5a7 P 5ee4739615124e308720b4aeddfe29cb: Time spent starting tablet: real 0.011s user 0.009s sys 0.001s I0703 01:42:24.497371 29194 raft_consensus.cc:593] T ca3c350378324abcb8b68a2aaa4ba5a7 P 5ee4739615124e308720b4aeddfe29cb [term 1 LEADER]: Becoming Leader. State: Replica: 5ee4739615124e308720b4aeddfe29cb, State: Running, Role: LEADER I0703 01:42:24.497541 29192 ts_tablet_manager.cc:1028] T b4acd291b108434da9b8791c61fd1dc2 P 5ee4739615124e308720b4aeddfe29cb: Bootstrapping tablet I0703 01:42:24.497748 29192 tablet_bootstrap.cc:427] T b4acd291b108434da9b8791c61fd1dc2 P 5ee4739615124e308720b4aeddfe29cb: Bootstrap starting. I0703 01:42:24.497853 29194 consensus_queue.cc:204] T ca3c350378324abcb8b68a2aaa4ba5a7 P 5ee4739615124e308720b4aeddfe29cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } } I0703 01:42:24.498965 29192 tablet_bootstrap.cc:578] T b4acd291b108434da9b8791c61fd1dc2 P 5ee4739615124e308720b4aeddfe29cb: No blocks or log segments found. Creating new log. I0703 01:42:24.502961 28257 catalog_manager.cc:3798] T ca3c350378324abcb8b68a2aaa4ba5a7 P 5ee4739615124e308720b4aeddfe29cb reported cstate change: term changed from 0 to 1, leader changed from to 5ee4739615124e308720b4aeddfe29cb (127.5.240.193). New cstate: current_term: 1 leader_uuid: "5ee4739615124e308720b4aeddfe29cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } health_report { overall_health: HEALTHY } } } I0703 01:42:24.503898 29192 tablet_bootstrap.cc:427] T b4acd291b108434da9b8791c61fd1dc2 P 5ee4739615124e308720b4aeddfe29cb: No bootstrap required, opened a new log I0703 01:42:24.504006 29192 ts_tablet_manager.cc:1045] T b4acd291b108434da9b8791c61fd1dc2 P 5ee4739615124e308720b4aeddfe29cb: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s I0703 01:42:24.504719 29192 raft_consensus.cc:325] T b4acd291b108434da9b8791c61fd1dc2 P 5ee4739615124e308720b4aeddfe29cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } } I0703 01:42:24.504814 29192 raft_consensus.cc:351] T b4acd291b108434da9b8791c61fd1dc2 P 5ee4739615124e308720b4aeddfe29cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:42:24.504858 29192 raft_consensus.cc:628] T b4acd291b108434da9b8791c61fd1dc2 P 5ee4739615124e308720b4aeddfe29cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ee4739615124e308720b4aeddfe29cb, State: Initialized, Role: FOLLOWER I0703 01:42:24.505010 29192 consensus_queue.cc:227] T b4acd291b108434da9b8791c61fd1dc2 P 5ee4739615124e308720b4aeddfe29cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } } I0703 01:42:24.505146 29192 raft_consensus.cc:365] T b4acd291b108434da9b8791c61fd1dc2 P 5ee4739615124e308720b4aeddfe29cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:42:24.505200 29192 raft_consensus.cc:457] T b4acd291b108434da9b8791c61fd1dc2 P 5ee4739615124e308720b4aeddfe29cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:42:24.505262 29192 raft_consensus.cc:2730] T b4acd291b108434da9b8791c61fd1dc2 P 5ee4739615124e308720b4aeddfe29cb: Snoozing failure detection for 2.154s (starting election) I0703 01:42:24.505329 29192 raft_consensus.cc:2782] T b4acd291b108434da9b8791c61fd1dc2 P 5ee4739615124e308720b4aeddfe29cb [term 0 FOLLOWER]: Advancing to term 1 I0703 01:42:24.507596 29192 raft_consensus.cc:479] T b4acd291b108434da9b8791c61fd1dc2 P 5ee4739615124e308720b4aeddfe29cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } } I0703 01:42:24.507794 29192 leader_election.cc:270] T b4acd291b108434da9b8791c61fd1dc2 P 5ee4739615124e308720b4aeddfe29cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:42:24.507979 29194 raft_consensus.cc:2730] T b4acd291b108434da9b8791c61fd1dc2 P 5ee4739615124e308720b4aeddfe29cb: Snoozing failure detection for 2.105s (election complete) I0703 01:42:24.508064 29194 raft_consensus.cc:2537] T b4acd291b108434da9b8791c61fd1dc2 P 5ee4739615124e308720b4aeddfe29cb [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:42:24.508225 29194 raft_consensus.cc:593] T b4acd291b108434da9b8791c61fd1dc2 P 5ee4739615124e308720b4aeddfe29cb [term 1 LEADER]: Becoming Leader. State: Replica: 5ee4739615124e308720b4aeddfe29cb, State: Running, Role: LEADER I0703 01:42:24.508306 29192 ts_tablet_manager.cc:1073] T b4acd291b108434da9b8791c61fd1dc2 P 5ee4739615124e308720b4aeddfe29cb: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s I0703 01:42:24.508390 29194 consensus_queue.cc:204] T b4acd291b108434da9b8791c61fd1dc2 P 5ee4739615124e308720b4aeddfe29cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } } I0703 01:42:24.508471 29192 ts_tablet_manager.cc:1028] T 45b3833ac0664a8d9889ab14b3d0dd69 P 5ee4739615124e308720b4aeddfe29cb: Bootstrapping tablet I0703 01:42:24.508677 29192 tablet_bootstrap.cc:427] T 45b3833ac0664a8d9889ab14b3d0dd69 P 5ee4739615124e308720b4aeddfe29cb: Bootstrap starting. I0703 01:42:24.510622 29192 tablet_bootstrap.cc:578] T 45b3833ac0664a8d9889ab14b3d0dd69 P 5ee4739615124e308720b4aeddfe29cb: No blocks or log segments found. Creating new log. I0703 01:42:24.511317 28257 catalog_manager.cc:3798] T b4acd291b108434da9b8791c61fd1dc2 P 5ee4739615124e308720b4aeddfe29cb reported cstate change: term changed from 0 to 1, leader changed from to 5ee4739615124e308720b4aeddfe29cb (127.5.240.193). New cstate: current_term: 1 leader_uuid: "5ee4739615124e308720b4aeddfe29cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } health_report { overall_health: HEALTHY } } } I0703 01:42:24.512615 29192 tablet_bootstrap.cc:427] T 45b3833ac0664a8d9889ab14b3d0dd69 P 5ee4739615124e308720b4aeddfe29cb: No bootstrap required, opened a new log I0703 01:42:24.512693 29192 ts_tablet_manager.cc:1045] T 45b3833ac0664a8d9889ab14b3d0dd69 P 5ee4739615124e308720b4aeddfe29cb: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s I0703 01:42:24.513105 29192 raft_consensus.cc:325] T 45b3833ac0664a8d9889ab14b3d0dd69 P 5ee4739615124e308720b4aeddfe29cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } } I0703 01:42:24.513222 29192 raft_consensus.cc:351] T 45b3833ac0664a8d9889ab14b3d0dd69 P 5ee4739615124e308720b4aeddfe29cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:42:24.513278 29192 raft_consensus.cc:628] T 45b3833ac0664a8d9889ab14b3d0dd69 P 5ee4739615124e308720b4aeddfe29cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ee4739615124e308720b4aeddfe29cb, State: Initialized, Role: FOLLOWER I0703 01:42:24.513453 29192 consensus_queue.cc:227] T 45b3833ac0664a8d9889ab14b3d0dd69 P 5ee4739615124e308720b4aeddfe29cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } } I0703 01:42:24.513732 29192 raft_consensus.cc:365] T 45b3833ac0664a8d9889ab14b3d0dd69 P 5ee4739615124e308720b4aeddfe29cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:42:24.513820 29192 raft_consensus.cc:457] T 45b3833ac0664a8d9889ab14b3d0dd69 P 5ee4739615124e308720b4aeddfe29cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:42:24.513872 29192 raft_consensus.cc:2730] T 45b3833ac0664a8d9889ab14b3d0dd69 P 5ee4739615124e308720b4aeddfe29cb: Snoozing failure detection for 1.599s (starting election) I0703 01:42:24.513914 29192 raft_consensus.cc:2782] T 45b3833ac0664a8d9889ab14b3d0dd69 P 5ee4739615124e308720b4aeddfe29cb [term 0 FOLLOWER]: Advancing to term 1 I0703 01:42:24.515095 29192 raft_consensus.cc:479] T 45b3833ac0664a8d9889ab14b3d0dd69 P 5ee4739615124e308720b4aeddfe29cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } } I0703 01:42:24.515226 29192 leader_election.cc:270] T 45b3833ac0664a8d9889ab14b3d0dd69 P 5ee4739615124e308720b4aeddfe29cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:42:24.515430 29196 raft_consensus.cc:2730] T 45b3833ac0664a8d9889ab14b3d0dd69 P 5ee4739615124e308720b4aeddfe29cb: Snoozing failure detection for 1.966s (election complete) I0703 01:42:24.515604 29196 raft_consensus.cc:2537] T 45b3833ac0664a8d9889ab14b3d0dd69 P 5ee4739615124e308720b4aeddfe29cb [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:42:24.515753 29192 ts_tablet_manager.cc:1073] T 45b3833ac0664a8d9889ab14b3d0dd69 P 5ee4739615124e308720b4aeddfe29cb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s I0703 01:42:24.515863 29196 raft_consensus.cc:593] T 45b3833ac0664a8d9889ab14b3d0dd69 P 5ee4739615124e308720b4aeddfe29cb [term 1 LEADER]: Becoming Leader. State: Replica: 5ee4739615124e308720b4aeddfe29cb, State: Running, Role: LEADER I0703 01:42:24.516057 29196 consensus_queue.cc:204] T 45b3833ac0664a8d9889ab14b3d0dd69 P 5ee4739615124e308720b4aeddfe29cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } } I0703 01:42:24.517972 28257 catalog_manager.cc:3798] T 45b3833ac0664a8d9889ab14b3d0dd69 P 5ee4739615124e308720b4aeddfe29cb reported cstate change: term changed from 0 to 1, leader changed from to 5ee4739615124e308720b4aeddfe29cb (127.5.240.193). New cstate: current_term: 1 leader_uuid: "5ee4739615124e308720b4aeddfe29cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } health_report { overall_health: HEALTHY } } } W0703 01:42:24.538413 28421 tablet.cc:1678] T 45b3833ac0664a8d9889ab14b3d0dd69 P 5ee4739615124e308720b4aeddfe29cb: Can't schedule compaction. Clean time has not been advanced past its initial value. W0703 01:42:24.753069 29202 flags.cc:406] Enabled unsafe flag: --never_fsync=true Using table 'rebalance_test_table_0' Generator report time total : 162.631 ms time per row: 0.0813154 ms I0703 01:42:25.008707 28255 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.5.240.254:44777: name: "rebalance_test_table_1" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 1 split_rows_range_bounds { } partition_schema { hash_bucket_schemas { columns { name: "key" } num_buckets: 3 seed: 0 } } I0703 01:42:25.014874 28479 tablet_service.cc:759] Processing CreateTablet for tablet 1cd5cbfe70ec4f8a957fa074ae08fcb1 (table=rebalance_test_table_1 [id=14bf1112832a4171b3d41a08ed95d54e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I0703 01:42:25.014906 28478 tablet_service.cc:759] Processing CreateTablet for tablet 41c52d5ecddf4e5897acf3fe5ee44f34 (table=rebalance_test_table_1 [id=14bf1112832a4171b3d41a08ed95d54e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I0703 01:42:25.015091 28480 tablet_service.cc:759] Processing CreateTablet for tablet d0d06a73ceef43ffae542e800f00ff20 (table=rebalance_test_table_1 [id=14bf1112832a4171b3d41a08ed95d54e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I0703 01:42:25.015302 28479 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 1cd5cbfe70ec4f8a957fa074ae08fcb1. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:25.015591 28480 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet d0d06a73ceef43ffae542e800f00ff20. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:25.015867 28478 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 41c52d5ecddf4e5897acf3fe5ee44f34. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:25.022737 28478 ts_tablet_manager.cc:1173] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 31c252b54241470ba1115181c3672664: Registered tablet (data state: TABLET_DATA_READY) I0703 01:42:25.022971 28479 ts_tablet_manager.cc:1173] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 31c252b54241470ba1115181c3672664: Registered tablet (data state: TABLET_DATA_READY) I0703 01:42:25.024366 29221 ts_tablet_manager.cc:1028] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 31c252b54241470ba1115181c3672664: Bootstrapping tablet I0703 01:42:25.024559 29221 tablet_bootstrap.cc:427] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 31c252b54241470ba1115181c3672664: Bootstrap starting. I0703 01:42:25.024866 28480 ts_tablet_manager.cc:1173] T d0d06a73ceef43ffae542e800f00ff20 P 31c252b54241470ba1115181c3672664: Registered tablet (data state: TABLET_DATA_READY) I0703 01:42:25.026268 29221 tablet_bootstrap.cc:578] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 31c252b54241470ba1115181c3672664: No blocks or log segments found. Creating new log. I0703 01:42:25.027637 29221 tablet_bootstrap.cc:427] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 31c252b54241470ba1115181c3672664: No bootstrap required, opened a new log I0703 01:42:25.027897 29221 ts_tablet_manager.cc:1045] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 31c252b54241470ba1115181c3672664: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s I0703 01:42:25.028667 29221 raft_consensus.cc:325] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 31c252b54241470ba1115181c3672664 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } } I0703 01:42:25.028771 29221 raft_consensus.cc:351] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 31c252b54241470ba1115181c3672664 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:42:25.028812 29221 raft_consensus.cc:628] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 31c252b54241470ba1115181c3672664 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31c252b54241470ba1115181c3672664, State: Initialized, Role: FOLLOWER I0703 01:42:25.028959 29221 consensus_queue.cc:227] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 31c252b54241470ba1115181c3672664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } } I0703 01:42:25.029062 29221 raft_consensus.cc:365] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 31c252b54241470ba1115181c3672664 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:42:25.029103 29221 raft_consensus.cc:457] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 31c252b54241470ba1115181c3672664 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:42:25.029144 29221 raft_consensus.cc:2730] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 31c252b54241470ba1115181c3672664: Snoozing failure detection for 1.733s (starting election) I0703 01:42:25.029191 29221 raft_consensus.cc:2782] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 31c252b54241470ba1115181c3672664 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:42:25.030833 29221 raft_consensus.cc:479] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 31c252b54241470ba1115181c3672664 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } } I0703 01:42:25.030993 29221 leader_election.cc:270] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 31c252b54241470ba1115181c3672664 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:42:25.031525 29223 raft_consensus.cc:2730] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 31c252b54241470ba1115181c3672664: Snoozing failure detection for 1.824s (election complete) I0703 01:42:25.031734 29223 raft_consensus.cc:2537] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 31c252b54241470ba1115181c3672664 [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:42:25.032327 29221 ts_tablet_manager.cc:1073] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 31c252b54241470ba1115181c3672664: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s I0703 01:42:25.032572 29223 raft_consensus.cc:593] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 31c252b54241470ba1115181c3672664 [term 1 LEADER]: Becoming Leader. State: Replica: 31c252b54241470ba1115181c3672664, State: Running, Role: LEADER I0703 01:42:25.032711 29221 ts_tablet_manager.cc:1028] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 31c252b54241470ba1115181c3672664: Bootstrapping tablet I0703 01:42:25.032958 29221 tablet_bootstrap.cc:427] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 31c252b54241470ba1115181c3672664: Bootstrap starting. I0703 01:42:25.032905 29223 consensus_queue.cc:204] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 31c252b54241470ba1115181c3672664 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } } I0703 01:42:25.035037 29221 tablet_bootstrap.cc:578] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 31c252b54241470ba1115181c3672664: No blocks or log segments found. Creating new log. I0703 01:42:25.034909 28255 catalog_manager.cc:3798] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 31c252b54241470ba1115181c3672664 reported cstate change: term changed from 0 to 1, leader changed from to 31c252b54241470ba1115181c3672664 (127.5.240.194). New cstate: current_term: 1 leader_uuid: "31c252b54241470ba1115181c3672664" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } health_report { overall_health: HEALTHY } } } I0703 01:42:25.038372 29221 tablet_bootstrap.cc:427] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 31c252b54241470ba1115181c3672664: No bootstrap required, opened a new log I0703 01:42:25.038472 29221 ts_tablet_manager.cc:1045] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 31c252b54241470ba1115181c3672664: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s I0703 01:42:25.039031 29221 raft_consensus.cc:325] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 31c252b54241470ba1115181c3672664 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } } I0703 01:42:25.039294 29221 raft_consensus.cc:351] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 31c252b54241470ba1115181c3672664 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:42:25.039340 29221 raft_consensus.cc:628] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 31c252b54241470ba1115181c3672664 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31c252b54241470ba1115181c3672664, State: Initialized, Role: FOLLOWER I0703 01:42:25.039511 29221 consensus_queue.cc:227] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 31c252b54241470ba1115181c3672664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } } I0703 01:42:25.039616 29221 raft_consensus.cc:365] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 31c252b54241470ba1115181c3672664 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:42:25.039692 29221 raft_consensus.cc:457] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 31c252b54241470ba1115181c3672664 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:42:25.039746 29221 raft_consensus.cc:2730] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 31c252b54241470ba1115181c3672664: Snoozing failure detection for 2.060s (starting election) I0703 01:42:25.039808 29221 raft_consensus.cc:2782] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 31c252b54241470ba1115181c3672664 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:42:25.041060 29221 raft_consensus.cc:479] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 31c252b54241470ba1115181c3672664 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } } I0703 01:42:25.041223 29221 leader_election.cc:270] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 31c252b54241470ba1115181c3672664 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:42:25.041447 29223 raft_consensus.cc:2730] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 31c252b54241470ba1115181c3672664: Snoozing failure detection for 1.747s (election complete) I0703 01:42:25.041584 29223 raft_consensus.cc:2537] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 31c252b54241470ba1115181c3672664 [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:42:25.041745 29223 raft_consensus.cc:593] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 31c252b54241470ba1115181c3672664 [term 1 LEADER]: Becoming Leader. State: Replica: 31c252b54241470ba1115181c3672664, State: Running, Role: LEADER I0703 01:42:25.042073 29221 ts_tablet_manager.cc:1073] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 31c252b54241470ba1115181c3672664: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s I0703 01:42:25.042208 29221 ts_tablet_manager.cc:1028] T d0d06a73ceef43ffae542e800f00ff20 P 31c252b54241470ba1115181c3672664: Bootstrapping tablet I0703 01:42:25.041965 29223 consensus_queue.cc:204] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 31c252b54241470ba1115181c3672664 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } } I0703 01:42:25.042376 29221 tablet_bootstrap.cc:427] T d0d06a73ceef43ffae542e800f00ff20 P 31c252b54241470ba1115181c3672664: Bootstrap starting. I0703 01:42:25.044359 28255 catalog_manager.cc:3798] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 31c252b54241470ba1115181c3672664 reported cstate change: term changed from 0 to 1, leader changed from to 31c252b54241470ba1115181c3672664 (127.5.240.194). New cstate: current_term: 1 leader_uuid: "31c252b54241470ba1115181c3672664" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } health_report { overall_health: HEALTHY } } } I0703 01:42:25.044895 29221 tablet_bootstrap.cc:578] T d0d06a73ceef43ffae542e800f00ff20 P 31c252b54241470ba1115181c3672664: No blocks or log segments found. Creating new log. I0703 01:42:25.046859 29221 tablet_bootstrap.cc:427] T d0d06a73ceef43ffae542e800f00ff20 P 31c252b54241470ba1115181c3672664: No bootstrap required, opened a new log I0703 01:42:25.046967 29221 ts_tablet_manager.cc:1045] T d0d06a73ceef43ffae542e800f00ff20 P 31c252b54241470ba1115181c3672664: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s I0703 01:42:25.047441 29221 raft_consensus.cc:325] T d0d06a73ceef43ffae542e800f00ff20 P 31c252b54241470ba1115181c3672664 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } } I0703 01:42:25.047544 29221 raft_consensus.cc:351] T d0d06a73ceef43ffae542e800f00ff20 P 31c252b54241470ba1115181c3672664 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:42:25.047603 29221 raft_consensus.cc:628] T d0d06a73ceef43ffae542e800f00ff20 P 31c252b54241470ba1115181c3672664 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31c252b54241470ba1115181c3672664, State: Initialized, Role: FOLLOWER I0703 01:42:25.047802 29221 consensus_queue.cc:227] T d0d06a73ceef43ffae542e800f00ff20 P 31c252b54241470ba1115181c3672664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } } I0703 01:42:25.047906 29221 raft_consensus.cc:365] T d0d06a73ceef43ffae542e800f00ff20 P 31c252b54241470ba1115181c3672664 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:42:25.047957 29221 raft_consensus.cc:457] T d0d06a73ceef43ffae542e800f00ff20 P 31c252b54241470ba1115181c3672664 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:42:25.048003 29221 raft_consensus.cc:2730] T d0d06a73ceef43ffae542e800f00ff20 P 31c252b54241470ba1115181c3672664: Snoozing failure detection for 1.853s (starting election) I0703 01:42:25.048060 29221 raft_consensus.cc:2782] T d0d06a73ceef43ffae542e800f00ff20 P 31c252b54241470ba1115181c3672664 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:42:25.049316 29221 raft_consensus.cc:479] T d0d06a73ceef43ffae542e800f00ff20 P 31c252b54241470ba1115181c3672664 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } } I0703 01:42:25.049479 29221 leader_election.cc:270] T d0d06a73ceef43ffae542e800f00ff20 P 31c252b54241470ba1115181c3672664 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:42:25.049847 29223 raft_consensus.cc:2730] T d0d06a73ceef43ffae542e800f00ff20 P 31c252b54241470ba1115181c3672664: Snoozing failure detection for 2.040s (election complete) I0703 01:42:25.049958 29223 raft_consensus.cc:2537] T d0d06a73ceef43ffae542e800f00ff20 P 31c252b54241470ba1115181c3672664 [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:42:25.050117 29221 ts_tablet_manager.cc:1073] T d0d06a73ceef43ffae542e800f00ff20 P 31c252b54241470ba1115181c3672664: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s I0703 01:42:25.050117 29223 raft_consensus.cc:593] T d0d06a73ceef43ffae542e800f00ff20 P 31c252b54241470ba1115181c3672664 [term 1 LEADER]: Becoming Leader. State: Replica: 31c252b54241470ba1115181c3672664, State: Running, Role: LEADER I0703 01:42:25.050257 29223 consensus_queue.cc:204] T d0d06a73ceef43ffae542e800f00ff20 P 31c252b54241470ba1115181c3672664 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } } I0703 01:42:25.052135 28255 catalog_manager.cc:3798] T d0d06a73ceef43ffae542e800f00ff20 P 31c252b54241470ba1115181c3672664 reported cstate change: term changed from 0 to 1, leader changed from to 31c252b54241470ba1115181c3672664 (127.5.240.194). New cstate: current_term: 1 leader_uuid: "31c252b54241470ba1115181c3672664" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } health_report { overall_health: HEALTHY } } } W0703 01:42:25.271795 29231 flags.cc:406] Enabled unsafe flag: --never_fsync=true Using table 'rebalance_test_table_1' Generator report time total : 168.077 ms time per row: 0.0840385 ms I0703 01:42:25.534049 28253 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.5.240.254:44777: name: "rebalance_test_table_2" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 1 split_rows_range_bounds { } partition_schema { hash_bucket_schemas { columns { name: "key" } num_buckets: 3 seed: 0 } } I0703 01:42:25.540819 28355 tablet_service.cc:759] Processing CreateTablet for tablet 12020f47e415490ebd506a2685335058 (table=rebalance_test_table_2 [id=b5ac6ea59801477a8e4cca2b80c8beec]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I0703 01:42:25.540820 28356 tablet_service.cc:759] Processing CreateTablet for tablet 776d4f14ef3e4b5494f5834ae1676782 (table=rebalance_test_table_2 [id=b5ac6ea59801477a8e4cca2b80c8beec]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I0703 01:42:25.540820 28357 tablet_service.cc:759] Processing CreateTablet for tablet 107baa5504214d5f8362cef92ed80291 (table=rebalance_test_table_2 [id=b5ac6ea59801477a8e4cca2b80c8beec]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I0703 01:42:25.541266 28356 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 776d4f14ef3e4b5494f5834ae1676782. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:25.541452 28357 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 107baa5504214d5f8362cef92ed80291. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:25.541829 28355 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 12020f47e415490ebd506a2685335058. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:25.546880 28355 ts_tablet_manager.cc:1173] T 12020f47e415490ebd506a2685335058 P 5ee4739615124e308720b4aeddfe29cb: Registered tablet (data state: TABLET_DATA_READY) I0703 01:42:25.547610 28356 ts_tablet_manager.cc:1173] T 776d4f14ef3e4b5494f5834ae1676782 P 5ee4739615124e308720b4aeddfe29cb: Registered tablet (data state: TABLET_DATA_READY) I0703 01:42:25.547610 29251 ts_tablet_manager.cc:1028] T 12020f47e415490ebd506a2685335058 P 5ee4739615124e308720b4aeddfe29cb: Bootstrapping tablet I0703 01:42:25.547803 29251 tablet_bootstrap.cc:427] T 12020f47e415490ebd506a2685335058 P 5ee4739615124e308720b4aeddfe29cb: Bootstrap starting. I0703 01:42:25.547802 28357 ts_tablet_manager.cc:1173] T 107baa5504214d5f8362cef92ed80291 P 5ee4739615124e308720b4aeddfe29cb: Registered tablet (data state: TABLET_DATA_READY) I0703 01:42:25.548832 29251 tablet_bootstrap.cc:578] T 12020f47e415490ebd506a2685335058 P 5ee4739615124e308720b4aeddfe29cb: No blocks or log segments found. Creating new log. I0703 01:42:25.550186 29251 tablet_bootstrap.cc:427] T 12020f47e415490ebd506a2685335058 P 5ee4739615124e308720b4aeddfe29cb: No bootstrap required, opened a new log I0703 01:42:25.550282 29251 ts_tablet_manager.cc:1045] T 12020f47e415490ebd506a2685335058 P 5ee4739615124e308720b4aeddfe29cb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s I0703 01:42:25.550781 29251 raft_consensus.cc:325] T 12020f47e415490ebd506a2685335058 P 5ee4739615124e308720b4aeddfe29cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } } I0703 01:42:25.550891 29251 raft_consensus.cc:351] T 12020f47e415490ebd506a2685335058 P 5ee4739615124e308720b4aeddfe29cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:42:25.550940 29251 raft_consensus.cc:628] T 12020f47e415490ebd506a2685335058 P 5ee4739615124e308720b4aeddfe29cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ee4739615124e308720b4aeddfe29cb, State: Initialized, Role: FOLLOWER I0703 01:42:25.551105 29251 consensus_queue.cc:227] T 12020f47e415490ebd506a2685335058 P 5ee4739615124e308720b4aeddfe29cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } } I0703 01:42:25.551210 29251 raft_consensus.cc:365] T 12020f47e415490ebd506a2685335058 P 5ee4739615124e308720b4aeddfe29cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:42:25.551267 29251 raft_consensus.cc:457] T 12020f47e415490ebd506a2685335058 P 5ee4739615124e308720b4aeddfe29cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:42:25.551316 29251 raft_consensus.cc:2730] T 12020f47e415490ebd506a2685335058 P 5ee4739615124e308720b4aeddfe29cb: Snoozing failure detection for 2.020s (starting election) I0703 01:42:25.551378 29251 raft_consensus.cc:2782] T 12020f47e415490ebd506a2685335058 P 5ee4739615124e308720b4aeddfe29cb [term 0 FOLLOWER]: Advancing to term 1 I0703 01:42:25.552567 29251 raft_consensus.cc:479] T 12020f47e415490ebd506a2685335058 P 5ee4739615124e308720b4aeddfe29cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } } I0703 01:42:25.552740 29251 leader_election.cc:270] T 12020f47e415490ebd506a2685335058 P 5ee4739615124e308720b4aeddfe29cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:42:25.553200 29253 raft_consensus.cc:2730] T 12020f47e415490ebd506a2685335058 P 5ee4739615124e308720b4aeddfe29cb: Snoozing failure detection for 1.939s (election complete) I0703 01:42:25.553771 29253 raft_consensus.cc:2537] T 12020f47e415490ebd506a2685335058 P 5ee4739615124e308720b4aeddfe29cb [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:42:25.554360 29251 ts_tablet_manager.cc:1073] T 12020f47e415490ebd506a2685335058 P 5ee4739615124e308720b4aeddfe29cb: Time spent starting tablet: real 0.004s user 0.002s sys 0.002s I0703 01:42:25.554538 29251 ts_tablet_manager.cc:1028] T 776d4f14ef3e4b5494f5834ae1676782 P 5ee4739615124e308720b4aeddfe29cb: Bootstrapping tablet I0703 01:42:25.554757 29251 tablet_bootstrap.cc:427] T 776d4f14ef3e4b5494f5834ae1676782 P 5ee4739615124e308720b4aeddfe29cb: Bootstrap starting. I0703 01:42:25.554761 29253 raft_consensus.cc:593] T 12020f47e415490ebd506a2685335058 P 5ee4739615124e308720b4aeddfe29cb [term 1 LEADER]: Becoming Leader. State: Replica: 5ee4739615124e308720b4aeddfe29cb, State: Running, Role: LEADER I0703 01:42:25.554966 29253 consensus_queue.cc:204] T 12020f47e415490ebd506a2685335058 P 5ee4739615124e308720b4aeddfe29cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } } I0703 01:42:25.556533 29251 tablet_bootstrap.cc:578] T 776d4f14ef3e4b5494f5834ae1676782 P 5ee4739615124e308720b4aeddfe29cb: No blocks or log segments found. Creating new log. I0703 01:42:25.556982 28253 catalog_manager.cc:3798] T 12020f47e415490ebd506a2685335058 P 5ee4739615124e308720b4aeddfe29cb reported cstate change: term changed from 0 to 1, leader changed from to 5ee4739615124e308720b4aeddfe29cb (127.5.240.193). New cstate: current_term: 1 leader_uuid: "5ee4739615124e308720b4aeddfe29cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } health_report { overall_health: HEALTHY } } } I0703 01:42:25.559067 29251 tablet_bootstrap.cc:427] T 776d4f14ef3e4b5494f5834ae1676782 P 5ee4739615124e308720b4aeddfe29cb: No bootstrap required, opened a new log I0703 01:42:25.559170 29251 ts_tablet_manager.cc:1045] T 776d4f14ef3e4b5494f5834ae1676782 P 5ee4739615124e308720b4aeddfe29cb: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s I0703 01:42:25.559926 29251 raft_consensus.cc:325] T 776d4f14ef3e4b5494f5834ae1676782 P 5ee4739615124e308720b4aeddfe29cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } } I0703 01:42:25.560107 29251 raft_consensus.cc:351] T 776d4f14ef3e4b5494f5834ae1676782 P 5ee4739615124e308720b4aeddfe29cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:42:25.560163 29251 raft_consensus.cc:628] T 776d4f14ef3e4b5494f5834ae1676782 P 5ee4739615124e308720b4aeddfe29cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ee4739615124e308720b4aeddfe29cb, State: Initialized, Role: FOLLOWER I0703 01:42:25.560358 29251 consensus_queue.cc:227] T 776d4f14ef3e4b5494f5834ae1676782 P 5ee4739615124e308720b4aeddfe29cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } } I0703 01:42:25.560468 29251 raft_consensus.cc:365] T 776d4f14ef3e4b5494f5834ae1676782 P 5ee4739615124e308720b4aeddfe29cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:42:25.560521 29251 raft_consensus.cc:457] T 776d4f14ef3e4b5494f5834ae1676782 P 5ee4739615124e308720b4aeddfe29cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:42:25.560602 29251 raft_consensus.cc:2730] T 776d4f14ef3e4b5494f5834ae1676782 P 5ee4739615124e308720b4aeddfe29cb: Snoozing failure detection for 1.911s (starting election) I0703 01:42:25.560662 29251 raft_consensus.cc:2782] T 776d4f14ef3e4b5494f5834ae1676782 P 5ee4739615124e308720b4aeddfe29cb [term 0 FOLLOWER]: Advancing to term 1 I0703 01:42:25.562113 29251 raft_consensus.cc:479] T 776d4f14ef3e4b5494f5834ae1676782 P 5ee4739615124e308720b4aeddfe29cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } } I0703 01:42:25.562253 29251 leader_election.cc:270] T 776d4f14ef3e4b5494f5834ae1676782 P 5ee4739615124e308720b4aeddfe29cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:42:25.562444 29253 raft_consensus.cc:2730] T 776d4f14ef3e4b5494f5834ae1676782 P 5ee4739615124e308720b4aeddfe29cb: Snoozing failure detection for 2.011s (election complete) I0703 01:42:25.562571 29253 raft_consensus.cc:2537] T 776d4f14ef3e4b5494f5834ae1676782 P 5ee4739615124e308720b4aeddfe29cb [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:42:25.562700 29253 raft_consensus.cc:593] T 776d4f14ef3e4b5494f5834ae1676782 P 5ee4739615124e308720b4aeddfe29cb [term 1 LEADER]: Becoming Leader. State: Replica: 5ee4739615124e308720b4aeddfe29cb, State: Running, Role: LEADER I0703 01:42:25.562834 29251 ts_tablet_manager.cc:1073] T 776d4f14ef3e4b5494f5834ae1676782 P 5ee4739615124e308720b4aeddfe29cb: Time spent starting tablet: real 0.004s user 0.002s sys 0.002s I0703 01:42:25.562970 29251 ts_tablet_manager.cc:1028] T 107baa5504214d5f8362cef92ed80291 P 5ee4739615124e308720b4aeddfe29cb: Bootstrapping tablet I0703 01:42:25.562871 29253 consensus_queue.cc:204] T 776d4f14ef3e4b5494f5834ae1676782 P 5ee4739615124e308720b4aeddfe29cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } } I0703 01:42:25.563097 29251 tablet_bootstrap.cc:427] T 107baa5504214d5f8362cef92ed80291 P 5ee4739615124e308720b4aeddfe29cb: Bootstrap starting. I0703 01:42:25.565024 29251 tablet_bootstrap.cc:578] T 107baa5504214d5f8362cef92ed80291 P 5ee4739615124e308720b4aeddfe29cb: No blocks or log segments found. Creating new log. I0703 01:42:25.565011 28253 catalog_manager.cc:3798] T 776d4f14ef3e4b5494f5834ae1676782 P 5ee4739615124e308720b4aeddfe29cb reported cstate change: term changed from 0 to 1, leader changed from to 5ee4739615124e308720b4aeddfe29cb (127.5.240.193). New cstate: current_term: 1 leader_uuid: "5ee4739615124e308720b4aeddfe29cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } health_report { overall_health: HEALTHY } } } I0703 01:42:25.567997 29251 tablet_bootstrap.cc:427] T 107baa5504214d5f8362cef92ed80291 P 5ee4739615124e308720b4aeddfe29cb: No bootstrap required, opened a new log I0703 01:42:25.568123 29251 ts_tablet_manager.cc:1045] T 107baa5504214d5f8362cef92ed80291 P 5ee4739615124e308720b4aeddfe29cb: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s I0703 01:42:25.568609 29251 raft_consensus.cc:325] T 107baa5504214d5f8362cef92ed80291 P 5ee4739615124e308720b4aeddfe29cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } } I0703 01:42:25.568819 29251 raft_consensus.cc:351] T 107baa5504214d5f8362cef92ed80291 P 5ee4739615124e308720b4aeddfe29cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:42:25.568893 29251 raft_consensus.cc:628] T 107baa5504214d5f8362cef92ed80291 P 5ee4739615124e308720b4aeddfe29cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ee4739615124e308720b4aeddfe29cb, State: Initialized, Role: FOLLOWER I0703 01:42:25.569048 29251 consensus_queue.cc:227] T 107baa5504214d5f8362cef92ed80291 P 5ee4739615124e308720b4aeddfe29cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } } I0703 01:42:25.569149 29251 raft_consensus.cc:365] T 107baa5504214d5f8362cef92ed80291 P 5ee4739615124e308720b4aeddfe29cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:42:25.569212 29251 raft_consensus.cc:457] T 107baa5504214d5f8362cef92ed80291 P 5ee4739615124e308720b4aeddfe29cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:42:25.569263 29251 raft_consensus.cc:2730] T 107baa5504214d5f8362cef92ed80291 P 5ee4739615124e308720b4aeddfe29cb: Snoozing failure detection for 1.618s (starting election) I0703 01:42:25.569329 29251 raft_consensus.cc:2782] T 107baa5504214d5f8362cef92ed80291 P 5ee4739615124e308720b4aeddfe29cb [term 0 FOLLOWER]: Advancing to term 1 I0703 01:42:25.570494 29251 raft_consensus.cc:479] T 107baa5504214d5f8362cef92ed80291 P 5ee4739615124e308720b4aeddfe29cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } } I0703 01:42:25.570659 29251 leader_election.cc:270] T 107baa5504214d5f8362cef92ed80291 P 5ee4739615124e308720b4aeddfe29cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:42:25.570900 29253 raft_consensus.cc:2730] T 107baa5504214d5f8362cef92ed80291 P 5ee4739615124e308720b4aeddfe29cb: Snoozing failure detection for 1.632s (election complete) I0703 01:42:25.571010 29253 raft_consensus.cc:2537] T 107baa5504214d5f8362cef92ed80291 P 5ee4739615124e308720b4aeddfe29cb [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:42:25.571184 29253 raft_consensus.cc:593] T 107baa5504214d5f8362cef92ed80291 P 5ee4739615124e308720b4aeddfe29cb [term 1 LEADER]: Becoming Leader. State: Replica: 5ee4739615124e308720b4aeddfe29cb, State: Running, Role: LEADER I0703 01:42:25.571300 29251 ts_tablet_manager.cc:1073] T 107baa5504214d5f8362cef92ed80291 P 5ee4739615124e308720b4aeddfe29cb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s I0703 01:42:25.571367 29253 consensus_queue.cc:204] T 107baa5504214d5f8362cef92ed80291 P 5ee4739615124e308720b4aeddfe29cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } } I0703 01:42:25.572798 28253 catalog_manager.cc:3798] T 107baa5504214d5f8362cef92ed80291 P 5ee4739615124e308720b4aeddfe29cb reported cstate change: term changed from 0 to 1, leader changed from to 5ee4739615124e308720b4aeddfe29cb (127.5.240.193). New cstate: current_term: 1 leader_uuid: "5ee4739615124e308720b4aeddfe29cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } health_report { overall_health: HEALTHY } } } W0703 01:42:25.798377 29261 flags.cc:406] Enabled unsafe flag: --never_fsync=true Using table 'rebalance_test_table_2' Generator report time total : 175.139 ms time per row: 0.0875697 ms I0703 01:42:26.068102 28253 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.5.240.254:44777: name: "rebalance_test_table_3" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 1 split_rows_range_bounds { } partition_schema { hash_bucket_schemas { columns { name: "key" } num_buckets: 3 seed: 0 } } I0703 01:42:26.074187 28480 tablet_service.cc:759] Processing CreateTablet for tablet ef65b31bc12e40c2b3f1e4714cc4218e (table=rebalance_test_table_3 [id=75241f23aa654e04a8d5a26fb0d09c07]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I0703 01:42:26.074471 28479 tablet_service.cc:759] Processing CreateTablet for tablet 775bd6dae9ed48ebbbb9df088d833772 (table=rebalance_test_table_3 [id=75241f23aa654e04a8d5a26fb0d09c07]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I0703 01:42:26.074611 28480 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet ef65b31bc12e40c2b3f1e4714cc4218e. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:26.074187 28478 tablet_service.cc:759] Processing CreateTablet for tablet 46f9123e80ac47f59d142ca87e99a1d5 (table=rebalance_test_table_3 [id=75241f23aa654e04a8d5a26fb0d09c07]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I0703 01:42:26.074959 28478 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 46f9123e80ac47f59d142ca87e99a1d5. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:26.077113 28479 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 775bd6dae9ed48ebbbb9df088d833772. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:26.080664 28478 ts_tablet_manager.cc:1173] T 46f9123e80ac47f59d142ca87e99a1d5 P 31c252b54241470ba1115181c3672664: Registered tablet (data state: TABLET_DATA_READY) I0703 01:42:26.080893 28480 ts_tablet_manager.cc:1173] T ef65b31bc12e40c2b3f1e4714cc4218e P 31c252b54241470ba1115181c3672664: Registered tablet (data state: TABLET_DATA_READY) I0703 01:42:26.081645 29281 ts_tablet_manager.cc:1028] T 46f9123e80ac47f59d142ca87e99a1d5 P 31c252b54241470ba1115181c3672664: Bootstrapping tablet I0703 01:42:26.081835 29281 tablet_bootstrap.cc:427] T 46f9123e80ac47f59d142ca87e99a1d5 P 31c252b54241470ba1115181c3672664: Bootstrap starting. I0703 01:42:26.082114 28479 ts_tablet_manager.cc:1173] T 775bd6dae9ed48ebbbb9df088d833772 P 31c252b54241470ba1115181c3672664: Registered tablet (data state: TABLET_DATA_READY) I0703 01:42:26.083573 29281 tablet_bootstrap.cc:578] T 46f9123e80ac47f59d142ca87e99a1d5 P 31c252b54241470ba1115181c3672664: No blocks or log segments found. Creating new log. I0703 01:42:26.084897 29281 tablet_bootstrap.cc:427] T 46f9123e80ac47f59d142ca87e99a1d5 P 31c252b54241470ba1115181c3672664: No bootstrap required, opened a new log I0703 01:42:26.084983 29281 ts_tablet_manager.cc:1045] T 46f9123e80ac47f59d142ca87e99a1d5 P 31c252b54241470ba1115181c3672664: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s I0703 01:42:26.085470 29281 raft_consensus.cc:325] T 46f9123e80ac47f59d142ca87e99a1d5 P 31c252b54241470ba1115181c3672664 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } } I0703 01:42:26.085592 29281 raft_consensus.cc:351] T 46f9123e80ac47f59d142ca87e99a1d5 P 31c252b54241470ba1115181c3672664 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:42:26.085634 29281 raft_consensus.cc:628] T 46f9123e80ac47f59d142ca87e99a1d5 P 31c252b54241470ba1115181c3672664 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31c252b54241470ba1115181c3672664, State: Initialized, Role: FOLLOWER I0703 01:42:26.085788 29281 consensus_queue.cc:227] T 46f9123e80ac47f59d142ca87e99a1d5 P 31c252b54241470ba1115181c3672664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } } I0703 01:42:26.085914 29281 raft_consensus.cc:365] T 46f9123e80ac47f59d142ca87e99a1d5 P 31c252b54241470ba1115181c3672664 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:42:26.085958 29281 raft_consensus.cc:457] T 46f9123e80ac47f59d142ca87e99a1d5 P 31c252b54241470ba1115181c3672664 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:42:26.085997 29281 raft_consensus.cc:2730] T 46f9123e80ac47f59d142ca87e99a1d5 P 31c252b54241470ba1115181c3672664: Snoozing failure detection for 1.782s (starting election) I0703 01:42:26.086043 29281 raft_consensus.cc:2782] T 46f9123e80ac47f59d142ca87e99a1d5 P 31c252b54241470ba1115181c3672664 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:42:26.087632 29281 raft_consensus.cc:479] T 46f9123e80ac47f59d142ca87e99a1d5 P 31c252b54241470ba1115181c3672664 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } } I0703 01:42:26.087795 29281 leader_election.cc:270] T 46f9123e80ac47f59d142ca87e99a1d5 P 31c252b54241470ba1115181c3672664 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:42:26.088227 29283 raft_consensus.cc:2730] T 46f9123e80ac47f59d142ca87e99a1d5 P 31c252b54241470ba1115181c3672664: Snoozing failure detection for 1.742s (election complete) I0703 01:42:26.088408 29283 raft_consensus.cc:2537] T 46f9123e80ac47f59d142ca87e99a1d5 P 31c252b54241470ba1115181c3672664 [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:42:26.088997 29281 ts_tablet_manager.cc:1073] T 46f9123e80ac47f59d142ca87e99a1d5 P 31c252b54241470ba1115181c3672664: Time spent starting tablet: real 0.004s user 0.002s sys 0.002s I0703 01:42:26.089268 29281 ts_tablet_manager.cc:1028] T ef65b31bc12e40c2b3f1e4714cc4218e P 31c252b54241470ba1115181c3672664: Bootstrapping tablet I0703 01:42:26.089259 29283 raft_consensus.cc:593] T 46f9123e80ac47f59d142ca87e99a1d5 P 31c252b54241470ba1115181c3672664 [term 1 LEADER]: Becoming Leader. State: Replica: 31c252b54241470ba1115181c3672664, State: Running, Role: LEADER I0703 01:42:26.089423 29281 tablet_bootstrap.cc:427] T ef65b31bc12e40c2b3f1e4714cc4218e P 31c252b54241470ba1115181c3672664: Bootstrap starting. I0703 01:42:26.089452 29283 consensus_queue.cc:204] T 46f9123e80ac47f59d142ca87e99a1d5 P 31c252b54241470ba1115181c3672664 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } } I0703 01:42:26.091042 29281 tablet_bootstrap.cc:578] T ef65b31bc12e40c2b3f1e4714cc4218e P 31c252b54241470ba1115181c3672664: No blocks or log segments found. Creating new log. I0703 01:42:26.091593 28253 catalog_manager.cc:3798] T 46f9123e80ac47f59d142ca87e99a1d5 P 31c252b54241470ba1115181c3672664 reported cstate change: term changed from 0 to 1, leader changed from to 31c252b54241470ba1115181c3672664 (127.5.240.194). New cstate: current_term: 1 leader_uuid: "31c252b54241470ba1115181c3672664" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } health_report { overall_health: HEALTHY } } } I0703 01:42:26.093431 29281 tablet_bootstrap.cc:427] T ef65b31bc12e40c2b3f1e4714cc4218e P 31c252b54241470ba1115181c3672664: No bootstrap required, opened a new log I0703 01:42:26.093597 29281 ts_tablet_manager.cc:1045] T ef65b31bc12e40c2b3f1e4714cc4218e P 31c252b54241470ba1115181c3672664: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s I0703 01:42:26.094122 29281 raft_consensus.cc:325] T ef65b31bc12e40c2b3f1e4714cc4218e P 31c252b54241470ba1115181c3672664 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } } I0703 01:42:26.094286 29281 raft_consensus.cc:351] T ef65b31bc12e40c2b3f1e4714cc4218e P 31c252b54241470ba1115181c3672664 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:42:26.094357 29281 raft_consensus.cc:628] T ef65b31bc12e40c2b3f1e4714cc4218e P 31c252b54241470ba1115181c3672664 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31c252b54241470ba1115181c3672664, State: Initialized, Role: FOLLOWER I0703 01:42:26.094525 29281 consensus_queue.cc:227] T ef65b31bc12e40c2b3f1e4714cc4218e P 31c252b54241470ba1115181c3672664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } } I0703 01:42:26.094640 29281 raft_consensus.cc:365] T ef65b31bc12e40c2b3f1e4714cc4218e P 31c252b54241470ba1115181c3672664 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:42:26.094697 29281 raft_consensus.cc:457] T ef65b31bc12e40c2b3f1e4714cc4218e P 31c252b54241470ba1115181c3672664 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:42:26.094774 29281 raft_consensus.cc:2730] T ef65b31bc12e40c2b3f1e4714cc4218e P 31c252b54241470ba1115181c3672664: Snoozing failure detection for 1.646s (starting election) I0703 01:42:26.094861 29281 raft_consensus.cc:2782] T ef65b31bc12e40c2b3f1e4714cc4218e P 31c252b54241470ba1115181c3672664 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:42:26.096362 29281 raft_consensus.cc:479] T ef65b31bc12e40c2b3f1e4714cc4218e P 31c252b54241470ba1115181c3672664 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } } I0703 01:42:26.096525 29281 leader_election.cc:270] T ef65b31bc12e40c2b3f1e4714cc4218e P 31c252b54241470ba1115181c3672664 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:42:26.096736 29283 raft_consensus.cc:2730] T ef65b31bc12e40c2b3f1e4714cc4218e P 31c252b54241470ba1115181c3672664: Snoozing failure detection for 2.037s (election complete) I0703 01:42:26.096859 29283 raft_consensus.cc:2537] T ef65b31bc12e40c2b3f1e4714cc4218e P 31c252b54241470ba1115181c3672664 [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:42:26.097046 29281 ts_tablet_manager.cc:1073] T ef65b31bc12e40c2b3f1e4714cc4218e P 31c252b54241470ba1115181c3672664: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s I0703 01:42:26.097045 29283 raft_consensus.cc:593] T ef65b31bc12e40c2b3f1e4714cc4218e P 31c252b54241470ba1115181c3672664 [term 1 LEADER]: Becoming Leader. State: Replica: 31c252b54241470ba1115181c3672664, State: Running, Role: LEADER I0703 01:42:26.097153 29281 ts_tablet_manager.cc:1028] T 775bd6dae9ed48ebbbb9df088d833772 P 31c252b54241470ba1115181c3672664: Bootstrapping tablet I0703 01:42:26.097275 29281 tablet_bootstrap.cc:427] T 775bd6dae9ed48ebbbb9df088d833772 P 31c252b54241470ba1115181c3672664: Bootstrap starting. I0703 01:42:26.097391 29283 consensus_queue.cc:204] T ef65b31bc12e40c2b3f1e4714cc4218e P 31c252b54241470ba1115181c3672664 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } } I0703 01:42:26.098884 29281 tablet_bootstrap.cc:578] T 775bd6dae9ed48ebbbb9df088d833772 P 31c252b54241470ba1115181c3672664: No blocks or log segments found. Creating new log. I0703 01:42:26.099377 28253 catalog_manager.cc:3798] T ef65b31bc12e40c2b3f1e4714cc4218e P 31c252b54241470ba1115181c3672664 reported cstate change: term changed from 0 to 1, leader changed from to 31c252b54241470ba1115181c3672664 (127.5.240.194). New cstate: current_term: 1 leader_uuid: "31c252b54241470ba1115181c3672664" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } health_report { overall_health: HEALTHY } } } I0703 01:42:26.101037 29281 tablet_bootstrap.cc:427] T 775bd6dae9ed48ebbbb9df088d833772 P 31c252b54241470ba1115181c3672664: No bootstrap required, opened a new log I0703 01:42:26.101166 29281 ts_tablet_manager.cc:1045] T 775bd6dae9ed48ebbbb9df088d833772 P 31c252b54241470ba1115181c3672664: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s I0703 01:42:26.101677 29281 raft_consensus.cc:325] T 775bd6dae9ed48ebbbb9df088d833772 P 31c252b54241470ba1115181c3672664 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } } I0703 01:42:26.101801 29281 raft_consensus.cc:351] T 775bd6dae9ed48ebbbb9df088d833772 P 31c252b54241470ba1115181c3672664 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:42:26.101876 29281 raft_consensus.cc:628] T 775bd6dae9ed48ebbbb9df088d833772 P 31c252b54241470ba1115181c3672664 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31c252b54241470ba1115181c3672664, State: Initialized, Role: FOLLOWER I0703 01:42:26.102030 29281 consensus_queue.cc:227] T 775bd6dae9ed48ebbbb9df088d833772 P 31c252b54241470ba1115181c3672664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } } I0703 01:42:26.102159 29281 raft_consensus.cc:365] T 775bd6dae9ed48ebbbb9df088d833772 P 31c252b54241470ba1115181c3672664 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:42:26.102212 29281 raft_consensus.cc:457] T 775bd6dae9ed48ebbbb9df088d833772 P 31c252b54241470ba1115181c3672664 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:42:26.102264 29281 raft_consensus.cc:2730] T 775bd6dae9ed48ebbbb9df088d833772 P 31c252b54241470ba1115181c3672664: Snoozing failure detection for 1.884s (starting election) I0703 01:42:26.102329 29281 raft_consensus.cc:2782] T 775bd6dae9ed48ebbbb9df088d833772 P 31c252b54241470ba1115181c3672664 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:42:26.103693 29281 raft_consensus.cc:479] T 775bd6dae9ed48ebbbb9df088d833772 P 31c252b54241470ba1115181c3672664 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } } I0703 01:42:26.103843 29281 leader_election.cc:270] T 775bd6dae9ed48ebbbb9df088d833772 P 31c252b54241470ba1115181c3672664 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:42:26.104073 29283 raft_consensus.cc:2730] T 775bd6dae9ed48ebbbb9df088d833772 P 31c252b54241470ba1115181c3672664: Snoozing failure detection for 1.863s (election complete) I0703 01:42:26.104176 29283 raft_consensus.cc:2537] T 775bd6dae9ed48ebbbb9df088d833772 P 31c252b54241470ba1115181c3672664 [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:42:26.104379 29283 raft_consensus.cc:593] T 775bd6dae9ed48ebbbb9df088d833772 P 31c252b54241470ba1115181c3672664 [term 1 LEADER]: Becoming Leader. State: Replica: 31c252b54241470ba1115181c3672664, State: Running, Role: LEADER I0703 01:42:26.104475 29281 ts_tablet_manager.cc:1073] T 775bd6dae9ed48ebbbb9df088d833772 P 31c252b54241470ba1115181c3672664: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s I0703 01:42:26.104596 29283 consensus_queue.cc:204] T 775bd6dae9ed48ebbbb9df088d833772 P 31c252b54241470ba1115181c3672664 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } } I0703 01:42:26.105973 28253 catalog_manager.cc:3798] T 775bd6dae9ed48ebbbb9df088d833772 P 31c252b54241470ba1115181c3672664 reported cstate change: term changed from 0 to 1, leader changed from to 31c252b54241470ba1115181c3672664 (127.5.240.194). New cstate: current_term: 1 leader_uuid: "31c252b54241470ba1115181c3672664" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } health_report { overall_health: HEALTHY } } } W0703 01:42:26.337260 29291 flags.cc:406] Enabled unsafe flag: --never_fsync=true Using table 'rebalance_test_table_3' Generator report time total : 156.607 ms time per row: 0.0783034 ms I0703 01:42:26.587702 28251 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.5.240.254:44777: name: "rebalance_test_table_4" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 1 split_rows_range_bounds { } partition_schema { hash_bucket_schemas { columns { name: "key" } num_buckets: 3 seed: 0 } } I0703 01:42:26.593740 28355 tablet_service.cc:759] Processing CreateTablet for tablet 74b62768046f4b8690aab7d2ec0f803a (table=rebalance_test_table_4 [id=991632a32296482d80382591992c76aa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I0703 01:42:26.593740 28357 tablet_service.cc:759] Processing CreateTablet for tablet 4d69010513df4b68840b90be3161f402 (table=rebalance_test_table_4 [id=991632a32296482d80382591992c76aa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I0703 01:42:26.593740 28356 tablet_service.cc:759] Processing CreateTablet for tablet f6594322fcdd4cd3a1357b7bda51b648 (table=rebalance_test_table_4 [id=991632a32296482d80382591992c76aa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I0703 01:42:26.594162 28355 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 74b62768046f4b8690aab7d2ec0f803a. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:26.594350 28357 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 4d69010513df4b68840b90be3161f402. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:26.594763 28356 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet f6594322fcdd4cd3a1357b7bda51b648. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:26.598973 28357 ts_tablet_manager.cc:1173] T 4d69010513df4b68840b90be3161f402 P 5ee4739615124e308720b4aeddfe29cb: Registered tablet (data state: TABLET_DATA_READY) I0703 01:42:26.600196 29311 ts_tablet_manager.cc:1028] T 4d69010513df4b68840b90be3161f402 P 5ee4739615124e308720b4aeddfe29cb: Bootstrapping tablet I0703 01:42:26.600387 29311 tablet_bootstrap.cc:427] T 4d69010513df4b68840b90be3161f402 P 5ee4739615124e308720b4aeddfe29cb: Bootstrap starting. I0703 01:42:26.601151 28355 ts_tablet_manager.cc:1173] T 74b62768046f4b8690aab7d2ec0f803a P 5ee4739615124e308720b4aeddfe29cb: Registered tablet (data state: TABLET_DATA_READY) I0703 01:42:26.601603 28356 ts_tablet_manager.cc:1173] T f6594322fcdd4cd3a1357b7bda51b648 P 5ee4739615124e308720b4aeddfe29cb: Registered tablet (data state: TABLET_DATA_READY) I0703 01:42:26.602763 29311 tablet_bootstrap.cc:578] T 4d69010513df4b68840b90be3161f402 P 5ee4739615124e308720b4aeddfe29cb: No blocks or log segments found. Creating new log. I0703 01:42:26.604218 29311 tablet_bootstrap.cc:427] T 4d69010513df4b68840b90be3161f402 P 5ee4739615124e308720b4aeddfe29cb: No bootstrap required, opened a new log I0703 01:42:26.604310 29311 ts_tablet_manager.cc:1045] T 4d69010513df4b68840b90be3161f402 P 5ee4739615124e308720b4aeddfe29cb: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s I0703 01:42:26.604791 29311 raft_consensus.cc:325] T 4d69010513df4b68840b90be3161f402 P 5ee4739615124e308720b4aeddfe29cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } } I0703 01:42:26.604898 29311 raft_consensus.cc:351] T 4d69010513df4b68840b90be3161f402 P 5ee4739615124e308720b4aeddfe29cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:42:26.604943 29311 raft_consensus.cc:628] T 4d69010513df4b68840b90be3161f402 P 5ee4739615124e308720b4aeddfe29cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ee4739615124e308720b4aeddfe29cb, State: Initialized, Role: FOLLOWER I0703 01:42:26.605109 29311 consensus_queue.cc:227] T 4d69010513df4b68840b90be3161f402 P 5ee4739615124e308720b4aeddfe29cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } } I0703 01:42:26.605216 29311 raft_consensus.cc:365] T 4d69010513df4b68840b90be3161f402 P 5ee4739615124e308720b4aeddfe29cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:42:26.605268 29311 raft_consensus.cc:457] T 4d69010513df4b68840b90be3161f402 P 5ee4739615124e308720b4aeddfe29cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:42:26.605317 29311 raft_consensus.cc:2730] T 4d69010513df4b68840b90be3161f402 P 5ee4739615124e308720b4aeddfe29cb: Snoozing failure detection for 2.061s (starting election) I0703 01:42:26.605376 29311 raft_consensus.cc:2782] T 4d69010513df4b68840b90be3161f402 P 5ee4739615124e308720b4aeddfe29cb [term 0 FOLLOWER]: Advancing to term 1 I0703 01:42:26.607062 29311 raft_consensus.cc:479] T 4d69010513df4b68840b90be3161f402 P 5ee4739615124e308720b4aeddfe29cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } } I0703 01:42:26.607228 29311 leader_election.cc:270] T 4d69010513df4b68840b90be3161f402 P 5ee4739615124e308720b4aeddfe29cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:42:26.607816 29313 raft_consensus.cc:2730] T 4d69010513df4b68840b90be3161f402 P 5ee4739615124e308720b4aeddfe29cb: Snoozing failure detection for 1.971s (election complete) I0703 01:42:26.607962 29313 raft_consensus.cc:2537] T 4d69010513df4b68840b90be3161f402 P 5ee4739615124e308720b4aeddfe29cb [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:42:26.608935 29313 raft_consensus.cc:593] T 4d69010513df4b68840b90be3161f402 P 5ee4739615124e308720b4aeddfe29cb [term 1 LEADER]: Becoming Leader. State: Replica: 5ee4739615124e308720b4aeddfe29cb, State: Running, Role: LEADER I0703 01:42:26.609115 29311 ts_tablet_manager.cc:1073] T 4d69010513df4b68840b90be3161f402 P 5ee4739615124e308720b4aeddfe29cb: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s I0703 01:42:26.609185 29313 consensus_queue.cc:204] T 4d69010513df4b68840b90be3161f402 P 5ee4739615124e308720b4aeddfe29cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } } I0703 01:42:26.609278 29311 ts_tablet_manager.cc:1028] T 74b62768046f4b8690aab7d2ec0f803a P 5ee4739615124e308720b4aeddfe29cb: Bootstrapping tablet I0703 01:42:26.609393 29311 tablet_bootstrap.cc:427] T 74b62768046f4b8690aab7d2ec0f803a P 5ee4739615124e308720b4aeddfe29cb: Bootstrap starting. I0703 01:42:26.611058 29311 tablet_bootstrap.cc:578] T 74b62768046f4b8690aab7d2ec0f803a P 5ee4739615124e308720b4aeddfe29cb: No blocks or log segments found. Creating new log. I0703 01:42:26.611191 28251 catalog_manager.cc:3798] T 4d69010513df4b68840b90be3161f402 P 5ee4739615124e308720b4aeddfe29cb reported cstate change: term changed from 0 to 1, leader changed from to 5ee4739615124e308720b4aeddfe29cb (127.5.240.193). New cstate: current_term: 1 leader_uuid: "5ee4739615124e308720b4aeddfe29cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } health_report { overall_health: HEALTHY } } } I0703 01:42:26.612865 29311 tablet_bootstrap.cc:427] T 74b62768046f4b8690aab7d2ec0f803a P 5ee4739615124e308720b4aeddfe29cb: No bootstrap required, opened a new log I0703 01:42:26.612993 29311 ts_tablet_manager.cc:1045] T 74b62768046f4b8690aab7d2ec0f803a P 5ee4739615124e308720b4aeddfe29cb: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s I0703 01:42:26.613541 29311 raft_consensus.cc:325] T 74b62768046f4b8690aab7d2ec0f803a P 5ee4739615124e308720b4aeddfe29cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } } I0703 01:42:26.613739 29311 raft_consensus.cc:351] T 74b62768046f4b8690aab7d2ec0f803a P 5ee4739615124e308720b4aeddfe29cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:42:26.613859 29311 raft_consensus.cc:628] T 74b62768046f4b8690aab7d2ec0f803a P 5ee4739615124e308720b4aeddfe29cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ee4739615124e308720b4aeddfe29cb, State: Initialized, Role: FOLLOWER I0703 01:42:26.614046 29311 consensus_queue.cc:227] T 74b62768046f4b8690aab7d2ec0f803a P 5ee4739615124e308720b4aeddfe29cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } } I0703 01:42:26.614269 29311 raft_consensus.cc:365] T 74b62768046f4b8690aab7d2ec0f803a P 5ee4739615124e308720b4aeddfe29cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:42:26.614346 29311 raft_consensus.cc:457] T 74b62768046f4b8690aab7d2ec0f803a P 5ee4739615124e308720b4aeddfe29cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:42:26.614406 29311 raft_consensus.cc:2730] T 74b62768046f4b8690aab7d2ec0f803a P 5ee4739615124e308720b4aeddfe29cb: Snoozing failure detection for 1.800s (starting election) I0703 01:42:26.614467 29311 raft_consensus.cc:2782] T 74b62768046f4b8690aab7d2ec0f803a P 5ee4739615124e308720b4aeddfe29cb [term 0 FOLLOWER]: Advancing to term 1 I0703 01:42:26.615846 29311 raft_consensus.cc:479] T 74b62768046f4b8690aab7d2ec0f803a P 5ee4739615124e308720b4aeddfe29cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } } I0703 01:42:26.615993 29311 leader_election.cc:270] T 74b62768046f4b8690aab7d2ec0f803a P 5ee4739615124e308720b4aeddfe29cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:42:26.616216 29313 raft_consensus.cc:2730] T 74b62768046f4b8690aab7d2ec0f803a P 5ee4739615124e308720b4aeddfe29cb: Snoozing failure detection for 2.003s (election complete) I0703 01:42:26.616315 29313 raft_consensus.cc:2537] T 74b62768046f4b8690aab7d2ec0f803a P 5ee4739615124e308720b4aeddfe29cb [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:42:26.616503 29311 ts_tablet_manager.cc:1073] T 74b62768046f4b8690aab7d2ec0f803a P 5ee4739615124e308720b4aeddfe29cb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s I0703 01:42:26.616500 29313 raft_consensus.cc:593] T 74b62768046f4b8690aab7d2ec0f803a P 5ee4739615124e308720b4aeddfe29cb [term 1 LEADER]: Becoming Leader. State: Replica: 5ee4739615124e308720b4aeddfe29cb, State: Running, Role: LEADER I0703 01:42:26.616650 29311 ts_tablet_manager.cc:1028] T f6594322fcdd4cd3a1357b7bda51b648 P 5ee4739615124e308720b4aeddfe29cb: Bootstrapping tablet I0703 01:42:26.616780 29311 tablet_bootstrap.cc:427] T f6594322fcdd4cd3a1357b7bda51b648 P 5ee4739615124e308720b4aeddfe29cb: Bootstrap starting. I0703 01:42:26.616683 29313 consensus_queue.cc:204] T 74b62768046f4b8690aab7d2ec0f803a P 5ee4739615124e308720b4aeddfe29cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } } I0703 01:42:26.618410 29311 tablet_bootstrap.cc:578] T f6594322fcdd4cd3a1357b7bda51b648 P 5ee4739615124e308720b4aeddfe29cb: No blocks or log segments found. Creating new log. I0703 01:42:26.618345 28251 catalog_manager.cc:3798] T 74b62768046f4b8690aab7d2ec0f803a P 5ee4739615124e308720b4aeddfe29cb reported cstate change: term changed from 0 to 1, leader changed from to 5ee4739615124e308720b4aeddfe29cb (127.5.240.193). New cstate: current_term: 1 leader_uuid: "5ee4739615124e308720b4aeddfe29cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } health_report { overall_health: HEALTHY } } } I0703 01:42:26.620823 29311 tablet_bootstrap.cc:427] T f6594322fcdd4cd3a1357b7bda51b648 P 5ee4739615124e308720b4aeddfe29cb: No bootstrap required, opened a new log I0703 01:42:26.620980 29311 ts_tablet_manager.cc:1045] T f6594322fcdd4cd3a1357b7bda51b648 P 5ee4739615124e308720b4aeddfe29cb: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s I0703 01:42:26.621611 29311 raft_consensus.cc:325] T f6594322fcdd4cd3a1357b7bda51b648 P 5ee4739615124e308720b4aeddfe29cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } } I0703 01:42:26.621762 29311 raft_consensus.cc:351] T f6594322fcdd4cd3a1357b7bda51b648 P 5ee4739615124e308720b4aeddfe29cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:42:26.621835 29311 raft_consensus.cc:628] T f6594322fcdd4cd3a1357b7bda51b648 P 5ee4739615124e308720b4aeddfe29cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ee4739615124e308720b4aeddfe29cb, State: Initialized, Role: FOLLOWER I0703 01:42:26.622042 29311 consensus_queue.cc:227] T f6594322fcdd4cd3a1357b7bda51b648 P 5ee4739615124e308720b4aeddfe29cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } } I0703 01:42:26.622140 29311 raft_consensus.cc:365] T f6594322fcdd4cd3a1357b7bda51b648 P 5ee4739615124e308720b4aeddfe29cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:42:26.622251 29311 raft_consensus.cc:457] T f6594322fcdd4cd3a1357b7bda51b648 P 5ee4739615124e308720b4aeddfe29cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:42:26.622351 29311 raft_consensus.cc:2730] T f6594322fcdd4cd3a1357b7bda51b648 P 5ee4739615124e308720b4aeddfe29cb: Snoozing failure detection for 1.510s (starting election) I0703 01:42:26.622447 29311 raft_consensus.cc:2782] T f6594322fcdd4cd3a1357b7bda51b648 P 5ee4739615124e308720b4aeddfe29cb [term 0 FOLLOWER]: Advancing to term 1 I0703 01:42:26.623656 29311 raft_consensus.cc:479] T f6594322fcdd4cd3a1357b7bda51b648 P 5ee4739615124e308720b4aeddfe29cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } } I0703 01:42:26.623795 29311 leader_election.cc:270] T f6594322fcdd4cd3a1357b7bda51b648 P 5ee4739615124e308720b4aeddfe29cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:42:26.623991 29313 raft_consensus.cc:2730] T f6594322fcdd4cd3a1357b7bda51b648 P 5ee4739615124e308720b4aeddfe29cb: Snoozing failure detection for 1.512s (election complete) I0703 01:42:26.624109 29313 raft_consensus.cc:2537] T f6594322fcdd4cd3a1357b7bda51b648 P 5ee4739615124e308720b4aeddfe29cb [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:42:26.624310 29313 raft_consensus.cc:593] T f6594322fcdd4cd3a1357b7bda51b648 P 5ee4739615124e308720b4aeddfe29cb [term 1 LEADER]: Becoming Leader. State: Replica: 5ee4739615124e308720b4aeddfe29cb, State: Running, Role: LEADER I0703 01:42:26.624415 29311 ts_tablet_manager.cc:1073] T f6594322fcdd4cd3a1357b7bda51b648 P 5ee4739615124e308720b4aeddfe29cb: Time spent starting tablet: real 0.003s user 0.002s sys 0.002s I0703 01:42:26.624511 29313 consensus_queue.cc:204] T f6594322fcdd4cd3a1357b7bda51b648 P 5ee4739615124e308720b4aeddfe29cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } } I0703 01:42:26.625942 28251 catalog_manager.cc:3798] T f6594322fcdd4cd3a1357b7bda51b648 P 5ee4739615124e308720b4aeddfe29cb reported cstate change: term changed from 0 to 1, leader changed from to 5ee4739615124e308720b4aeddfe29cb (127.5.240.193). New cstate: current_term: 1 leader_uuid: "5ee4739615124e308720b4aeddfe29cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } health_report { overall_health: HEALTHY } } } W0703 01:42:26.852202 29321 flags.cc:406] Enabled unsafe flag: --never_fsync=true Using table 'rebalance_test_table_4' Generator report time total : 144.331 ms time per row: 0.0721653 ms I0703 01:42:27.089566 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.5.240.195:46219 --local_ip_for_outbound_sockets=127.5.240.195 --webserver_port=43479 --webserver_interface=127.5.240.195 --tserver_master_addrs=127.5.240.254:47851 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:42:27.252547 29341 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:42:27.253664 29341 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:42:27.255672 29341 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0703 01:42:27.255738 29341 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:42:27.255856 29341 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.195 I0703 01:42:27.263285 29341 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.195:46219 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.5.240.195 --webserver_port=43479 --tserver_master_addrs=127.5.240.254:47851 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.195 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:42:27.269971 29341 tablet_server_main.cc:85] Initializing tablet server... I0703 01:42:27.271499 29341 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 227208us I0703 01:42:27.272084 29341 fs_manager.cc:261] Metadata directory not provided I0703 01:42:27.272161 29341 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:42:27.278007 29341 fs_manager.cc:397] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s I0703 01:42:27.278823 29341 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:42:27.279209 29341 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:42:27.282714 29341 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0703 01:42:27.282779 29341 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal uuid: "0ae780b86cc348d3a55bbaf5cf5a6217" format_stamp: "Formatted at 2018-07-03 08:42:18 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:42:27.283650 29341 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:42:27.337286 29341 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:42:27.339670 29341 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:42:27.340010 29341 tablet_server_main.cc:90] Starting tablet server... I0703 01:42:27.426175 29341 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.195:46219 I0703 01:42:27.426645 29341 webserver.cc:175] Starting webserver on 127.5.240.195:43479 I0703 01:42:27.426689 29341 webserver.cc:186] Document root disabled I0703 01:42:27.427469 29341 webserver.cc:313] Webserver started. Bound to: http://127.5.240.195:43479/ I0703 01:42:27.428140 29341 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb I0703 01:42:27.430480 29341 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:42:27.437077 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 29341 I0703 01:42:27.437463 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.5.240.196:50368 --local_ip_for_outbound_sockets=127.5.240.196 --webserver_port=37872 --webserver_interface=127.5.240.196 --tserver_master_addrs=127.5.240.254:47851 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0703 01:42:27.441632 29460 heartbeater.cc:346] Connected to a master server at 127.5.240.254:47851 I0703 01:42:27.441799 29460 heartbeater.cc:426] Registering TS with master... I0703 01:42:27.442288 29460 heartbeater.cc:475] Master 127.5.240.254:47851 requested a full tablet report, sending... I0703 01:42:27.443393 28253 ts_manager.cc:104] Re-registered known tserver with Master: 0ae780b86cc348d3a55bbaf5cf5a6217 (127.5.240.195:46219) I0703 01:42:27.444263 28253 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.195:40747 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:42:27.602959 29464 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:42:27.603996 29464 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:42:27.606003 29464 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0703 01:42:27.606068 29464 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:42:27.606170 29464 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.196 I0703 01:42:27.613767 29464 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.196:50368 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb --webserver_interface=127.5.240.196 --webserver_port=37872 --tserver_master_addrs=127.5.240.254:47851 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.196 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:42:27.620388 29464 tablet_server_main.cc:85] Initializing tablet server... I0703 01:42:27.621959 29464 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 227208us I0703 01:42:27.622537 29464 fs_manager.cc:261] Metadata directory not provided I0703 01:42:27.622628 29464 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:42:27.628387 29464 fs_manager.cc:397] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s I0703 01:42:27.629194 29464 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:42:27.629688 29464 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:42:27.633087 29464 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0703 01:42:27.633152 29464 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal uuid: "35f0266742fc4707a8cc10c6437b84ed" format_stamp: "Formatted at 2018-07-03 08:42:19 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:42:27.634014 29464 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:42:27.675752 29464 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:42:27.677170 29464 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:42:27.677448 29464 tablet_server_main.cc:90] Starting tablet server... I0703 01:42:27.763339 29464 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.196:50368 I0703 01:42:27.763756 29464 webserver.cc:175] Starting webserver on 127.5.240.196:37872 I0703 01:42:27.763797 29464 webserver.cc:186] Document root disabled I0703 01:42:27.764966 29464 webserver.cc:313] Webserver started. Bound to: http://127.5.240.196:37872/ I0703 01:42:27.765597 29464 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb I0703 01:42:27.767215 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 29464 I0703 01:42:27.767779 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data --block_manager=log --rpc_bind_addresses=127.5.240.197:41466 --local_ip_for_outbound_sockets=127.5.240.197 --webserver_port=33425 --webserver_interface=127.5.240.197 --tserver_master_addrs=127.5.240.254:47851 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0703 01:42:27.771823 29464 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:42:27.780071 29583 heartbeater.cc:346] Connected to a master server at 127.5.240.254:47851 I0703 01:42:27.780262 29583 heartbeater.cc:426] Registering TS with master... I0703 01:42:27.780787 29583 heartbeater.cc:475] Master 127.5.240.254:47851 requested a full tablet report, sending... I0703 01:42:27.781986 28253 ts_manager.cc:104] Re-registered known tserver with Master: 35f0266742fc4707a8cc10c6437b84ed (127.5.240.196:50368) I0703 01:42:27.783244 28253 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.196:59868 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:42:27.931705 29586 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:42:27.932691 29586 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:42:27.934710 29586 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0703 01:42:27.934761 29586 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:42:27.934849 29586 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.197 I0703 01:42:27.942168 29586 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.197:41466 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb --webserver_interface=127.5.240.197 --webserver_port=33425 --tserver_master_addrs=127.5.240.254:47851 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.197 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:42:27.948842 29586 tablet_server_main.cc:85] Initializing tablet server... I0703 01:42:27.950336 29586 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 227208us I0703 01:42:27.950906 29586 fs_manager.cc:261] Metadata directory not provided I0703 01:42:27.950971 29586 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:42:27.957105 29586 fs_manager.cc:397] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s I0703 01:42:27.957890 29586 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:42:27.958266 29586 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:42:27.961297 29586 fs_manager.cc:417] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s I0703 01:42:27.961354 29586 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal uuid: "e98d29f36f2347f29170e987cb83ced4" format_stamp: "Formatted at 2018-07-03 08:42:19 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:42:27.962196 29586 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:42:28.030428 29586 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:42:28.032336 29586 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:42:28.032667 29586 tablet_server_main.cc:90] Starting tablet server... I0703 01:42:28.122445 29586 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.197:41466 I0703 01:42:28.122931 29586 webserver.cc:175] Starting webserver on 127.5.240.197:33425 I0703 01:42:28.122977 29586 webserver.cc:186] Document root disabled I0703 01:42:28.123677 29586 webserver.cc:313] Webserver started. Bound to: http://127.5.240.197:33425/ I0703 01:42:28.124300 29586 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb I0703 01:42:28.126633 29586 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:42:28.128028 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 29586 I0703 01:42:28.128581 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data --block_manager=log --rpc_bind_addresses=127.5.240.198:39152 --local_ip_for_outbound_sockets=127.5.240.198 --webserver_port=40981 --webserver_interface=127.5.240.198 --tserver_master_addrs=127.5.240.254:47851 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0703 01:42:28.138048 29705 heartbeater.cc:346] Connected to a master server at 127.5.240.254:47851 I0703 01:42:28.138216 29705 heartbeater.cc:426] Registering TS with master... I0703 01:42:28.138720 29705 heartbeater.cc:475] Master 127.5.240.254:47851 requested a full tablet report, sending... I0703 01:42:28.139876 28253 ts_manager.cc:104] Re-registered known tserver with Master: e98d29f36f2347f29170e987cb83ced4 (127.5.240.197:41466) I0703 01:42:28.141151 28253 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.197:52096 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:42:28.294234 29708 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:42:28.295260 29708 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:42:28.297314 29708 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0703 01:42:28.297379 29708 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:42:28.297484 29708 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.198 I0703 01:42:28.305140 29708 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.198:39152 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data/info.pb --webserver_interface=127.5.240.198 --webserver_port=40981 --tserver_master_addrs=127.5.240.254:47851 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.198 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:42:28.311748 29708 tablet_server_main.cc:85] Initializing tablet server... I0703 01:42:28.313258 29708 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 227708us I0703 01:42:28.313853 29708 fs_manager.cc:261] Metadata directory not provided I0703 01:42:28.313930 29708 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:42:28.319800 29708 fs_manager.cc:397] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s I0703 01:42:28.320566 29708 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:42:28.320960 29708 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:42:28.324416 29708 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s I0703 01:42:28.324479 29708 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/wal uuid: "575620f4889c4704b7a7c00da34b6a53" format_stamp: "Formatted at 2018-07-03 08:42:19 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:42:28.325330 29708 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:42:28.378985 29708 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:42:28.380844 29708 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:42:28.381114 29708 tablet_server_main.cc:90] Starting tablet server... I0703 01:42:28.446302 29460 heartbeater.cc:467] Master 127.5.240.254:47851 was elected leader, sending a full tablet report... I0703 01:42:28.465857 29708 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.198:39152 I0703 01:42:28.466254 29708 webserver.cc:175] Starting webserver on 127.5.240.198:40981 I0703 01:42:28.466292 29708 webserver.cc:186] Document root disabled I0703 01:42:28.467458 29708 webserver.cc:313] Webserver started. Bound to: http://127.5.240.198:40981/ I0703 01:42:28.468102 29708 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data/info.pb I0703 01:42:28.468664 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 29708 I0703 01:42:28.469182 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data --block_manager=log --rpc_bind_addresses=127.5.240.199:33903 --local_ip_for_outbound_sockets=127.5.240.199 --webserver_port=47396 --webserver_interface=127.5.240.199 --tserver_master_addrs=127.5.240.254:47851 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0703 01:42:28.470190 29708 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:42:28.482919 29827 heartbeater.cc:346] Connected to a master server at 127.5.240.254:47851 I0703 01:42:28.483067 29827 heartbeater.cc:426] Registering TS with master... I0703 01:42:28.483474 29827 heartbeater.cc:475] Master 127.5.240.254:47851 requested a full tablet report, sending... I0703 01:42:28.484467 28253 ts_manager.cc:104] Re-registered known tserver with Master: 575620f4889c4704b7a7c00da34b6a53 (127.5.240.198:39152) I0703 01:42:28.485327 28253 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.198:45849 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:42:28.634641 29830 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:42:28.635668 29830 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:42:28.637743 29830 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0703 01:42:28.637809 29830 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:42:28.637913 29830 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.199 I0703 01:42:28.645599 29830 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.199:33903 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data/info.pb --webserver_interface=127.5.240.199 --webserver_port=47396 --tserver_master_addrs=127.5.240.254:47851 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.199 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:42:28.652364 29830 tablet_server_main.cc:85] Initializing tablet server... I0703 01:42:28.653908 29830 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 227708us I0703 01:42:28.654471 29830 fs_manager.cc:261] Metadata directory not provided I0703 01:42:28.654546 29830 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:42:28.660365 29830 fs_manager.cc:397] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s I0703 01:42:28.661187 29830 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:42:28.661609 29830 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:42:28.665172 29830 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0703 01:42:28.665241 29830 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/wal uuid: "6505fcf874c440788e377dac9aa99209" format_stamp: "Formatted at 2018-07-03 08:42:20 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:42:28.666128 29830 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:42:28.716106 29830 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:42:28.717509 29830 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:42:28.717787 29830 tablet_server_main.cc:90] Starting tablet server... I0703 01:42:28.785614 29583 heartbeater.cc:467] Master 127.5.240.254:47851 was elected leader, sending a full tablet report... I0703 01:42:28.803244 29830 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.199:33903 I0703 01:42:28.803704 29830 webserver.cc:175] Starting webserver on 127.5.240.199:47396 I0703 01:42:28.803752 29830 webserver.cc:186] Document root disabled I0703 01:42:28.804917 29830 webserver.cc:313] Webserver started. Bound to: http://127.5.240.199:47396/ I0703 01:42:28.805559 29830 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data/info.pb I0703 01:42:28.807454 29830 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:42:28.811022 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 29830 I0703 01:42:28.819015 29949 heartbeater.cc:346] Connected to a master server at 127.5.240.254:47851 I0703 01:42:28.819185 29949 heartbeater.cc:426] Registering TS with master... I0703 01:42:28.819697 29949 heartbeater.cc:475] Master 127.5.240.254:47851 requested a full tablet report, sending... I0703 01:42:28.820745 28253 ts_manager.cc:104] Re-registered known tserver with Master: 6505fcf874c440788e377dac9aa99209 (127.5.240.199:33903) I0703 01:42:28.821635 28253 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.199:42019 I0703 01:42:29.143239 29705 heartbeater.cc:467] Master 127.5.240.254:47851 was elected leader, sending a full tablet report... I0703 01:42:29.299810 28377 consensus_queue.cc:204] T 45b3833ac0664a8d9889ab14b3d0dd69 P 5ee4739615124e308720b4aeddfe29cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "e98d29f36f2347f29170e987cb83ced4" member_type: NON_VOTER last_known_addr { host: "127.5.240.197" port: 41466 } attrs { promote: true } } I0703 01:42:29.301982 30023 raft_consensus.cc:2684] T 45b3833ac0664a8d9889ab14b3d0dd69 P 5ee4739615124e308720b4aeddfe29cb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e98d29f36f2347f29170e987cb83ced4 (127.5.240.197) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "e98d29f36f2347f29170e987cb83ced4" member_type: NON_VOTER last_known_addr { host: "127.5.240.197" port: 41466 } attrs { promote: true } } } I0703 01:42:29.306901 28251 catalog_manager.cc:3798] T 45b3833ac0664a8d9889ab14b3d0dd69 P 5ee4739615124e308720b4aeddfe29cb reported cstate change: config changed from index -1 to 4, NON_VOTER e98d29f36f2347f29170e987cb83ced4 (127.5.240.197) added. New cstate: current_term: 1 leader_uuid: "5ee4739615124e308720b4aeddfe29cb" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e98d29f36f2347f29170e987cb83ced4" member_type: NON_VOTER last_known_addr { host: "127.5.240.197" port: 41466 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } W0703 01:42:29.308187 28310 consensus_peers.cc:433] T 45b3833ac0664a8d9889ab14b3d0dd69 P 5ee4739615124e308720b4aeddfe29cb -> Peer e98d29f36f2347f29170e987cb83ced4 (127.5.240.197:41466): Couldn't send request to peer e98d29f36f2347f29170e987cb83ced4 for tablet 45b3833ac0664a8d9889ab14b3d0dd69. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 45b3833ac0664a8d9889ab14b3d0dd69. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:42:29.323663 28500 consensus_queue.cc:204] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 31c252b54241470ba1115181c3672664 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } attrs { replace: true } } peers { permanent_uuid: "6505fcf874c440788e377dac9aa99209" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 33903 } attrs { promote: true } } I0703 01:42:29.325955 30035 raft_consensus.cc:2684] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 31c252b54241470ba1115181c3672664 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6505fcf874c440788e377dac9aa99209 (127.5.240.199) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } attrs { replace: true } } peers { permanent_uuid: "6505fcf874c440788e377dac9aa99209" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 33903 } attrs { promote: true } } } I0703 01:42:29.330744 28251 catalog_manager.cc:3798] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 31c252b54241470ba1115181c3672664 reported cstate change: config changed from index -1 to 4, NON_VOTER 6505fcf874c440788e377dac9aa99209 (127.5.240.199) added. New cstate: current_term: 1 leader_uuid: "31c252b54241470ba1115181c3672664" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6505fcf874c440788e377dac9aa99209" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 33903 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } W0703 01:42:29.334060 28432 consensus_peers.cc:433] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 31c252b54241470ba1115181c3672664 -> Peer 6505fcf874c440788e377dac9aa99209 (127.5.240.199:33903): Couldn't send request to peer 6505fcf874c440788e377dac9aa99209 for tablet 41c52d5ecddf4e5897acf3fe5ee44f34. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 41c52d5ecddf4e5897acf3fe5ee44f34. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:42:29.344934 28377 consensus_queue.cc:204] T 107baa5504214d5f8362cef92ed80291 P 5ee4739615124e308720b4aeddfe29cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "35f0266742fc4707a8cc10c6437b84ed" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 50368 } attrs { promote: true } } I0703 01:42:29.346335 30023 raft_consensus.cc:2684] T 107baa5504214d5f8362cef92ed80291 P 5ee4739615124e308720b4aeddfe29cb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 35f0266742fc4707a8cc10c6437b84ed (127.5.240.196) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "35f0266742fc4707a8cc10c6437b84ed" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 50368 } attrs { promote: true } } } I0703 01:42:29.349759 28251 catalog_manager.cc:3798] T 107baa5504214d5f8362cef92ed80291 P 5ee4739615124e308720b4aeddfe29cb reported cstate change: config changed from index -1 to 4, NON_VOTER 35f0266742fc4707a8cc10c6437b84ed (127.5.240.196) added. New cstate: current_term: 1 leader_uuid: "5ee4739615124e308720b4aeddfe29cb" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35f0266742fc4707a8cc10c6437b84ed" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 50368 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } W0703 01:42:29.351888 28310 consensus_peers.cc:433] T 107baa5504214d5f8362cef92ed80291 P 5ee4739615124e308720b4aeddfe29cb -> Peer 35f0266742fc4707a8cc10c6437b84ed (127.5.240.196:50368): Couldn't send request to peer 35f0266742fc4707a8cc10c6437b84ed for tablet 107baa5504214d5f8362cef92ed80291. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 107baa5504214d5f8362cef92ed80291. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:42:29.359392 28500 consensus_queue.cc:204] T ef65b31bc12e40c2b3f1e4714cc4218e P 31c252b54241470ba1115181c3672664 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } attrs { replace: true } } peers { permanent_uuid: "575620f4889c4704b7a7c00da34b6a53" member_type: NON_VOTER last_known_addr { host: "127.5.240.198" port: 39152 } attrs { promote: true } } I0703 01:42:29.360882 30036 raft_consensus.cc:2684] T ef65b31bc12e40c2b3f1e4714cc4218e P 31c252b54241470ba1115181c3672664 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 575620f4889c4704b7a7c00da34b6a53 (127.5.240.198) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } attrs { replace: true } } peers { permanent_uuid: "575620f4889c4704b7a7c00da34b6a53" member_type: NON_VOTER last_known_addr { host: "127.5.240.198" port: 39152 } attrs { promote: true } } } I0703 01:42:29.365388 28251 catalog_manager.cc:3798] T ef65b31bc12e40c2b3f1e4714cc4218e P 31c252b54241470ba1115181c3672664 reported cstate change: config changed from index -1 to 4, NON_VOTER 575620f4889c4704b7a7c00da34b6a53 (127.5.240.198) added. New cstate: current_term: 1 leader_uuid: "31c252b54241470ba1115181c3672664" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "575620f4889c4704b7a7c00da34b6a53" member_type: NON_VOTER last_known_addr { host: "127.5.240.198" port: 39152 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } W0703 01:42:29.366847 28433 consensus_peers.cc:433] T ef65b31bc12e40c2b3f1e4714cc4218e P 31c252b54241470ba1115181c3672664 -> Peer 575620f4889c4704b7a7c00da34b6a53 (127.5.240.198:39152): Couldn't send request to peer 575620f4889c4704b7a7c00da34b6a53 for tablet ef65b31bc12e40c2b3f1e4714cc4218e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ef65b31bc12e40c2b3f1e4714cc4218e. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:42:29.373885 28377 consensus_queue.cc:204] T 4d69010513df4b68840b90be3161f402 P 5ee4739615124e308720b4aeddfe29cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "0ae780b86cc348d3a55bbaf5cf5a6217" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 46219 } attrs { promote: true } } I0703 01:42:29.375773 30023 raft_consensus.cc:2684] T 4d69010513df4b68840b90be3161f402 P 5ee4739615124e308720b4aeddfe29cb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0ae780b86cc348d3a55bbaf5cf5a6217 (127.5.240.195) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "0ae780b86cc348d3a55bbaf5cf5a6217" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 46219 } attrs { promote: true } } } I0703 01:42:29.379779 28251 catalog_manager.cc:3798] T 4d69010513df4b68840b90be3161f402 P 5ee4739615124e308720b4aeddfe29cb reported cstate change: config changed from index -1 to 4, NON_VOTER 0ae780b86cc348d3a55bbaf5cf5a6217 (127.5.240.195) added. New cstate: current_term: 1 leader_uuid: "5ee4739615124e308720b4aeddfe29cb" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ae780b86cc348d3a55bbaf5cf5a6217" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 46219 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } W0703 01:42:29.385232 28309 consensus_peers.cc:433] T 4d69010513df4b68840b90be3161f402 P 5ee4739615124e308720b4aeddfe29cb -> Peer 0ae780b86cc348d3a55bbaf5cf5a6217 (127.5.240.195:46219): Couldn't send request to peer 0ae780b86cc348d3a55bbaf5cf5a6217 for tablet 4d69010513df4b68840b90be3161f402. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4d69010513df4b68840b90be3161f402. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:42:29.388932 28500 consensus_queue.cc:204] T 775bd6dae9ed48ebbbb9df088d833772 P 31c252b54241470ba1115181c3672664 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } attrs { replace: true } } peers { permanent_uuid: "e98d29f36f2347f29170e987cb83ced4" member_type: NON_VOTER last_known_addr { host: "127.5.240.197" port: 41466 } attrs { promote: true } } I0703 01:42:29.390348 30036 raft_consensus.cc:2684] T 775bd6dae9ed48ebbbb9df088d833772 P 31c252b54241470ba1115181c3672664 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e98d29f36f2347f29170e987cb83ced4 (127.5.240.197) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } attrs { replace: true } } peers { permanent_uuid: "e98d29f36f2347f29170e987cb83ced4" member_type: NON_VOTER last_known_addr { host: "127.5.240.197" port: 41466 } attrs { promote: true } } } I0703 01:42:29.393785 28251 catalog_manager.cc:3798] T 775bd6dae9ed48ebbbb9df088d833772 P 31c252b54241470ba1115181c3672664 reported cstate change: config changed from index -1 to 4, NON_VOTER e98d29f36f2347f29170e987cb83ced4 (127.5.240.197) added. New cstate: current_term: 1 leader_uuid: "31c252b54241470ba1115181c3672664" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e98d29f36f2347f29170e987cb83ced4" member_type: NON_VOTER last_known_addr { host: "127.5.240.197" port: 41466 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } W0703 01:42:29.394989 28433 consensus_peers.cc:433] T 775bd6dae9ed48ebbbb9df088d833772 P 31c252b54241470ba1115181c3672664 -> Peer e98d29f36f2347f29170e987cb83ced4 (127.5.240.197:41466): Couldn't send request to peer e98d29f36f2347f29170e987cb83ced4 for tablet 775bd6dae9ed48ebbbb9df088d833772. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 775bd6dae9ed48ebbbb9df088d833772. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:42:29.402076 28377 consensus_queue.cc:204] T b4acd291b108434da9b8791c61fd1dc2 P 5ee4739615124e308720b4aeddfe29cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "6505fcf874c440788e377dac9aa99209" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 33903 } attrs { promote: true } } I0703 01:42:29.403643 30023 raft_consensus.cc:2684] T b4acd291b108434da9b8791c61fd1dc2 P 5ee4739615124e308720b4aeddfe29cb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6505fcf874c440788e377dac9aa99209 (127.5.240.199) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "6505fcf874c440788e377dac9aa99209" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 33903 } attrs { promote: true } } } I0703 01:42:29.407124 28251 catalog_manager.cc:3798] T b4acd291b108434da9b8791c61fd1dc2 P 5ee4739615124e308720b4aeddfe29cb reported cstate change: config changed from index -1 to 4, NON_VOTER 6505fcf874c440788e377dac9aa99209 (127.5.240.199) added. New cstate: current_term: 1 leader_uuid: "5ee4739615124e308720b4aeddfe29cb" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6505fcf874c440788e377dac9aa99209" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 33903 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } W0703 01:42:29.408582 28309 consensus_peers.cc:433] T b4acd291b108434da9b8791c61fd1dc2 P 5ee4739615124e308720b4aeddfe29cb -> Peer 6505fcf874c440788e377dac9aa99209 (127.5.240.199:33903): Couldn't send request to peer 6505fcf874c440788e377dac9aa99209 for tablet b4acd291b108434da9b8791c61fd1dc2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b4acd291b108434da9b8791c61fd1dc2. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:42:29.415352 28500 consensus_queue.cc:204] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 31c252b54241470ba1115181c3672664 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } attrs { replace: true } } peers { permanent_uuid: "35f0266742fc4707a8cc10c6437b84ed" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 50368 } attrs { promote: true } } I0703 01:42:29.416666 30036 raft_consensus.cc:2684] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 31c252b54241470ba1115181c3672664 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 35f0266742fc4707a8cc10c6437b84ed (127.5.240.196) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } attrs { replace: true } } peers { permanent_uuid: "35f0266742fc4707a8cc10c6437b84ed" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 50368 } attrs { promote: true } } } I0703 01:42:29.419836 28251 catalog_manager.cc:3798] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 31c252b54241470ba1115181c3672664 reported cstate change: config changed from index -1 to 4, NON_VOTER 35f0266742fc4707a8cc10c6437b84ed (127.5.240.196) added. New cstate: current_term: 1 leader_uuid: "31c252b54241470ba1115181c3672664" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35f0266742fc4707a8cc10c6437b84ed" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 50368 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } W0703 01:42:29.421425 28433 consensus_peers.cc:433] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 31c252b54241470ba1115181c3672664 -> Peer 35f0266742fc4707a8cc10c6437b84ed (127.5.240.196:50368): Couldn't send request to peer 35f0266742fc4707a8cc10c6437b84ed for tablet 1cd5cbfe70ec4f8a957fa074ae08fcb1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1cd5cbfe70ec4f8a957fa074ae08fcb1. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:42:29.428678 28377 consensus_queue.cc:204] T 74b62768046f4b8690aab7d2ec0f803a P 5ee4739615124e308720b4aeddfe29cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "575620f4889c4704b7a7c00da34b6a53" member_type: NON_VOTER last_known_addr { host: "127.5.240.198" port: 39152 } attrs { promote: true } } I0703 01:42:29.430198 30023 raft_consensus.cc:2684] T 74b62768046f4b8690aab7d2ec0f803a P 5ee4739615124e308720b4aeddfe29cb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 575620f4889c4704b7a7c00da34b6a53 (127.5.240.198) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "575620f4889c4704b7a7c00da34b6a53" member_type: NON_VOTER last_known_addr { host: "127.5.240.198" port: 39152 } attrs { promote: true } } } I0703 01:42:29.433425 28253 catalog_manager.cc:3798] T 74b62768046f4b8690aab7d2ec0f803a P 5ee4739615124e308720b4aeddfe29cb reported cstate change: config changed from index -1 to 4, NON_VOTER 575620f4889c4704b7a7c00da34b6a53 (127.5.240.198) added. New cstate: current_term: 1 leader_uuid: "5ee4739615124e308720b4aeddfe29cb" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "575620f4889c4704b7a7c00da34b6a53" member_type: NON_VOTER last_known_addr { host: "127.5.240.198" port: 39152 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } W0703 01:42:29.435166 28310 consensus_peers.cc:433] T 74b62768046f4b8690aab7d2ec0f803a P 5ee4739615124e308720b4aeddfe29cb -> Peer 575620f4889c4704b7a7c00da34b6a53 (127.5.240.198:39152): Couldn't send request to peer 575620f4889c4704b7a7c00da34b6a53 for tablet 74b62768046f4b8690aab7d2ec0f803a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 74b62768046f4b8690aab7d2ec0f803a. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:42:29.487104 29827 heartbeater.cc:467] Master 127.5.240.254:47851 was elected leader, sending a full tablet report... I0703 01:42:29.713675 30144 ts_tablet_manager.cc:669] T 45b3833ac0664a8d9889ab14b3d0dd69 P e98d29f36f2347f29170e987cb83ced4: Initiating tablet copy from peer 5ee4739615124e308720b4aeddfe29cb (127.5.240.193:54206) I0703 01:42:29.713930 30144 tablet_copy_client.cc:239] T 45b3833ac0664a8d9889ab14b3d0dd69 P e98d29f36f2347f29170e987cb83ced4: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.193:54206 I0703 01:42:29.721681 28397 tablet_copy_service.cc:135] P 5ee4739615124e308720b4aeddfe29cb: Received BeginTabletCopySession request for tablet 45b3833ac0664a8d9889ab14b3d0dd69 from peer e98d29f36f2347f29170e987cb83ced4 ({username='jenkins'} at 127.5.240.197:38481) I0703 01:42:29.721856 28397 tablet_copy_service.cc:156] P 5ee4739615124e308720b4aeddfe29cb: Beginning new tablet copy session on tablet 45b3833ac0664a8d9889ab14b3d0dd69 from peer e98d29f36f2347f29170e987cb83ced4 at {username='jenkins'} at 127.5.240.197:38481: session id = e98d29f36f2347f29170e987cb83ced4-45b3833ac0664a8d9889ab14b3d0dd69 I0703 01:42:29.723769 28397 tablet_copy_source_session.cc:202] T 45b3833ac0664a8d9889ab14b3d0dd69 P 5ee4739615124e308720b4aeddfe29cb: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:42:29.726752 30144 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 45b3833ac0664a8d9889ab14b3d0dd69. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:29.734700 30144 ts_tablet_manager.cc:1173] T 45b3833ac0664a8d9889ab14b3d0dd69 P e98d29f36f2347f29170e987cb83ced4: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:42:29.734941 30144 tablet_copy_client.cc:566] T 45b3833ac0664a8d9889ab14b3d0dd69 P e98d29f36f2347f29170e987cb83ced4: tablet copy: Starting download of 0 data blocks... I0703 01:42:29.735136 30144 tablet_copy_client.cc:529] T 45b3833ac0664a8d9889ab14b3d0dd69 P e98d29f36f2347f29170e987cb83ced4: tablet copy: Starting download of 1 WAL segments... I0703 01:42:29.737454 30144 tablet_copy_client.cc:405] T 45b3833ac0664a8d9889ab14b3d0dd69 P e98d29f36f2347f29170e987cb83ced4: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:42:29.739310 30144 ts_tablet_manager.cc:1028] T 45b3833ac0664a8d9889ab14b3d0dd69 P e98d29f36f2347f29170e987cb83ced4: Bootstrapping tablet I0703 01:42:29.739559 30144 tablet_bootstrap.cc:427] T 45b3833ac0664a8d9889ab14b3d0dd69 P e98d29f36f2347f29170e987cb83ced4: Bootstrap starting. I0703 01:42:29.747252 30144 log.cc:525] T 45b3833ac0664a8d9889ab14b3d0dd69 P e98d29f36f2347f29170e987cb83ced4: Log is configured to *not* fsync() on all Append() calls I0703 01:42:29.786098 30144 tablet_bootstrap.cc:427] T 45b3833ac0664a8d9889ab14b3d0dd69 P e98d29f36f2347f29170e987cb83ced4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=656 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:42:29.786492 30144 tablet_bootstrap.cc:427] T 45b3833ac0664a8d9889ab14b3d0dd69 P e98d29f36f2347f29170e987cb83ced4: Bootstrap complete. I0703 01:42:29.786722 30144 ts_tablet_manager.cc:1045] T 45b3833ac0664a8d9889ab14b3d0dd69 P e98d29f36f2347f29170e987cb83ced4: Time spent bootstrapping tablet: real 0.047s user 0.044s sys 0.003s I0703 01:42:29.790407 30144 raft_consensus.cc:325] T 45b3833ac0664a8d9889ab14b3d0dd69 P e98d29f36f2347f29170e987cb83ced4 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "e98d29f36f2347f29170e987cb83ced4" member_type: NON_VOTER last_known_addr { host: "127.5.240.197" port: 41466 } attrs { promote: true } } I0703 01:42:29.790706 30144 raft_consensus.cc:628] T 45b3833ac0664a8d9889ab14b3d0dd69 P e98d29f36f2347f29170e987cb83ced4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e98d29f36f2347f29170e987cb83ced4, State: Initialized, Role: LEARNER I0703 01:42:29.790980 30144 consensus_queue.cc:227] T 45b3833ac0664a8d9889ab14b3d0dd69 P e98d29f36f2347f29170e987cb83ced4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "e98d29f36f2347f29170e987cb83ced4" member_type: NON_VOTER last_known_addr { host: "127.5.240.197" port: 41466 } attrs { promote: true } } I0703 01:42:29.792150 30144 ts_tablet_manager.cc:1073] T 45b3833ac0664a8d9889ab14b3d0dd69 P e98d29f36f2347f29170e987cb83ced4: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s I0703 01:42:29.793110 28397 tablet_copy_service.cc:325] P 5ee4739615124e308720b4aeddfe29cb: Request end of tablet copy session e98d29f36f2347f29170e987cb83ced4-45b3833ac0664a8d9889ab14b3d0dd69 received from {username='jenkins'} at 127.5.240.197:38481 I0703 01:42:29.793197 28397 tablet_copy_service.cc:413] P 5ee4739615124e308720b4aeddfe29cb: ending tablet copy session e98d29f36f2347f29170e987cb83ced4-45b3833ac0664a8d9889ab14b3d0dd69 on tablet 45b3833ac0664a8d9889ab14b3d0dd69 with peer e98d29f36f2347f29170e987cb83ced4 I0703 01:42:29.812279 30175 ts_tablet_manager.cc:669] T 4d69010513df4b68840b90be3161f402 P 0ae780b86cc348d3a55bbaf5cf5a6217: Initiating tablet copy from peer 5ee4739615124e308720b4aeddfe29cb (127.5.240.193:54206) I0703 01:42:29.812508 30175 tablet_copy_client.cc:239] T 4d69010513df4b68840b90be3161f402 P 0ae780b86cc348d3a55bbaf5cf5a6217: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.193:54206 I0703 01:42:29.816999 28397 tablet_copy_service.cc:135] P 5ee4739615124e308720b4aeddfe29cb: Received BeginTabletCopySession request for tablet 4d69010513df4b68840b90be3161f402 from peer 0ae780b86cc348d3a55bbaf5cf5a6217 ({username='jenkins'} at 127.5.240.195:54164) I0703 01:42:29.817090 28397 tablet_copy_service.cc:156] P 5ee4739615124e308720b4aeddfe29cb: Beginning new tablet copy session on tablet 4d69010513df4b68840b90be3161f402 from peer 0ae780b86cc348d3a55bbaf5cf5a6217 at {username='jenkins'} at 127.5.240.195:54164: session id = 0ae780b86cc348d3a55bbaf5cf5a6217-4d69010513df4b68840b90be3161f402 I0703 01:42:29.818683 28397 tablet_copy_source_session.cc:202] T 4d69010513df4b68840b90be3161f402 P 5ee4739615124e308720b4aeddfe29cb: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:42:29.821274 30175 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 4d69010513df4b68840b90be3161f402. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:29.823813 29949 heartbeater.cc:467] Master 127.5.240.254:47851 was elected leader, sending a full tablet report... I0703 01:42:29.829932 30175 ts_tablet_manager.cc:1173] T 4d69010513df4b68840b90be3161f402 P 0ae780b86cc348d3a55bbaf5cf5a6217: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:42:29.830164 30175 tablet_copy_client.cc:566] T 4d69010513df4b68840b90be3161f402 P 0ae780b86cc348d3a55bbaf5cf5a6217: tablet copy: Starting download of 0 data blocks... I0703 01:42:29.830447 30175 tablet_copy_client.cc:529] T 4d69010513df4b68840b90be3161f402 P 0ae780b86cc348d3a55bbaf5cf5a6217: tablet copy: Starting download of 1 WAL segments... I0703 01:42:29.832243 30175 tablet_copy_client.cc:405] T 4d69010513df4b68840b90be3161f402 P 0ae780b86cc348d3a55bbaf5cf5a6217: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:42:29.833904 30175 ts_tablet_manager.cc:1028] T 4d69010513df4b68840b90be3161f402 P 0ae780b86cc348d3a55bbaf5cf5a6217: Bootstrapping tablet I0703 01:42:29.834101 30175 tablet_bootstrap.cc:427] T 4d69010513df4b68840b90be3161f402 P 0ae780b86cc348d3a55bbaf5cf5a6217: Bootstrap starting. I0703 01:42:29.839938 30177 ts_tablet_manager.cc:669] T b4acd291b108434da9b8791c61fd1dc2 P 6505fcf874c440788e377dac9aa99209: Initiating tablet copy from peer 5ee4739615124e308720b4aeddfe29cb (127.5.240.193:54206) I0703 01:42:29.840113 30175 log.cc:525] T 4d69010513df4b68840b90be3161f402 P 0ae780b86cc348d3a55bbaf5cf5a6217: Log is configured to *not* fsync() on all Append() calls I0703 01:42:29.840153 30177 tablet_copy_client.cc:239] T b4acd291b108434da9b8791c61fd1dc2 P 6505fcf874c440788e377dac9aa99209: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.193:54206 I0703 01:42:29.844866 28397 tablet_copy_service.cc:135] P 5ee4739615124e308720b4aeddfe29cb: Received BeginTabletCopySession request for tablet b4acd291b108434da9b8791c61fd1dc2 from peer 6505fcf874c440788e377dac9aa99209 ({username='jenkins'} at 127.5.240.199:54734) I0703 01:42:29.844967 28397 tablet_copy_service.cc:156] P 5ee4739615124e308720b4aeddfe29cb: Beginning new tablet copy session on tablet b4acd291b108434da9b8791c61fd1dc2 from peer 6505fcf874c440788e377dac9aa99209 at {username='jenkins'} at 127.5.240.199:54734: session id = 6505fcf874c440788e377dac9aa99209-b4acd291b108434da9b8791c61fd1dc2 I0703 01:42:29.846433 28397 tablet_copy_source_session.cc:202] T b4acd291b108434da9b8791c61fd1dc2 P 5ee4739615124e308720b4aeddfe29cb: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:42:29.848914 30177 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet b4acd291b108434da9b8791c61fd1dc2. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:29.856220 30177 ts_tablet_manager.cc:1173] T b4acd291b108434da9b8791c61fd1dc2 P 6505fcf874c440788e377dac9aa99209: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:42:29.856462 30177 tablet_copy_client.cc:566] T b4acd291b108434da9b8791c61fd1dc2 P 6505fcf874c440788e377dac9aa99209: tablet copy: Starting download of 0 data blocks... I0703 01:42:29.856756 30177 tablet_copy_client.cc:529] T b4acd291b108434da9b8791c61fd1dc2 P 6505fcf874c440788e377dac9aa99209: tablet copy: Starting download of 1 WAL segments... I0703 01:42:29.858723 30177 tablet_copy_client.cc:405] T b4acd291b108434da9b8791c61fd1dc2 P 6505fcf874c440788e377dac9aa99209: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:42:29.860266 30177 ts_tablet_manager.cc:1028] T b4acd291b108434da9b8791c61fd1dc2 P 6505fcf874c440788e377dac9aa99209: Bootstrapping tablet I0703 01:42:29.860466 30177 tablet_bootstrap.cc:427] T b4acd291b108434da9b8791c61fd1dc2 P 6505fcf874c440788e377dac9aa99209: Bootstrap starting. I0703 01:42:29.866370 30177 log.cc:525] T b4acd291b108434da9b8791c61fd1dc2 P 6505fcf874c440788e377dac9aa99209: Log is configured to *not* fsync() on all Append() calls I0703 01:42:29.884184 30175 tablet_bootstrap.cc:427] T 4d69010513df4b68840b90be3161f402 P 0ae780b86cc348d3a55bbaf5cf5a6217: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=691 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:42:29.884575 30175 tablet_bootstrap.cc:427] T 4d69010513df4b68840b90be3161f402 P 0ae780b86cc348d3a55bbaf5cf5a6217: Bootstrap complete. I0703 01:42:29.884824 30175 ts_tablet_manager.cc:1045] T 4d69010513df4b68840b90be3161f402 P 0ae780b86cc348d3a55bbaf5cf5a6217: Time spent bootstrapping tablet: real 0.051s user 0.048s sys 0.002s I0703 01:42:29.888654 30175 raft_consensus.cc:325] T 4d69010513df4b68840b90be3161f402 P 0ae780b86cc348d3a55bbaf5cf5a6217 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "0ae780b86cc348d3a55bbaf5cf5a6217" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 46219 } attrs { promote: true } } I0703 01:42:29.888931 30175 raft_consensus.cc:628] T 4d69010513df4b68840b90be3161f402 P 0ae780b86cc348d3a55bbaf5cf5a6217 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0ae780b86cc348d3a55bbaf5cf5a6217, State: Initialized, Role: LEARNER I0703 01:42:29.889194 30175 consensus_queue.cc:227] T 4d69010513df4b68840b90be3161f402 P 0ae780b86cc348d3a55bbaf5cf5a6217 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "0ae780b86cc348d3a55bbaf5cf5a6217" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 46219 } attrs { promote: true } } I0703 01:42:29.890734 30175 ts_tablet_manager.cc:1073] T 4d69010513df4b68840b90be3161f402 P 0ae780b86cc348d3a55bbaf5cf5a6217: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s I0703 01:42:29.891659 28397 tablet_copy_service.cc:325] P 5ee4739615124e308720b4aeddfe29cb: Request end of tablet copy session 0ae780b86cc348d3a55bbaf5cf5a6217-4d69010513df4b68840b90be3161f402 received from {username='jenkins'} at 127.5.240.195:54164 I0703 01:42:29.891765 28397 tablet_copy_service.cc:413] P 5ee4739615124e308720b4aeddfe29cb: ending tablet copy session 0ae780b86cc348d3a55bbaf5cf5a6217-4d69010513df4b68840b90be3161f402 on tablet 4d69010513df4b68840b90be3161f402 with peer 0ae780b86cc348d3a55bbaf5cf5a6217 I0703 01:42:29.903290 30177 tablet_bootstrap.cc:427] T b4acd291b108434da9b8791c61fd1dc2 P 6505fcf874c440788e377dac9aa99209: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=691 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:42:29.903903 30177 tablet_bootstrap.cc:427] T b4acd291b108434da9b8791c61fd1dc2 P 6505fcf874c440788e377dac9aa99209: Bootstrap complete. I0703 01:42:29.904110 30177 ts_tablet_manager.cc:1045] T b4acd291b108434da9b8791c61fd1dc2 P 6505fcf874c440788e377dac9aa99209: Time spent bootstrapping tablet: real 0.044s user 0.039s sys 0.004s I0703 01:42:29.907737 30177 raft_consensus.cc:325] T b4acd291b108434da9b8791c61fd1dc2 P 6505fcf874c440788e377dac9aa99209 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "6505fcf874c440788e377dac9aa99209" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 33903 } attrs { promote: true } } I0703 01:42:29.908164 30177 raft_consensus.cc:628] T b4acd291b108434da9b8791c61fd1dc2 P 6505fcf874c440788e377dac9aa99209 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6505fcf874c440788e377dac9aa99209, State: Initialized, Role: LEARNER I0703 01:42:29.908394 30177 consensus_queue.cc:227] T b4acd291b108434da9b8791c61fd1dc2 P 6505fcf874c440788e377dac9aa99209 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "6505fcf874c440788e377dac9aa99209" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 33903 } attrs { promote: true } } I0703 01:42:29.909373 30177 ts_tablet_manager.cc:1073] T b4acd291b108434da9b8791c61fd1dc2 P 6505fcf874c440788e377dac9aa99209: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s I0703 01:42:29.910084 28397 tablet_copy_service.cc:325] P 5ee4739615124e308720b4aeddfe29cb: Request end of tablet copy session 6505fcf874c440788e377dac9aa99209-b4acd291b108434da9b8791c61fd1dc2 received from {username='jenkins'} at 127.5.240.199:54734 I0703 01:42:29.910152 28397 tablet_copy_service.cc:413] P 5ee4739615124e308720b4aeddfe29cb: ending tablet copy session 6505fcf874c440788e377dac9aa99209-b4acd291b108434da9b8791c61fd1dc2 on tablet b4acd291b108434da9b8791c61fd1dc2 with peer 6505fcf874c440788e377dac9aa99209 I0703 01:42:29.920827 30183 ts_tablet_manager.cc:669] T 107baa5504214d5f8362cef92ed80291 P 35f0266742fc4707a8cc10c6437b84ed: Initiating tablet copy from peer 5ee4739615124e308720b4aeddfe29cb (127.5.240.193:54206) I0703 01:42:29.921109 30183 tablet_copy_client.cc:239] T 107baa5504214d5f8362cef92ed80291 P 35f0266742fc4707a8cc10c6437b84ed: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.193:54206 I0703 01:42:29.925501 28397 tablet_copy_service.cc:135] P 5ee4739615124e308720b4aeddfe29cb: Received BeginTabletCopySession request for tablet 107baa5504214d5f8362cef92ed80291 from peer 35f0266742fc4707a8cc10c6437b84ed ({username='jenkins'} at 127.5.240.196:38747) I0703 01:42:29.925599 28397 tablet_copy_service.cc:156] P 5ee4739615124e308720b4aeddfe29cb: Beginning new tablet copy session on tablet 107baa5504214d5f8362cef92ed80291 from peer 35f0266742fc4707a8cc10c6437b84ed at {username='jenkins'} at 127.5.240.196:38747: session id = 35f0266742fc4707a8cc10c6437b84ed-107baa5504214d5f8362cef92ed80291 I0703 01:42:29.927139 28397 tablet_copy_source_session.cc:202] T 107baa5504214d5f8362cef92ed80291 P 5ee4739615124e308720b4aeddfe29cb: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:42:29.929682 30183 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 107baa5504214d5f8362cef92ed80291. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:29.937762 30183 ts_tablet_manager.cc:1173] T 107baa5504214d5f8362cef92ed80291 P 35f0266742fc4707a8cc10c6437b84ed: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:42:29.937994 30183 tablet_copy_client.cc:566] T 107baa5504214d5f8362cef92ed80291 P 35f0266742fc4707a8cc10c6437b84ed: tablet copy: Starting download of 0 data blocks... I0703 01:42:29.938205 30183 tablet_copy_client.cc:529] T 107baa5504214d5f8362cef92ed80291 P 35f0266742fc4707a8cc10c6437b84ed: tablet copy: Starting download of 1 WAL segments... I0703 01:42:29.940028 30183 tablet_copy_client.cc:405] T 107baa5504214d5f8362cef92ed80291 P 35f0266742fc4707a8cc10c6437b84ed: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:42:29.941121 30177 ts_tablet_manager.cc:669] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 6505fcf874c440788e377dac9aa99209: Initiating tablet copy from peer 31c252b54241470ba1115181c3672664 (127.5.240.194:38625) I0703 01:42:29.941257 30177 tablet_copy_client.cc:239] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 6505fcf874c440788e377dac9aa99209: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.194:38625 I0703 01:42:29.942353 30183 ts_tablet_manager.cc:1028] T 107baa5504214d5f8362cef92ed80291 P 35f0266742fc4707a8cc10c6437b84ed: Bootstrapping tablet I0703 01:42:29.942565 30183 tablet_bootstrap.cc:427] T 107baa5504214d5f8362cef92ed80291 P 35f0266742fc4707a8cc10c6437b84ed: Bootstrap starting. I0703 01:42:29.945586 28520 tablet_copy_service.cc:135] P 31c252b54241470ba1115181c3672664: Received BeginTabletCopySession request for tablet 41c52d5ecddf4e5897acf3fe5ee44f34 from peer 6505fcf874c440788e377dac9aa99209 ({username='jenkins'} at 127.5.240.199:55135) I0703 01:42:29.945736 28520 tablet_copy_service.cc:156] P 31c252b54241470ba1115181c3672664: Beginning new tablet copy session on tablet 41c52d5ecddf4e5897acf3fe5ee44f34 from peer 6505fcf874c440788e377dac9aa99209 at {username='jenkins'} at 127.5.240.199:55135: session id = 6505fcf874c440788e377dac9aa99209-41c52d5ecddf4e5897acf3fe5ee44f34 I0703 01:42:29.948571 28520 tablet_copy_source_session.cc:202] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 31c252b54241470ba1115181c3672664: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:42:29.949883 30177 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 41c52d5ecddf4e5897acf3fe5ee44f34. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:29.950562 30183 log.cc:525] T 107baa5504214d5f8362cef92ed80291 P 35f0266742fc4707a8cc10c6437b84ed: Log is configured to *not* fsync() on all Append() calls I0703 01:42:29.950841 30144 ts_tablet_manager.cc:669] T 775bd6dae9ed48ebbbb9df088d833772 P e98d29f36f2347f29170e987cb83ced4: Initiating tablet copy from peer 31c252b54241470ba1115181c3672664 (127.5.240.194:38625) I0703 01:42:29.950989 30144 tablet_copy_client.cc:239] T 775bd6dae9ed48ebbbb9df088d833772 P e98d29f36f2347f29170e987cb83ced4: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.194:38625 I0703 01:42:29.953917 30177 ts_tablet_manager.cc:1173] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 6505fcf874c440788e377dac9aa99209: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:42:29.954087 30177 tablet_copy_client.cc:566] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 6505fcf874c440788e377dac9aa99209: tablet copy: Starting download of 0 data blocks... I0703 01:42:29.954267 30177 tablet_copy_client.cc:529] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 6505fcf874c440788e377dac9aa99209: tablet copy: Starting download of 1 WAL segments... I0703 01:42:29.956524 28520 tablet_copy_service.cc:135] P 31c252b54241470ba1115181c3672664: Received BeginTabletCopySession request for tablet 775bd6dae9ed48ebbbb9df088d833772 from peer e98d29f36f2347f29170e987cb83ced4 ({username='jenkins'} at 127.5.240.197:52570) I0703 01:42:29.956634 28520 tablet_copy_service.cc:156] P 31c252b54241470ba1115181c3672664: Beginning new tablet copy session on tablet 775bd6dae9ed48ebbbb9df088d833772 from peer e98d29f36f2347f29170e987cb83ced4 at {username='jenkins'} at 127.5.240.197:52570: session id = e98d29f36f2347f29170e987cb83ced4-775bd6dae9ed48ebbbb9df088d833772 I0703 01:42:29.956786 30177 tablet_copy_client.cc:405] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 6505fcf874c440788e377dac9aa99209: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:42:29.958902 30177 ts_tablet_manager.cc:1028] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 6505fcf874c440788e377dac9aa99209: Bootstrapping tablet I0703 01:42:29.959024 28520 tablet_copy_source_session.cc:202] T 775bd6dae9ed48ebbbb9df088d833772 P 31c252b54241470ba1115181c3672664: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:42:29.959105 30177 tablet_bootstrap.cc:427] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 6505fcf874c440788e377dac9aa99209: Bootstrap starting. I0703 01:42:29.960386 30144 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 775bd6dae9ed48ebbbb9df088d833772. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:29.964648 30144 ts_tablet_manager.cc:1173] T 775bd6dae9ed48ebbbb9df088d833772 P e98d29f36f2347f29170e987cb83ced4: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:42:29.965010 30144 tablet_copy_client.cc:566] T 775bd6dae9ed48ebbbb9df088d833772 P e98d29f36f2347f29170e987cb83ced4: tablet copy: Starting download of 0 data blocks... I0703 01:42:29.965771 30144 tablet_copy_client.cc:529] T 775bd6dae9ed48ebbbb9df088d833772 P e98d29f36f2347f29170e987cb83ced4: tablet copy: Starting download of 1 WAL segments... I0703 01:42:29.967131 30187 ts_tablet_manager.cc:669] T 74b62768046f4b8690aab7d2ec0f803a P 575620f4889c4704b7a7c00da34b6a53: Initiating tablet copy from peer 5ee4739615124e308720b4aeddfe29cb (127.5.240.193:54206) I0703 01:42:29.967408 30187 tablet_copy_client.cc:239] T 74b62768046f4b8690aab7d2ec0f803a P 575620f4889c4704b7a7c00da34b6a53: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.193:54206 I0703 01:42:29.968219 30144 tablet_copy_client.cc:405] T 775bd6dae9ed48ebbbb9df088d833772 P e98d29f36f2347f29170e987cb83ced4: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:42:29.970296 30144 ts_tablet_manager.cc:1028] T 775bd6dae9ed48ebbbb9df088d833772 P e98d29f36f2347f29170e987cb83ced4: Bootstrapping tablet I0703 01:42:29.970417 30144 tablet_bootstrap.cc:427] T 775bd6dae9ed48ebbbb9df088d833772 P e98d29f36f2347f29170e987cb83ced4: Bootstrap starting. I0703 01:42:29.977669 28397 tablet_copy_service.cc:135] P 5ee4739615124e308720b4aeddfe29cb: Received BeginTabletCopySession request for tablet 74b62768046f4b8690aab7d2ec0f803a from peer 575620f4889c4704b7a7c00da34b6a53 ({username='jenkins'} at 127.5.240.198:44960) I0703 01:42:29.977773 28397 tablet_copy_service.cc:156] P 5ee4739615124e308720b4aeddfe29cb: Beginning new tablet copy session on tablet 74b62768046f4b8690aab7d2ec0f803a from peer 575620f4889c4704b7a7c00da34b6a53 at {username='jenkins'} at 127.5.240.198:44960: session id = 575620f4889c4704b7a7c00da34b6a53-74b62768046f4b8690aab7d2ec0f803a I0703 01:42:29.980095 28397 tablet_copy_source_session.cc:202] T 74b62768046f4b8690aab7d2ec0f803a P 5ee4739615124e308720b4aeddfe29cb: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:42:29.983711 30187 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 74b62768046f4b8690aab7d2ec0f803a. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:29.993100 30191 ts_tablet_manager.cc:669] T ef65b31bc12e40c2b3f1e4714cc4218e P 575620f4889c4704b7a7c00da34b6a53: Initiating tablet copy from peer 31c252b54241470ba1115181c3672664 (127.5.240.194:38625) I0703 01:42:29.994657 30191 tablet_copy_client.cc:239] T ef65b31bc12e40c2b3f1e4714cc4218e P 575620f4889c4704b7a7c00da34b6a53: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.194:38625 I0703 01:42:29.998713 30187 ts_tablet_manager.cc:1173] T 74b62768046f4b8690aab7d2ec0f803a P 575620f4889c4704b7a7c00da34b6a53: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:42:29.999315 30187 tablet_copy_client.cc:566] T 74b62768046f4b8690aab7d2ec0f803a P 575620f4889c4704b7a7c00da34b6a53: tablet copy: Starting download of 0 data blocks... I0703 01:42:29.999639 30187 tablet_copy_client.cc:529] T 74b62768046f4b8690aab7d2ec0f803a P 575620f4889c4704b7a7c00da34b6a53: tablet copy: Starting download of 1 WAL segments... I0703 01:42:30.002117 30187 tablet_copy_client.cc:405] T 74b62768046f4b8690aab7d2ec0f803a P 575620f4889c4704b7a7c00da34b6a53: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:42:30.010071 30187 ts_tablet_manager.cc:1028] T 74b62768046f4b8690aab7d2ec0f803a P 575620f4889c4704b7a7c00da34b6a53: Bootstrapping tablet I0703 01:42:30.010336 30187 tablet_bootstrap.cc:427] T 74b62768046f4b8690aab7d2ec0f803a P 575620f4889c4704b7a7c00da34b6a53: Bootstrap starting. I0703 01:42:30.016638 28520 tablet_copy_service.cc:135] P 31c252b54241470ba1115181c3672664: Received BeginTabletCopySession request for tablet ef65b31bc12e40c2b3f1e4714cc4218e from peer 575620f4889c4704b7a7c00da34b6a53 ({username='jenkins'} at 127.5.240.198:51171) I0703 01:42:30.016736 28520 tablet_copy_service.cc:156] P 31c252b54241470ba1115181c3672664: Beginning new tablet copy session on tablet ef65b31bc12e40c2b3f1e4714cc4218e from peer 575620f4889c4704b7a7c00da34b6a53 at {username='jenkins'} at 127.5.240.198:51171: session id = 575620f4889c4704b7a7c00da34b6a53-ef65b31bc12e40c2b3f1e4714cc4218e I0703 01:42:30.019316 28520 tablet_copy_source_session.cc:202] T ef65b31bc12e40c2b3f1e4714cc4218e P 31c252b54241470ba1115181c3672664: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:42:30.020627 30191 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet ef65b31bc12e40c2b3f1e4714cc4218e. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:30.024984 30191 ts_tablet_manager.cc:1173] T ef65b31bc12e40c2b3f1e4714cc4218e P 575620f4889c4704b7a7c00da34b6a53: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:42:30.028184 30191 tablet_copy_client.cc:566] T ef65b31bc12e40c2b3f1e4714cc4218e P 575620f4889c4704b7a7c00da34b6a53: tablet copy: Starting download of 0 data blocks... I0703 01:42:30.028507 30191 tablet_copy_client.cc:529] T ef65b31bc12e40c2b3f1e4714cc4218e P 575620f4889c4704b7a7c00da34b6a53: tablet copy: Starting download of 1 WAL segments... I0703 01:42:30.027737 30187 log.cc:525] T 74b62768046f4b8690aab7d2ec0f803a P 575620f4889c4704b7a7c00da34b6a53: Log is configured to *not* fsync() on all Append() calls I0703 01:42:30.030962 30201 ts_tablet_manager.cc:669] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 35f0266742fc4707a8cc10c6437b84ed: Initiating tablet copy from peer 31c252b54241470ba1115181c3672664 (127.5.240.194:38625) I0703 01:42:30.031158 30201 tablet_copy_client.cc:239] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 35f0266742fc4707a8cc10c6437b84ed: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.194:38625 I0703 01:42:30.037593 28520 tablet_copy_service.cc:135] P 31c252b54241470ba1115181c3672664: Received BeginTabletCopySession request for tablet 1cd5cbfe70ec4f8a957fa074ae08fcb1 from peer 35f0266742fc4707a8cc10c6437b84ed ({username='jenkins'} at 127.5.240.196:53692) I0703 01:42:30.037703 28520 tablet_copy_service.cc:156] P 31c252b54241470ba1115181c3672664: Beginning new tablet copy session on tablet 1cd5cbfe70ec4f8a957fa074ae08fcb1 from peer 35f0266742fc4707a8cc10c6437b84ed at {username='jenkins'} at 127.5.240.196:53692: session id = 35f0266742fc4707a8cc10c6437b84ed-1cd5cbfe70ec4f8a957fa074ae08fcb1 I0703 01:42:30.038792 30191 tablet_copy_client.cc:405] T ef65b31bc12e40c2b3f1e4714cc4218e P 575620f4889c4704b7a7c00da34b6a53: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:42:30.040108 28520 tablet_copy_source_session.cc:202] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 31c252b54241470ba1115181c3672664: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:42:30.041560 30201 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 1cd5cbfe70ec4f8a957fa074ae08fcb1. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:30.042861 30191 ts_tablet_manager.cc:1028] T ef65b31bc12e40c2b3f1e4714cc4218e P 575620f4889c4704b7a7c00da34b6a53: Bootstrapping tablet I0703 01:42:30.043010 30191 tablet_bootstrap.cc:427] T ef65b31bc12e40c2b3f1e4714cc4218e P 575620f4889c4704b7a7c00da34b6a53: Bootstrap starting. I0703 01:42:30.046618 30201 ts_tablet_manager.cc:1173] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 35f0266742fc4707a8cc10c6437b84ed: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:42:30.046839 30201 tablet_copy_client.cc:566] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 35f0266742fc4707a8cc10c6437b84ed: tablet copy: Starting download of 0 data blocks... I0703 01:42:30.047060 30201 tablet_copy_client.cc:529] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 35f0266742fc4707a8cc10c6437b84ed: tablet copy: Starting download of 1 WAL segments... I0703 01:42:30.049334 30201 tablet_copy_client.cc:405] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 35f0266742fc4707a8cc10c6437b84ed: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:42:30.055665 30201 ts_tablet_manager.cc:1028] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 35f0266742fc4707a8cc10c6437b84ed: Bootstrapping tablet I0703 01:42:30.055855 30201 tablet_bootstrap.cc:427] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 35f0266742fc4707a8cc10c6437b84ed: Bootstrap starting. I0703 01:42:30.057446 30183 tablet_bootstrap.cc:427] T 107baa5504214d5f8362cef92ed80291 P 35f0266742fc4707a8cc10c6437b84ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=691 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:42:30.064976 30177 tablet_bootstrap.cc:427] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 6505fcf874c440788e377dac9aa99209: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=653 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:42:30.069988 30183 tablet_bootstrap.cc:427] T 107baa5504214d5f8362cef92ed80291 P 35f0266742fc4707a8cc10c6437b84ed: Bootstrap complete. I0703 01:42:30.070324 30183 ts_tablet_manager.cc:1045] T 107baa5504214d5f8362cef92ed80291 P 35f0266742fc4707a8cc10c6437b84ed: Time spent bootstrapping tablet: real 0.128s user 0.077s sys 0.004s I0703 01:42:30.077322 30183 raft_consensus.cc:325] T 107baa5504214d5f8362cef92ed80291 P 35f0266742fc4707a8cc10c6437b84ed [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "35f0266742fc4707a8cc10c6437b84ed" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 50368 } attrs { promote: true } } I0703 01:42:30.077913 30177 tablet_bootstrap.cc:427] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 6505fcf874c440788e377dac9aa99209: Bootstrap complete. I0703 01:42:30.078112 30177 ts_tablet_manager.cc:1045] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 6505fcf874c440788e377dac9aa99209: Time spent bootstrapping tablet: real 0.119s user 0.063s sys 0.010s I0703 01:42:30.078892 30177 raft_consensus.cc:325] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 6505fcf874c440788e377dac9aa99209 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } attrs { replace: true } } peers { permanent_uuid: "6505fcf874c440788e377dac9aa99209" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 33903 } attrs { promote: true } } I0703 01:42:30.079164 30177 raft_consensus.cc:628] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 6505fcf874c440788e377dac9aa99209 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6505fcf874c440788e377dac9aa99209, State: Initialized, Role: LEARNER I0703 01:42:30.079327 30177 consensus_queue.cc:227] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 6505fcf874c440788e377dac9aa99209 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } attrs { replace: true } } peers { permanent_uuid: "6505fcf874c440788e377dac9aa99209" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 33903 } attrs { promote: true } } I0703 01:42:30.080490 30144 tablet_bootstrap.cc:427] T 775bd6dae9ed48ebbbb9df088d833772 P e98d29f36f2347f29170e987cb83ced4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=653 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:42:30.080889 30144 tablet_bootstrap.cc:427] T 775bd6dae9ed48ebbbb9df088d833772 P e98d29f36f2347f29170e987cb83ced4: Bootstrap complete. I0703 01:42:30.081077 30144 ts_tablet_manager.cc:1045] T 775bd6dae9ed48ebbbb9df088d833772 P e98d29f36f2347f29170e987cb83ced4: Time spent bootstrapping tablet: real 0.111s user 0.059s sys 0.005s I0703 01:42:30.081799 30144 raft_consensus.cc:325] T 775bd6dae9ed48ebbbb9df088d833772 P e98d29f36f2347f29170e987cb83ced4 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } attrs { replace: true } } peers { permanent_uuid: "e98d29f36f2347f29170e987cb83ced4" member_type: NON_VOTER last_known_addr { host: "127.5.240.197" port: 41466 } attrs { promote: true } } I0703 01:42:30.082006 30144 raft_consensus.cc:628] T 775bd6dae9ed48ebbbb9df088d833772 P e98d29f36f2347f29170e987cb83ced4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e98d29f36f2347f29170e987cb83ced4, State: Initialized, Role: LEARNER I0703 01:42:30.082141 30144 consensus_queue.cc:227] T 775bd6dae9ed48ebbbb9df088d833772 P e98d29f36f2347f29170e987cb83ced4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } attrs { replace: true } } peers { permanent_uuid: "e98d29f36f2347f29170e987cb83ced4" member_type: NON_VOTER last_known_addr { host: "127.5.240.197" port: 41466 } attrs { promote: true } } I0703 01:42:30.083001 30144 ts_tablet_manager.cc:1073] T 775bd6dae9ed48ebbbb9df088d833772 P e98d29f36f2347f29170e987cb83ced4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s I0703 01:42:30.084676 28520 tablet_copy_service.cc:325] P 31c252b54241470ba1115181c3672664: Request end of tablet copy session e98d29f36f2347f29170e987cb83ced4-775bd6dae9ed48ebbbb9df088d833772 received from {username='jenkins'} at 127.5.240.197:52570 I0703 01:42:30.084753 28520 tablet_copy_service.cc:413] P 31c252b54241470ba1115181c3672664: ending tablet copy session e98d29f36f2347f29170e987cb83ced4-775bd6dae9ed48ebbbb9df088d833772 on tablet 775bd6dae9ed48ebbbb9df088d833772 with peer e98d29f36f2347f29170e987cb83ced4 I0703 01:42:30.086220 30183 raft_consensus.cc:628] T 107baa5504214d5f8362cef92ed80291 P 35f0266742fc4707a8cc10c6437b84ed [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 35f0266742fc4707a8cc10c6437b84ed, State: Initialized, Role: LEARNER I0703 01:42:30.086545 30183 consensus_queue.cc:227] T 107baa5504214d5f8362cef92ed80291 P 35f0266742fc4707a8cc10c6437b84ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "35f0266742fc4707a8cc10c6437b84ed" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 50368 } attrs { promote: true } } I0703 01:42:30.089962 30183 ts_tablet_manager.cc:1073] T 107baa5504214d5f8362cef92ed80291 P 35f0266742fc4707a8cc10c6437b84ed: Time spent starting tablet: real 0.020s user 0.007s sys 0.001s I0703 01:42:30.090854 28397 tablet_copy_service.cc:325] P 5ee4739615124e308720b4aeddfe29cb: Request end of tablet copy session 35f0266742fc4707a8cc10c6437b84ed-107baa5504214d5f8362cef92ed80291 received from {username='jenkins'} at 127.5.240.196:38747 I0703 01:42:30.090947 28397 tablet_copy_service.cc:413] P 5ee4739615124e308720b4aeddfe29cb: ending tablet copy session 35f0266742fc4707a8cc10c6437b84ed-107baa5504214d5f8362cef92ed80291 on tablet 107baa5504214d5f8362cef92ed80291 with peer 35f0266742fc4707a8cc10c6437b84ed I0703 01:42:30.091368 30177 ts_tablet_manager.cc:1073] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 6505fcf874c440788e377dac9aa99209: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s I0703 01:42:30.092244 28520 tablet_copy_service.cc:325] P 31c252b54241470ba1115181c3672664: Request end of tablet copy session 6505fcf874c440788e377dac9aa99209-41c52d5ecddf4e5897acf3fe5ee44f34 received from {username='jenkins'} at 127.5.240.199:55135 I0703 01:42:30.092316 28520 tablet_copy_service.cc:413] P 31c252b54241470ba1115181c3672664: ending tablet copy session 6505fcf874c440788e377dac9aa99209-41c52d5ecddf4e5897acf3fe5ee44f34 on tablet 41c52d5ecddf4e5897acf3fe5ee44f34 with peer 6505fcf874c440788e377dac9aa99209 I0703 01:42:30.149940 30187 tablet_bootstrap.cc:427] T 74b62768046f4b8690aab7d2ec0f803a P 575620f4889c4704b7a7c00da34b6a53: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=653 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:42:30.150516 30187 tablet_bootstrap.cc:427] T 74b62768046f4b8690aab7d2ec0f803a P 575620f4889c4704b7a7c00da34b6a53: Bootstrap complete. I0703 01:42:30.150868 30187 ts_tablet_manager.cc:1045] T 74b62768046f4b8690aab7d2ec0f803a P 575620f4889c4704b7a7c00da34b6a53: Time spent bootstrapping tablet: real 0.141s user 0.064s sys 0.013s I0703 01:42:30.152253 30201 tablet_bootstrap.cc:427] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 35f0266742fc4707a8cc10c6437b84ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=656 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:42:30.152863 30201 tablet_bootstrap.cc:427] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 35f0266742fc4707a8cc10c6437b84ed: Bootstrap complete. I0703 01:42:30.153062 30201 ts_tablet_manager.cc:1045] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 35f0266742fc4707a8cc10c6437b84ed: Time spent bootstrapping tablet: real 0.097s user 0.070s sys 0.003s I0703 01:42:30.153834 30201 raft_consensus.cc:325] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 35f0266742fc4707a8cc10c6437b84ed [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } attrs { replace: true } } peers { permanent_uuid: "35f0266742fc4707a8cc10c6437b84ed" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 50368 } attrs { promote: true } } I0703 01:42:30.154054 30201 raft_consensus.cc:628] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 35f0266742fc4707a8cc10c6437b84ed [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 35f0266742fc4707a8cc10c6437b84ed, State: Initialized, Role: LEARNER I0703 01:42:30.154201 30201 consensus_queue.cc:227] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 35f0266742fc4707a8cc10c6437b84ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } attrs { replace: true } } peers { permanent_uuid: "35f0266742fc4707a8cc10c6437b84ed" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 50368 } attrs { promote: true } } I0703 01:42:30.154966 30201 ts_tablet_manager.cc:1073] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 35f0266742fc4707a8cc10c6437b84ed: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s I0703 01:42:30.155670 28520 tablet_copy_service.cc:325] P 31c252b54241470ba1115181c3672664: Request end of tablet copy session 35f0266742fc4707a8cc10c6437b84ed-1cd5cbfe70ec4f8a957fa074ae08fcb1 received from {username='jenkins'} at 127.5.240.196:53692 I0703 01:42:30.155746 28520 tablet_copy_service.cc:413] P 31c252b54241470ba1115181c3672664: ending tablet copy session 35f0266742fc4707a8cc10c6437b84ed-1cd5cbfe70ec4f8a957fa074ae08fcb1 on tablet 1cd5cbfe70ec4f8a957fa074ae08fcb1 with peer 35f0266742fc4707a8cc10c6437b84ed I0703 01:42:30.160001 30187 raft_consensus.cc:325] T 74b62768046f4b8690aab7d2ec0f803a P 575620f4889c4704b7a7c00da34b6a53 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "575620f4889c4704b7a7c00da34b6a53" member_type: NON_VOTER last_known_addr { host: "127.5.240.198" port: 39152 } attrs { promote: true } } I0703 01:42:30.160387 30187 raft_consensus.cc:628] T 74b62768046f4b8690aab7d2ec0f803a P 575620f4889c4704b7a7c00da34b6a53 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 575620f4889c4704b7a7c00da34b6a53, State: Initialized, Role: LEARNER I0703 01:42:30.160704 30187 consensus_queue.cc:227] T 74b62768046f4b8690aab7d2ec0f803a P 575620f4889c4704b7a7c00da34b6a53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "575620f4889c4704b7a7c00da34b6a53" member_type: NON_VOTER last_known_addr { host: "127.5.240.198" port: 39152 } attrs { promote: true } } I0703 01:42:30.163242 30191 tablet_bootstrap.cc:427] T ef65b31bc12e40c2b3f1e4714cc4218e P 575620f4889c4704b7a7c00da34b6a53: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=691 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:42:30.163784 30191 tablet_bootstrap.cc:427] T ef65b31bc12e40c2b3f1e4714cc4218e P 575620f4889c4704b7a7c00da34b6a53: Bootstrap complete. I0703 01:42:30.163941 30191 ts_tablet_manager.cc:1045] T ef65b31bc12e40c2b3f1e4714cc4218e P 575620f4889c4704b7a7c00da34b6a53: Time spent bootstrapping tablet: real 0.121s user 0.069s sys 0.010s I0703 01:42:30.164572 30191 raft_consensus.cc:325] T ef65b31bc12e40c2b3f1e4714cc4218e P 575620f4889c4704b7a7c00da34b6a53 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } attrs { replace: true } } peers { permanent_uuid: "575620f4889c4704b7a7c00da34b6a53" member_type: NON_VOTER last_known_addr { host: "127.5.240.198" port: 39152 } attrs { promote: true } } I0703 01:42:30.164999 30191 raft_consensus.cc:628] T ef65b31bc12e40c2b3f1e4714cc4218e P 575620f4889c4704b7a7c00da34b6a53 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 575620f4889c4704b7a7c00da34b6a53, State: Initialized, Role: LEARNER I0703 01:42:30.165181 30187 ts_tablet_manager.cc:1073] T 74b62768046f4b8690aab7d2ec0f803a P 575620f4889c4704b7a7c00da34b6a53: Time spent starting tablet: real 0.014s user 0.004s sys 0.005s I0703 01:42:30.165154 30191 consensus_queue.cc:227] T ef65b31bc12e40c2b3f1e4714cc4218e P 575620f4889c4704b7a7c00da34b6a53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } attrs { replace: true } } peers { permanent_uuid: "575620f4889c4704b7a7c00da34b6a53" member_type: NON_VOTER last_known_addr { host: "127.5.240.198" port: 39152 } attrs { promote: true } } I0703 01:42:30.165690 30191 ts_tablet_manager.cc:1073] T ef65b31bc12e40c2b3f1e4714cc4218e P 575620f4889c4704b7a7c00da34b6a53: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s I0703 01:42:30.165994 28397 tablet_copy_service.cc:325] P 5ee4739615124e308720b4aeddfe29cb: Request end of tablet copy session 575620f4889c4704b7a7c00da34b6a53-74b62768046f4b8690aab7d2ec0f803a received from {username='jenkins'} at 127.5.240.198:44960 I0703 01:42:30.166066 28397 tablet_copy_service.cc:413] P 5ee4739615124e308720b4aeddfe29cb: ending tablet copy session 575620f4889c4704b7a7c00da34b6a53-74b62768046f4b8690aab7d2ec0f803a on tablet 74b62768046f4b8690aab7d2ec0f803a with peer 575620f4889c4704b7a7c00da34b6a53 I0703 01:42:30.166512 28520 tablet_copy_service.cc:325] P 31c252b54241470ba1115181c3672664: Request end of tablet copy session 575620f4889c4704b7a7c00da34b6a53-ef65b31bc12e40c2b3f1e4714cc4218e received from {username='jenkins'} at 127.5.240.198:51171 I0703 01:42:30.166604 28520 tablet_copy_service.cc:413] P 31c252b54241470ba1115181c3672664: ending tablet copy session 575620f4889c4704b7a7c00da34b6a53-ef65b31bc12e40c2b3f1e4714cc4218e on tablet ef65b31bc12e40c2b3f1e4714cc4218e with peer 575620f4889c4704b7a7c00da34b6a53 I0703 01:42:30.206523 29662 raft_consensus.cc:1026] T 45b3833ac0664a8d9889ab14b3d0dd69 P e98d29f36f2347f29170e987cb83ced4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[] I0703 01:42:30.260844 29906 raft_consensus.cc:1026] T b4acd291b108434da9b8791c61fd1dc2 P 6505fcf874c440788e377dac9aa99209 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[] I0703 01:42:30.287613 29417 raft_consensus.cc:1026] T 4d69010513df4b68840b90be3161f402 P 0ae780b86cc348d3a55bbaf5cf5a6217 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[] I0703 01:42:30.340656 29783 raft_consensus.cc:1026] T 74b62768046f4b8690aab7d2ec0f803a P 575620f4889c4704b7a7c00da34b6a53 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[] I0703 01:42:30.427218 29540 raft_consensus.cc:1026] T 107baa5504214d5f8362cef92ed80291 P 35f0266742fc4707a8cc10c6437b84ed [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[] I0703 01:42:30.432922 29906 raft_consensus.cc:1026] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 6505fcf874c440788e377dac9aa99209 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[] I0703 01:42:30.433161 29540 raft_consensus.cc:1026] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 35f0266742fc4707a8cc10c6437b84ed [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[] I0703 01:42:30.492213 29783 raft_consensus.cc:1026] T ef65b31bc12e40c2b3f1e4714cc4218e P 575620f4889c4704b7a7c00da34b6a53 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[] I0703 01:42:30.512960 29662 raft_consensus.cc:1026] T 775bd6dae9ed48ebbbb9df088d833772 P e98d29f36f2347f29170e987cb83ced4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[] I0703 01:42:30.640573 30023 raft_consensus.cc:890] T 45b3833ac0664a8d9889ab14b3d0dd69 P 5ee4739615124e308720b4aeddfe29cb: attempting to promote NON_VOTER e98d29f36f2347f29170e987cb83ced4 to VOTER I0703 01:42:30.640889 30023 consensus_peers.cc:455] T 45b3833ac0664a8d9889ab14b3d0dd69 P 5ee4739615124e308720b4aeddfe29cb -> Peer e98d29f36f2347f29170e987cb83ced4 (127.5.240.197:41466): Closing peer: e98d29f36f2347f29170e987cb83ced4 I0703 01:42:30.641167 30023 consensus_queue.cc:204] T 45b3833ac0664a8d9889ab14b3d0dd69 P 5ee4739615124e308720b4aeddfe29cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "e98d29f36f2347f29170e987cb83ced4" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 41466 } attrs { promote: false } } I0703 01:42:30.642756 29662 raft_consensus.cc:1081] T 45b3833ac0664a8d9889ab14b3d0dd69 P e98d29f36f2347f29170e987cb83ced4 [term 1 LEARNER]: Refusing update from remote peer 5ee4739615124e308720b4aeddfe29cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I0703 01:42:30.643345 30245 consensus_queue.cc:962] T 45b3833ac0664a8d9889ab14b3d0dd69 P 5ee4739615124e308720b4aeddfe29cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e98d29f36f2347f29170e987cb83ced4" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 41466 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I0703 01:42:30.646193 30023 raft_consensus.cc:2684] T 45b3833ac0664a8d9889ab14b3d0dd69 P 5ee4739615124e308720b4aeddfe29cb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e98d29f36f2347f29170e987cb83ced4 (127.5.240.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "e98d29f36f2347f29170e987cb83ced4" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 41466 } attrs { promote: false } } } I0703 01:42:30.647799 29662 raft_consensus.cc:2684] T 45b3833ac0664a8d9889ab14b3d0dd69 P e98d29f36f2347f29170e987cb83ced4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e98d29f36f2347f29170e987cb83ced4 (127.5.240.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "e98d29f36f2347f29170e987cb83ced4" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 41466 } attrs { promote: false } } } I0703 01:42:30.649224 28251 catalog_manager.cc:3798] T 45b3833ac0664a8d9889ab14b3d0dd69 P 5ee4739615124e308720b4aeddfe29cb reported cstate change: config changed from index 4 to 5, e98d29f36f2347f29170e987cb83ced4 (127.5.240.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5ee4739615124e308720b4aeddfe29cb" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e98d29f36f2347f29170e987cb83ced4" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 41466 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I0703 01:42:30.691031 30245 raft_consensus.cc:890] T b4acd291b108434da9b8791c61fd1dc2 P 5ee4739615124e308720b4aeddfe29cb: attempting to promote NON_VOTER 6505fcf874c440788e377dac9aa99209 to VOTER I0703 01:42:30.691241 30245 consensus_peers.cc:455] T b4acd291b108434da9b8791c61fd1dc2 P 5ee4739615124e308720b4aeddfe29cb -> Peer 6505fcf874c440788e377dac9aa99209 (127.5.240.199:33903): Closing peer: 6505fcf874c440788e377dac9aa99209 I0703 01:42:30.691402 30245 consensus_queue.cc:204] T b4acd291b108434da9b8791c61fd1dc2 P 5ee4739615124e308720b4aeddfe29cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "6505fcf874c440788e377dac9aa99209" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 33903 } attrs { promote: false } } I0703 01:42:30.693183 29906 raft_consensus.cc:1081] T b4acd291b108434da9b8791c61fd1dc2 P 6505fcf874c440788e377dac9aa99209 [term 1 LEARNER]: Refusing update from remote peer 5ee4739615124e308720b4aeddfe29cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I0703 01:42:30.693951 30244 consensus_queue.cc:962] T b4acd291b108434da9b8791c61fd1dc2 P 5ee4739615124e308720b4aeddfe29cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "6505fcf874c440788e377dac9aa99209" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 33903 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I0703 01:42:30.696841 30245 raft_consensus.cc:2684] T b4acd291b108434da9b8791c61fd1dc2 P 5ee4739615124e308720b4aeddfe29cb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6505fcf874c440788e377dac9aa99209 (127.5.240.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "6505fcf874c440788e377dac9aa99209" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 33903 } attrs { promote: false } } } I0703 01:42:30.699373 29906 raft_consensus.cc:2684] T b4acd291b108434da9b8791c61fd1dc2 P 6505fcf874c440788e377dac9aa99209 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6505fcf874c440788e377dac9aa99209 (127.5.240.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "6505fcf874c440788e377dac9aa99209" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 33903 } attrs { promote: false } } } I0703 01:42:30.702374 28253 catalog_manager.cc:3798] T b4acd291b108434da9b8791c61fd1dc2 P 5ee4739615124e308720b4aeddfe29cb reported cstate change: config changed from index 4 to 5, 6505fcf874c440788e377dac9aa99209 (127.5.240.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5ee4739615124e308720b4aeddfe29cb" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6505fcf874c440788e377dac9aa99209" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 33903 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I0703 01:42:30.730809 28377 tablet_service.cc:1140] Received LeaderStepDown RPC: tablet_id: "45b3833ac0664a8d9889ab14b3d0dd69" dest_uuid: "5ee4739615124e308720b4aeddfe29cb" from {username='jenkins'} at 127.5.240.193:54766 I0703 01:42:30.730959 28377 raft_consensus.cc:552] T 45b3833ac0664a8d9889ab14b3d0dd69 P 5ee4739615124e308720b4aeddfe29cb [term 1 LEADER]: Received request to step down I0703 01:42:30.731006 28377 raft_consensus.cc:628] T 45b3833ac0664a8d9889ab14b3d0dd69 P 5ee4739615124e308720b4aeddfe29cb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5ee4739615124e308720b4aeddfe29cb, State: Running, Role: LEADER I0703 01:42:30.731209 28377 consensus_queue.cc:227] T 45b3833ac0664a8d9889ab14b3d0dd69 P 5ee4739615124e308720b4aeddfe29cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "e98d29f36f2347f29170e987cb83ced4" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 41466 } attrs { promote: false } } I0703 01:42:30.731370 28377 consensus_peers.cc:455] T 45b3833ac0664a8d9889ab14b3d0dd69 P 5ee4739615124e308720b4aeddfe29cb -> Peer e98d29f36f2347f29170e987cb83ced4 (127.5.240.197:41466): Closing peer: e98d29f36f2347f29170e987cb83ced4 I0703 01:42:30.731492 28377 raft_consensus.cc:2730] T 45b3833ac0664a8d9889ab14b3d0dd69 P 5ee4739615124e308720b4aeddfe29cb: Snoozing failure detection for 3.000s (explicit stepdown request) I0703 01:42:30.854351 30245 raft_consensus.cc:890] T 4d69010513df4b68840b90be3161f402 P 5ee4739615124e308720b4aeddfe29cb: attempting to promote NON_VOTER 0ae780b86cc348d3a55bbaf5cf5a6217 to VOTER I0703 01:42:30.854635 30245 consensus_peers.cc:455] T 4d69010513df4b68840b90be3161f402 P 5ee4739615124e308720b4aeddfe29cb -> Peer 0ae780b86cc348d3a55bbaf5cf5a6217 (127.5.240.195:46219): Closing peer: 0ae780b86cc348d3a55bbaf5cf5a6217 I0703 01:42:30.854821 30245 consensus_queue.cc:204] T 4d69010513df4b68840b90be3161f402 P 5ee4739615124e308720b4aeddfe29cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "0ae780b86cc348d3a55bbaf5cf5a6217" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46219 } attrs { promote: false } } I0703 01:42:30.856395 29417 raft_consensus.cc:1081] T 4d69010513df4b68840b90be3161f402 P 0ae780b86cc348d3a55bbaf5cf5a6217 [term 1 LEARNER]: Refusing update from remote peer 5ee4739615124e308720b4aeddfe29cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I0703 01:42:30.856966 30245 consensus_queue.cc:962] T 4d69010513df4b68840b90be3161f402 P 5ee4739615124e308720b4aeddfe29cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ae780b86cc348d3a55bbaf5cf5a6217" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46219 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I0703 01:42:30.859830 30023 raft_consensus.cc:2684] T 4d69010513df4b68840b90be3161f402 P 5ee4739615124e308720b4aeddfe29cb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0ae780b86cc348d3a55bbaf5cf5a6217 (127.5.240.195) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "0ae780b86cc348d3a55bbaf5cf5a6217" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46219 } attrs { promote: false } } } I0703 01:42:30.860714 29417 raft_consensus.cc:2684] T 4d69010513df4b68840b90be3161f402 P 0ae780b86cc348d3a55bbaf5cf5a6217 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0ae780b86cc348d3a55bbaf5cf5a6217 (127.5.240.195) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "0ae780b86cc348d3a55bbaf5cf5a6217" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46219 } attrs { promote: false } } } I0703 01:42:30.863889 28251 catalog_manager.cc:3798] T 4d69010513df4b68840b90be3161f402 P 5ee4739615124e308720b4aeddfe29cb reported cstate change: config changed from index 4 to 5, 0ae780b86cc348d3a55bbaf5cf5a6217 (127.5.240.195) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5ee4739615124e308720b4aeddfe29cb" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ae780b86cc348d3a55bbaf5cf5a6217" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46219 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I0703 01:42:30.912626 30023 raft_consensus.cc:890] T 74b62768046f4b8690aab7d2ec0f803a P 5ee4739615124e308720b4aeddfe29cb: attempting to promote NON_VOTER 575620f4889c4704b7a7c00da34b6a53 to VOTER I0703 01:42:30.912839 30023 consensus_peers.cc:455] T 74b62768046f4b8690aab7d2ec0f803a P 5ee4739615124e308720b4aeddfe29cb -> Peer 575620f4889c4704b7a7c00da34b6a53 (127.5.240.198:39152): Closing peer: 575620f4889c4704b7a7c00da34b6a53 I0703 01:42:30.913015 30023 consensus_queue.cc:204] T 74b62768046f4b8690aab7d2ec0f803a P 5ee4739615124e308720b4aeddfe29cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "575620f4889c4704b7a7c00da34b6a53" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39152 } attrs { promote: false } } I0703 01:42:30.914507 29783 raft_consensus.cc:1081] T 74b62768046f4b8690aab7d2ec0f803a P 575620f4889c4704b7a7c00da34b6a53 [term 1 LEARNER]: Refusing update from remote peer 5ee4739615124e308720b4aeddfe29cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I0703 01:42:30.915081 30023 consensus_queue.cc:962] T 74b62768046f4b8690aab7d2ec0f803a P 5ee4739615124e308720b4aeddfe29cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "575620f4889c4704b7a7c00da34b6a53" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39152 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I0703 01:42:30.917743 30244 raft_consensus.cc:2684] T 74b62768046f4b8690aab7d2ec0f803a P 5ee4739615124e308720b4aeddfe29cb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 575620f4889c4704b7a7c00da34b6a53 (127.5.240.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "575620f4889c4704b7a7c00da34b6a53" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39152 } attrs { promote: false } } } I0703 01:42:30.918547 29783 raft_consensus.cc:2684] T 74b62768046f4b8690aab7d2ec0f803a P 575620f4889c4704b7a7c00da34b6a53 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 575620f4889c4704b7a7c00da34b6a53 (127.5.240.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "575620f4889c4704b7a7c00da34b6a53" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39152 } attrs { promote: false } } } I0703 01:42:30.921187 28253 catalog_manager.cc:3798] T 74b62768046f4b8690aab7d2ec0f803a P 5ee4739615124e308720b4aeddfe29cb reported cstate change: config changed from index 4 to 5, 575620f4889c4704b7a7c00da34b6a53 (127.5.240.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5ee4739615124e308720b4aeddfe29cb" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "575620f4889c4704b7a7c00da34b6a53" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39152 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I0703 01:42:30.936743 30185 raft_consensus.cc:890] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 31c252b54241470ba1115181c3672664: attempting to promote NON_VOTER 6505fcf874c440788e377dac9aa99209 to VOTER I0703 01:42:30.937121 30185 consensus_peers.cc:455] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 31c252b54241470ba1115181c3672664 -> Peer 6505fcf874c440788e377dac9aa99209 (127.5.240.199:33903): Closing peer: 6505fcf874c440788e377dac9aa99209 I0703 01:42:30.937443 30185 consensus_queue.cc:204] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 31c252b54241470ba1115181c3672664 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } attrs { replace: true } } peers { permanent_uuid: "6505fcf874c440788e377dac9aa99209" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 33903 } attrs { promote: false } } I0703 01:42:30.939502 29906 raft_consensus.cc:1081] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 6505fcf874c440788e377dac9aa99209 [term 1 LEARNER]: Refusing update from remote peer 31c252b54241470ba1115181c3672664: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I0703 01:42:30.940086 30278 consensus_queue.cc:962] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 31c252b54241470ba1115181c3672664 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6505fcf874c440788e377dac9aa99209" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 33903 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I0703 01:42:30.942927 30185 raft_consensus.cc:2684] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 31c252b54241470ba1115181c3672664 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6505fcf874c440788e377dac9aa99209 (127.5.240.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } attrs { replace: true } } peers { permanent_uuid: "6505fcf874c440788e377dac9aa99209" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 33903 } attrs { promote: false } } } I0703 01:42:30.944622 29906 raft_consensus.cc:2684] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 6505fcf874c440788e377dac9aa99209 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6505fcf874c440788e377dac9aa99209 (127.5.240.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } attrs { replace: true } } peers { permanent_uuid: "6505fcf874c440788e377dac9aa99209" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 33903 } attrs { promote: false } } } I0703 01:42:30.947206 28251 catalog_manager.cc:3798] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 31c252b54241470ba1115181c3672664 reported cstate change: config changed from index 4 to 5, 6505fcf874c440788e377dac9aa99209 (127.5.240.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "31c252b54241470ba1115181c3672664" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6505fcf874c440788e377dac9aa99209" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 33903 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I0703 01:42:30.948649 30244 raft_consensus.cc:890] T 107baa5504214d5f8362cef92ed80291 P 5ee4739615124e308720b4aeddfe29cb: attempting to promote NON_VOTER 35f0266742fc4707a8cc10c6437b84ed to VOTER I0703 01:42:30.949005 30244 consensus_peers.cc:455] T 107baa5504214d5f8362cef92ed80291 P 5ee4739615124e308720b4aeddfe29cb -> Peer 35f0266742fc4707a8cc10c6437b84ed (127.5.240.196:50368): Closing peer: 35f0266742fc4707a8cc10c6437b84ed I0703 01:42:30.949229 30244 consensus_queue.cc:204] T 107baa5504214d5f8362cef92ed80291 P 5ee4739615124e308720b4aeddfe29cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "35f0266742fc4707a8cc10c6437b84ed" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 50368 } attrs { promote: false } } I0703 01:42:30.951203 29540 raft_consensus.cc:1081] T 107baa5504214d5f8362cef92ed80291 P 35f0266742fc4707a8cc10c6437b84ed [term 1 LEARNER]: Refusing update from remote peer 5ee4739615124e308720b4aeddfe29cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I0703 01:42:30.951836 30023 consensus_queue.cc:962] T 107baa5504214d5f8362cef92ed80291 P 5ee4739615124e308720b4aeddfe29cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "35f0266742fc4707a8cc10c6437b84ed" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 50368 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I0703 01:42:30.954891 30244 raft_consensus.cc:2684] T 107baa5504214d5f8362cef92ed80291 P 5ee4739615124e308720b4aeddfe29cb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 35f0266742fc4707a8cc10c6437b84ed (127.5.240.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "35f0266742fc4707a8cc10c6437b84ed" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 50368 } attrs { promote: false } } } I0703 01:42:30.956701 28377 tablet_service.cc:1140] Received LeaderStepDown RPC: tablet_id: "b4acd291b108434da9b8791c61fd1dc2" dest_uuid: "5ee4739615124e308720b4aeddfe29cb" from {username='jenkins'} at 127.5.240.193:54769 I0703 01:42:30.956871 28377 raft_consensus.cc:552] T b4acd291b108434da9b8791c61fd1dc2 P 5ee4739615124e308720b4aeddfe29cb [term 1 LEADER]: Received request to step down I0703 01:42:30.956960 28377 raft_consensus.cc:628] T b4acd291b108434da9b8791c61fd1dc2 P 5ee4739615124e308720b4aeddfe29cb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5ee4739615124e308720b4aeddfe29cb, State: Running, Role: LEADER I0703 01:42:30.956676 29540 raft_consensus.cc:2684] T 107baa5504214d5f8362cef92ed80291 P 35f0266742fc4707a8cc10c6437b84ed [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 35f0266742fc4707a8cc10c6437b84ed (127.5.240.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "35f0266742fc4707a8cc10c6437b84ed" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 50368 } attrs { promote: false } } } I0703 01:42:30.957154 28377 consensus_queue.cc:227] T b4acd291b108434da9b8791c61fd1dc2 P 5ee4739615124e308720b4aeddfe29cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "6505fcf874c440788e377dac9aa99209" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 33903 } attrs { promote: false } } I0703 01:42:30.957370 28377 consensus_peers.cc:455] T b4acd291b108434da9b8791c61fd1dc2 P 5ee4739615124e308720b4aeddfe29cb -> Peer 6505fcf874c440788e377dac9aa99209 (127.5.240.199:33903): Closing peer: 6505fcf874c440788e377dac9aa99209 I0703 01:42:30.957490 28377 raft_consensus.cc:2730] T b4acd291b108434da9b8791c61fd1dc2 P 5ee4739615124e308720b4aeddfe29cb: Snoozing failure detection for 3.000s (explicit stepdown request) I0703 01:42:30.959260 28253 catalog_manager.cc:3798] T 107baa5504214d5f8362cef92ed80291 P 5ee4739615124e308720b4aeddfe29cb reported cstate change: config changed from index 4 to 5, 35f0266742fc4707a8cc10c6437b84ed (127.5.240.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5ee4739615124e308720b4aeddfe29cb" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35f0266742fc4707a8cc10c6437b84ed" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 50368 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I0703 01:42:30.969564 30185 raft_consensus.cc:890] T ef65b31bc12e40c2b3f1e4714cc4218e P 31c252b54241470ba1115181c3672664: attempting to promote NON_VOTER 575620f4889c4704b7a7c00da34b6a53 to VOTER I0703 01:42:30.969821 30185 consensus_peers.cc:455] T ef65b31bc12e40c2b3f1e4714cc4218e P 31c252b54241470ba1115181c3672664 -> Peer 575620f4889c4704b7a7c00da34b6a53 (127.5.240.198:39152): Closing peer: 575620f4889c4704b7a7c00da34b6a53 I0703 01:42:30.970042 30185 consensus_queue.cc:204] T ef65b31bc12e40c2b3f1e4714cc4218e P 31c252b54241470ba1115181c3672664 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } attrs { replace: true } } peers { permanent_uuid: "575620f4889c4704b7a7c00da34b6a53" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39152 } attrs { promote: false } } I0703 01:42:30.971500 29783 raft_consensus.cc:1081] T ef65b31bc12e40c2b3f1e4714cc4218e P 575620f4889c4704b7a7c00da34b6a53 [term 1 LEARNER]: Refusing update from remote peer 31c252b54241470ba1115181c3672664: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I0703 01:42:30.972084 30278 consensus_queue.cc:962] T ef65b31bc12e40c2b3f1e4714cc4218e P 31c252b54241470ba1115181c3672664 [LEADER]: Connected to new peer: Peer: permanent_uuid: "575620f4889c4704b7a7c00da34b6a53" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39152 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I0703 01:42:30.974344 30185 raft_consensus.cc:2684] T ef65b31bc12e40c2b3f1e4714cc4218e P 31c252b54241470ba1115181c3672664 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 575620f4889c4704b7a7c00da34b6a53 (127.5.240.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } attrs { replace: true } } peers { permanent_uuid: "575620f4889c4704b7a7c00da34b6a53" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39152 } attrs { promote: false } } } I0703 01:42:30.975308 29783 raft_consensus.cc:2684] T ef65b31bc12e40c2b3f1e4714cc4218e P 575620f4889c4704b7a7c00da34b6a53 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 575620f4889c4704b7a7c00da34b6a53 (127.5.240.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } attrs { replace: true } } peers { permanent_uuid: "575620f4889c4704b7a7c00da34b6a53" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39152 } attrs { promote: false } } } I0703 01:42:30.978440 28253 catalog_manager.cc:3798] T ef65b31bc12e40c2b3f1e4714cc4218e P 31c252b54241470ba1115181c3672664 reported cstate change: config changed from index 4 to 5, 575620f4889c4704b7a7c00da34b6a53 (127.5.240.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "31c252b54241470ba1115181c3672664" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "575620f4889c4704b7a7c00da34b6a53" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39152 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I0703 01:42:30.981784 30277 raft_consensus.cc:890] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 31c252b54241470ba1115181c3672664: attempting to promote NON_VOTER 35f0266742fc4707a8cc10c6437b84ed to VOTER I0703 01:42:30.981988 30277 consensus_peers.cc:455] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 31c252b54241470ba1115181c3672664 -> Peer 35f0266742fc4707a8cc10c6437b84ed (127.5.240.196:50368): Closing peer: 35f0266742fc4707a8cc10c6437b84ed I0703 01:42:30.982190 30277 consensus_queue.cc:204] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 31c252b54241470ba1115181c3672664 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } attrs { replace: true } } peers { permanent_uuid: "35f0266742fc4707a8cc10c6437b84ed" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 50368 } attrs { promote: false } } I0703 01:42:30.983930 28377 tablet_service.cc:1140] Received LeaderStepDown RPC: tablet_id: "4d69010513df4b68840b90be3161f402" dest_uuid: "5ee4739615124e308720b4aeddfe29cb" from {username='jenkins'} at 127.5.240.193:54772 I0703 01:42:30.984058 28377 raft_consensus.cc:552] T 4d69010513df4b68840b90be3161f402 P 5ee4739615124e308720b4aeddfe29cb [term 1 LEADER]: Received request to step down I0703 01:42:30.984098 28377 raft_consensus.cc:628] T 4d69010513df4b68840b90be3161f402 P 5ee4739615124e308720b4aeddfe29cb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5ee4739615124e308720b4aeddfe29cb, State: Running, Role: LEADER I0703 01:42:30.984184 29540 raft_consensus.cc:1081] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 35f0266742fc4707a8cc10c6437b84ed [term 1 LEARNER]: Refusing update from remote peer 31c252b54241470ba1115181c3672664: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I0703 01:42:30.984228 28377 consensus_queue.cc:227] T 4d69010513df4b68840b90be3161f402 P 5ee4739615124e308720b4aeddfe29cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "0ae780b86cc348d3a55bbaf5cf5a6217" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46219 } attrs { promote: false } } I0703 01:42:30.984344 28377 consensus_peers.cc:455] T 4d69010513df4b68840b90be3161f402 P 5ee4739615124e308720b4aeddfe29cb -> Peer 0ae780b86cc348d3a55bbaf5cf5a6217 (127.5.240.195:46219): Closing peer: 0ae780b86cc348d3a55bbaf5cf5a6217 I0703 01:42:30.984426 28377 raft_consensus.cc:2730] T 4d69010513df4b68840b90be3161f402 P 5ee4739615124e308720b4aeddfe29cb: Snoozing failure detection for 3.000s (explicit stepdown request) I0703 01:42:30.984784 30185 consensus_queue.cc:962] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 31c252b54241470ba1115181c3672664 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35f0266742fc4707a8cc10c6437b84ed" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 50368 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I0703 01:42:30.987423 30277 raft_consensus.cc:2684] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 31c252b54241470ba1115181c3672664 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 35f0266742fc4707a8cc10c6437b84ed (127.5.240.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } attrs { replace: true } } peers { permanent_uuid: "35f0266742fc4707a8cc10c6437b84ed" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 50368 } attrs { promote: false } } } I0703 01:42:30.988318 29540 raft_consensus.cc:2684] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 35f0266742fc4707a8cc10c6437b84ed [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 35f0266742fc4707a8cc10c6437b84ed (127.5.240.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } attrs { replace: true } } peers { permanent_uuid: "35f0266742fc4707a8cc10c6437b84ed" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 50368 } attrs { promote: false } } } I0703 01:42:30.992020 28251 catalog_manager.cc:3798] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 31c252b54241470ba1115181c3672664 reported cstate change: config changed from index 4 to 5, 35f0266742fc4707a8cc10c6437b84ed (127.5.240.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "31c252b54241470ba1115181c3672664" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35f0266742fc4707a8cc10c6437b84ed" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 50368 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I0703 01:42:31.000229 28377 tablet_service.cc:1140] Received LeaderStepDown RPC: tablet_id: "74b62768046f4b8690aab7d2ec0f803a" dest_uuid: "5ee4739615124e308720b4aeddfe29cb" from {username='jenkins'} at 127.5.240.193:54774 I0703 01:42:31.000335 28377 raft_consensus.cc:552] T 74b62768046f4b8690aab7d2ec0f803a P 5ee4739615124e308720b4aeddfe29cb [term 1 LEADER]: Received request to step down I0703 01:42:31.000385 28377 raft_consensus.cc:628] T 74b62768046f4b8690aab7d2ec0f803a P 5ee4739615124e308720b4aeddfe29cb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5ee4739615124e308720b4aeddfe29cb, State: Running, Role: LEADER I0703 01:42:31.000560 28377 consensus_queue.cc:227] T 74b62768046f4b8690aab7d2ec0f803a P 5ee4739615124e308720b4aeddfe29cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "575620f4889c4704b7a7c00da34b6a53" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39152 } attrs { promote: false } } I0703 01:42:31.000695 28377 consensus_peers.cc:455] T 74b62768046f4b8690aab7d2ec0f803a P 5ee4739615124e308720b4aeddfe29cb -> Peer 575620f4889c4704b7a7c00da34b6a53 (127.5.240.198:39152): Closing peer: 575620f4889c4704b7a7c00da34b6a53 I0703 01:42:31.000783 28377 raft_consensus.cc:2730] T 74b62768046f4b8690aab7d2ec0f803a P 5ee4739615124e308720b4aeddfe29cb: Snoozing failure detection for 3.000s (explicit stepdown request) I0703 01:42:31.012346 28500 tablet_service.cc:1140] Received LeaderStepDown RPC: tablet_id: "ef65b31bc12e40c2b3f1e4714cc4218e" dest_uuid: "31c252b54241470ba1115181c3672664" from {username='jenkins'} at 127.5.240.194:40868 I0703 01:42:31.012475 28500 raft_consensus.cc:552] T ef65b31bc12e40c2b3f1e4714cc4218e P 31c252b54241470ba1115181c3672664 [term 1 LEADER]: Received request to step down I0703 01:42:31.012518 28500 raft_consensus.cc:628] T ef65b31bc12e40c2b3f1e4714cc4218e P 31c252b54241470ba1115181c3672664 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 31c252b54241470ba1115181c3672664, State: Running, Role: LEADER I0703 01:42:31.012717 28500 consensus_queue.cc:227] T ef65b31bc12e40c2b3f1e4714cc4218e P 31c252b54241470ba1115181c3672664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } attrs { replace: true } } peers { permanent_uuid: "575620f4889c4704b7a7c00da34b6a53" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39152 } attrs { promote: false } } I0703 01:42:31.012857 28500 consensus_peers.cc:455] T ef65b31bc12e40c2b3f1e4714cc4218e P 31c252b54241470ba1115181c3672664 -> Peer 575620f4889c4704b7a7c00da34b6a53 (127.5.240.198:39152): Closing peer: 575620f4889c4704b7a7c00da34b6a53 I0703 01:42:31.012955 28500 raft_consensus.cc:2730] T ef65b31bc12e40c2b3f1e4714cc4218e P 31c252b54241470ba1115181c3672664: Snoozing failure detection for 3.000s (explicit stepdown request) I0703 01:42:31.024643 28377 tablet_service.cc:1140] Received LeaderStepDown RPC: tablet_id: "107baa5504214d5f8362cef92ed80291" dest_uuid: "5ee4739615124e308720b4aeddfe29cb" from {username='jenkins'} at 127.5.240.193:54778 I0703 01:42:31.024755 28377 raft_consensus.cc:552] T 107baa5504214d5f8362cef92ed80291 P 5ee4739615124e308720b4aeddfe29cb [term 1 LEADER]: Received request to step down I0703 01:42:31.024792 28377 raft_consensus.cc:628] T 107baa5504214d5f8362cef92ed80291 P 5ee4739615124e308720b4aeddfe29cb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5ee4739615124e308720b4aeddfe29cb, State: Running, Role: LEADER I0703 01:42:31.024957 28377 consensus_queue.cc:227] T 107baa5504214d5f8362cef92ed80291 P 5ee4739615124e308720b4aeddfe29cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "35f0266742fc4707a8cc10c6437b84ed" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 50368 } attrs { promote: false } } I0703 01:42:31.025070 28377 consensus_peers.cc:455] T 107baa5504214d5f8362cef92ed80291 P 5ee4739615124e308720b4aeddfe29cb -> Peer 35f0266742fc4707a8cc10c6437b84ed (127.5.240.196:50368): Closing peer: 35f0266742fc4707a8cc10c6437b84ed I0703 01:42:31.025148 28377 raft_consensus.cc:2730] T 107baa5504214d5f8362cef92ed80291 P 5ee4739615124e308720b4aeddfe29cb: Snoozing failure detection for 3.000s (explicit stepdown request) I0703 01:42:31.037768 28500 tablet_service.cc:1140] Received LeaderStepDown RPC: tablet_id: "41c52d5ecddf4e5897acf3fe5ee44f34" dest_uuid: "31c252b54241470ba1115181c3672664" from {username='jenkins'} at 127.5.240.194:40872 I0703 01:42:31.037864 28500 raft_consensus.cc:552] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 31c252b54241470ba1115181c3672664 [term 1 LEADER]: Received request to step down I0703 01:42:31.037907 28500 raft_consensus.cc:628] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 31c252b54241470ba1115181c3672664 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 31c252b54241470ba1115181c3672664, State: Running, Role: LEADER I0703 01:42:31.038046 28500 consensus_queue.cc:227] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 31c252b54241470ba1115181c3672664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } attrs { replace: true } } peers { permanent_uuid: "6505fcf874c440788e377dac9aa99209" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 33903 } attrs { promote: false } } I0703 01:42:31.038174 28500 consensus_peers.cc:455] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 31c252b54241470ba1115181c3672664 -> Peer 6505fcf874c440788e377dac9aa99209 (127.5.240.199:33903): Closing peer: 6505fcf874c440788e377dac9aa99209 I0703 01:42:31.038259 28500 raft_consensus.cc:2730] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 31c252b54241470ba1115181c3672664: Snoozing failure detection for 3.000s (explicit stepdown request) I0703 01:42:31.110090 30185 raft_consensus.cc:890] T 775bd6dae9ed48ebbbb9df088d833772 P 31c252b54241470ba1115181c3672664: attempting to promote NON_VOTER e98d29f36f2347f29170e987cb83ced4 to VOTER I0703 01:42:31.110302 30185 consensus_peers.cc:455] T 775bd6dae9ed48ebbbb9df088d833772 P 31c252b54241470ba1115181c3672664 -> Peer e98d29f36f2347f29170e987cb83ced4 (127.5.240.197:41466): Closing peer: e98d29f36f2347f29170e987cb83ced4 I0703 01:42:31.110487 30185 consensus_queue.cc:204] T 775bd6dae9ed48ebbbb9df088d833772 P 31c252b54241470ba1115181c3672664 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } attrs { replace: true } } peers { permanent_uuid: "e98d29f36f2347f29170e987cb83ced4" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 41466 } attrs { promote: false } } I0703 01:42:31.111979 29662 raft_consensus.cc:1081] T 775bd6dae9ed48ebbbb9df088d833772 P e98d29f36f2347f29170e987cb83ced4 [term 1 LEARNER]: Refusing update from remote peer 31c252b54241470ba1115181c3672664: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I0703 01:42:31.112490 30277 consensus_queue.cc:962] T 775bd6dae9ed48ebbbb9df088d833772 P 31c252b54241470ba1115181c3672664 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e98d29f36f2347f29170e987cb83ced4" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 41466 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I0703 01:42:31.114783 30185 raft_consensus.cc:2684] T 775bd6dae9ed48ebbbb9df088d833772 P 31c252b54241470ba1115181c3672664 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e98d29f36f2347f29170e987cb83ced4 (127.5.240.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } attrs { replace: true } } peers { permanent_uuid: "e98d29f36f2347f29170e987cb83ced4" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 41466 } attrs { promote: false } } } I0703 01:42:31.117166 29662 raft_consensus.cc:2684] T 775bd6dae9ed48ebbbb9df088d833772 P e98d29f36f2347f29170e987cb83ced4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e98d29f36f2347f29170e987cb83ced4 (127.5.240.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } attrs { replace: true } } peers { permanent_uuid: "e98d29f36f2347f29170e987cb83ced4" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 41466 } attrs { promote: false } } } I0703 01:42:31.117578 28253 catalog_manager.cc:3798] T 775bd6dae9ed48ebbbb9df088d833772 P 31c252b54241470ba1115181c3672664 reported cstate change: config changed from index 4 to 5, e98d29f36f2347f29170e987cb83ced4 (127.5.240.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "31c252b54241470ba1115181c3672664" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e98d29f36f2347f29170e987cb83ced4" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 41466 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I0703 01:42:31.256693 28500 tablet_service.cc:1140] Received LeaderStepDown RPC: tablet_id: "1cd5cbfe70ec4f8a957fa074ae08fcb1" dest_uuid: "31c252b54241470ba1115181c3672664" from {username='jenkins'} at 127.5.240.194:40875 I0703 01:42:31.256822 28500 raft_consensus.cc:552] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 31c252b54241470ba1115181c3672664 [term 1 LEADER]: Received request to step down I0703 01:42:31.256875 28500 raft_consensus.cc:628] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 31c252b54241470ba1115181c3672664 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 31c252b54241470ba1115181c3672664, State: Running, Role: LEADER I0703 01:42:31.257064 28500 consensus_queue.cc:227] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 31c252b54241470ba1115181c3672664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } attrs { replace: true } } peers { permanent_uuid: "35f0266742fc4707a8cc10c6437b84ed" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 50368 } attrs { promote: false } } I0703 01:42:31.257220 28500 consensus_peers.cc:455] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 31c252b54241470ba1115181c3672664 -> Peer 35f0266742fc4707a8cc10c6437b84ed (127.5.240.196:50368): Closing peer: 35f0266742fc4707a8cc10c6437b84ed I0703 01:42:31.257329 28500 raft_consensus.cc:2730] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 31c252b54241470ba1115181c3672664: Snoozing failure detection for 3.000s (explicit stepdown request) I0703 01:42:31.275471 28500 tablet_service.cc:1140] Received LeaderStepDown RPC: tablet_id: "775bd6dae9ed48ebbbb9df088d833772" dest_uuid: "31c252b54241470ba1115181c3672664" from {username='jenkins'} at 127.5.240.194:40878 I0703 01:42:31.275585 28500 raft_consensus.cc:552] T 775bd6dae9ed48ebbbb9df088d833772 P 31c252b54241470ba1115181c3672664 [term 1 LEADER]: Received request to step down I0703 01:42:31.275660 28500 raft_consensus.cc:628] T 775bd6dae9ed48ebbbb9df088d833772 P 31c252b54241470ba1115181c3672664 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 31c252b54241470ba1115181c3672664, State: Running, Role: LEADER I0703 01:42:31.275836 28500 consensus_queue.cc:227] T 775bd6dae9ed48ebbbb9df088d833772 P 31c252b54241470ba1115181c3672664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } attrs { replace: true } } peers { permanent_uuid: "e98d29f36f2347f29170e987cb83ced4" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 41466 } attrs { promote: false } } I0703 01:42:31.275987 28500 consensus_peers.cc:455] T 775bd6dae9ed48ebbbb9df088d833772 P 31c252b54241470ba1115181c3672664 -> Peer e98d29f36f2347f29170e987cb83ced4 (127.5.240.197:41466): Closing peer: e98d29f36f2347f29170e987cb83ced4 I0703 01:42:31.276096 28500 raft_consensus.cc:2730] T 775bd6dae9ed48ebbbb9df088d833772 P 31c252b54241470ba1115181c3672664: Snoozing failure detection for 3.000s (explicit stepdown request) I0703 01:42:32.149116 30633 raft_consensus.cc:457] T 45b3833ac0664a8d9889ab14b3d0dd69 P e98d29f36f2347f29170e987cb83ced4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ee4739615124e308720b4aeddfe29cb) I0703 01:42:32.149217 30633 raft_consensus.cc:2730] T 45b3833ac0664a8d9889ab14b3d0dd69 P e98d29f36f2347f29170e987cb83ced4: Snoozing failure detection for 2.057s (starting election) I0703 01:42:32.149322 30633 raft_consensus.cc:479] T 45b3833ac0664a8d9889ab14b3d0dd69 P e98d29f36f2347f29170e987cb83ced4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "e98d29f36f2347f29170e987cb83ced4" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 41466 } attrs { promote: false } } I0703 01:42:32.149947 30633 leader_election.cc:243] T 45b3833ac0664a8d9889ab14b3d0dd69 P e98d29f36f2347f29170e987cb83ced4 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 5ee4739615124e308720b4aeddfe29cb (127.5.240.193:54206) I0703 01:42:32.150779 28377 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "45b3833ac0664a8d9889ab14b3d0dd69" candidate_uuid: "e98d29f36f2347f29170e987cb83ced4" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "5ee4739615124e308720b4aeddfe29cb" is_pre_election: true I0703 01:42:32.150957 28377 raft_consensus.cc:2730] T 45b3833ac0664a8d9889ab14b3d0dd69 P 5ee4739615124e308720b4aeddfe29cb: Snoozing failure detection for 1.858s (vote granted) I0703 01:42:32.151023 28377 raft_consensus.cc:2243] T 45b3833ac0664a8d9889ab14b3d0dd69 P 5ee4739615124e308720b4aeddfe29cb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e98d29f36f2347f29170e987cb83ced4 in term 1. I0703 01:42:32.151537 29593 leader_election.cc:379] T 45b3833ac0664a8d9889ab14b3d0dd69 P e98d29f36f2347f29170e987cb83ced4 [CANDIDATE]: Term 2 pre-election: Vote granted by peer 5ee4739615124e308720b4aeddfe29cb (127.5.240.193:54206) I0703 01:42:32.151605 29593 leader_election.cc:270] T 45b3833ac0664a8d9889ab14b3d0dd69 P e98d29f36f2347f29170e987cb83ced4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. I0703 01:42:32.151872 30633 raft_consensus.cc:2730] T 45b3833ac0664a8d9889ab14b3d0dd69 P e98d29f36f2347f29170e987cb83ced4: Snoozing failure detection for 1.803s (election complete) I0703 01:42:32.151984 30633 raft_consensus.cc:2537] T 45b3833ac0664a8d9889ab14b3d0dd69 P e98d29f36f2347f29170e987cb83ced4 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0703 01:42:32.152050 30633 raft_consensus.cc:457] T 45b3833ac0664a8d9889ab14b3d0dd69 P e98d29f36f2347f29170e987cb83ced4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5ee4739615124e308720b4aeddfe29cb) I0703 01:42:32.152099 30633 raft_consensus.cc:2730] T 45b3833ac0664a8d9889ab14b3d0dd69 P e98d29f36f2347f29170e987cb83ced4: Snoozing failure detection for 1.726s (starting election) I0703 01:42:32.152159 30633 raft_consensus.cc:2782] T 45b3833ac0664a8d9889ab14b3d0dd69 P e98d29f36f2347f29170e987cb83ced4 [term 1 FOLLOWER]: Advancing to term 2 I0703 01:42:32.153626 30633 raft_consensus.cc:479] T 45b3833ac0664a8d9889ab14b3d0dd69 P e98d29f36f2347f29170e987cb83ced4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "e98d29f36f2347f29170e987cb83ced4" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 41466 } attrs { promote: false } } I0703 01:42:32.153873 30633 leader_election.cc:243] T 45b3833ac0664a8d9889ab14b3d0dd69 P e98d29f36f2347f29170e987cb83ced4 [CANDIDATE]: Term 2 election: Requesting vote from peer 5ee4739615124e308720b4aeddfe29cb (127.5.240.193:54206) I0703 01:42:32.154462 28377 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "45b3833ac0664a8d9889ab14b3d0dd69" candidate_uuid: "e98d29f36f2347f29170e987cb83ced4" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "5ee4739615124e308720b4aeddfe29cb" I0703 01:42:32.154554 28377 raft_consensus.cc:2782] T 45b3833ac0664a8d9889ab14b3d0dd69 P 5ee4739615124e308720b4aeddfe29cb [term 1 FOLLOWER]: Advancing to term 2 I0703 01:42:32.154664 28377 raft_consensus.cc:2730] T 45b3833ac0664a8d9889ab14b3d0dd69 P 5ee4739615124e308720b4aeddfe29cb: Snoozing failure detection for 1.637s (vote granted) I0703 01:42:32.155972 28377 raft_consensus.cc:2243] T 45b3833ac0664a8d9889ab14b3d0dd69 P 5ee4739615124e308720b4aeddfe29cb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e98d29f36f2347f29170e987cb83ced4 in term 2. I0703 01:42:32.156354 29593 leader_election.cc:379] T 45b3833ac0664a8d9889ab14b3d0dd69 P e98d29f36f2347f29170e987cb83ced4 [CANDIDATE]: Term 2 election: Vote granted by peer 5ee4739615124e308720b4aeddfe29cb (127.5.240.193:54206) I0703 01:42:32.156410 29593 leader_election.cc:270] T 45b3833ac0664a8d9889ab14b3d0dd69 P e98d29f36f2347f29170e987cb83ced4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0703 01:42:32.156644 30633 raft_consensus.cc:2730] T 45b3833ac0664a8d9889ab14b3d0dd69 P e98d29f36f2347f29170e987cb83ced4: Snoozing failure detection for 1.694s (election complete) I0703 01:42:32.156714 30633 raft_consensus.cc:2537] T 45b3833ac0664a8d9889ab14b3d0dd69 P e98d29f36f2347f29170e987cb83ced4 [term 2 FOLLOWER]: Leader election won for term 2 I0703 01:42:32.156967 30633 raft_consensus.cc:593] T 45b3833ac0664a8d9889ab14b3d0dd69 P e98d29f36f2347f29170e987cb83ced4 [term 2 LEADER]: Becoming Leader. State: Replica: e98d29f36f2347f29170e987cb83ced4, State: Running, Role: LEADER I0703 01:42:32.157299 30633 consensus_queue.cc:204] T 45b3833ac0664a8d9889ab14b3d0dd69 P e98d29f36f2347f29170e987cb83ced4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "e98d29f36f2347f29170e987cb83ced4" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 41466 } attrs { promote: false } } I0703 01:42:32.159574 28253 catalog_manager.cc:3798] T 45b3833ac0664a8d9889ab14b3d0dd69 P e98d29f36f2347f29170e987cb83ced4 reported cstate change: term changed from 1 to 2, leader changed from 5ee4739615124e308720b4aeddfe29cb (127.5.240.193) to e98d29f36f2347f29170e987cb83ced4 (127.5.240.197). New cstate: current_term: 2 leader_uuid: "e98d29f36f2347f29170e987cb83ced4" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e98d29f36f2347f29170e987cb83ced4" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 41466 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I0703 01:42:32.162963 28253 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 45b3833ac0664a8d9889ab14b3d0dd69 (attempt 1) I0703 01:42:32.168262 28240 catalog_manager.cc:3369] ChangeConfig:REMOVE_PEER RPC for tablet 45b3833ac0664a8d9889ab14b3d0dd69 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader e98d29f36f2347f29170e987cb83ced4 (127.5.240.197:41466) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term I0703 01:42:32.168350 28240 catalog_manager.cc:2903] Scheduling retry of ChangeConfig:REMOVE_PEER RPC for tablet 45b3833ac0664a8d9889ab14b3d0dd69 with cas_config_opid_index 5 with a delay of 32 ms (attempt = 1) I0703 01:42:32.200117 30642 raft_consensus.cc:457] T b4acd291b108434da9b8791c61fd1dc2 P 6505fcf874c440788e377dac9aa99209 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ee4739615124e308720b4aeddfe29cb) I0703 01:42:32.200207 30642 raft_consensus.cc:2730] T b4acd291b108434da9b8791c61fd1dc2 P 6505fcf874c440788e377dac9aa99209: Snoozing failure detection for 1.757s (starting election) I0703 01:42:32.200309 30642 raft_consensus.cc:479] T b4acd291b108434da9b8791c61fd1dc2 P 6505fcf874c440788e377dac9aa99209 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "6505fcf874c440788e377dac9aa99209" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 33903 } attrs { promote: false } } I0703 01:42:32.200768 28240 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 45b3833ac0664a8d9889ab14b3d0dd69 (attempt 2) I0703 01:42:32.200954 30642 leader_election.cc:243] T b4acd291b108434da9b8791c61fd1dc2 P 6505fcf874c440788e377dac9aa99209 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 5ee4739615124e308720b4aeddfe29cb (127.5.240.193:54206) I0703 01:42:32.201473 28240 catalog_manager.cc:3369] ChangeConfig:REMOVE_PEER RPC for tablet 45b3833ac0664a8d9889ab14b3d0dd69 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader e98d29f36f2347f29170e987cb83ced4 (127.5.240.197:41466) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term I0703 01:42:32.201537 28240 catalog_manager.cc:2903] Scheduling retry of ChangeConfig:REMOVE_PEER RPC for tablet 45b3833ac0664a8d9889ab14b3d0dd69 with cas_config_opid_index 5 with a delay of 79 ms (attempt = 2) I0703 01:42:32.201833 28377 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "b4acd291b108434da9b8791c61fd1dc2" candidate_uuid: "6505fcf874c440788e377dac9aa99209" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "5ee4739615124e308720b4aeddfe29cb" is_pre_election: true I0703 01:42:32.201948 28377 raft_consensus.cc:2730] T b4acd291b108434da9b8791c61fd1dc2 P 5ee4739615124e308720b4aeddfe29cb: Snoozing failure detection for 1.647s (vote granted) I0703 01:42:32.202018 28377 raft_consensus.cc:2243] T b4acd291b108434da9b8791c61fd1dc2 P 5ee4739615124e308720b4aeddfe29cb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6505fcf874c440788e377dac9aa99209 in term 1. I0703 01:42:32.202471 29837 leader_election.cc:379] T b4acd291b108434da9b8791c61fd1dc2 P 6505fcf874c440788e377dac9aa99209 [CANDIDATE]: Term 2 pre-election: Vote granted by peer 5ee4739615124e308720b4aeddfe29cb (127.5.240.193:54206) I0703 01:42:32.202553 29837 leader_election.cc:270] T b4acd291b108434da9b8791c61fd1dc2 P 6505fcf874c440788e377dac9aa99209 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. I0703 01:42:32.202826 30642 raft_consensus.cc:2730] T b4acd291b108434da9b8791c61fd1dc2 P 6505fcf874c440788e377dac9aa99209: Snoozing failure detection for 1.910s (election complete) I0703 01:42:32.202939 30642 raft_consensus.cc:2537] T b4acd291b108434da9b8791c61fd1dc2 P 6505fcf874c440788e377dac9aa99209 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0703 01:42:32.203003 30642 raft_consensus.cc:457] T b4acd291b108434da9b8791c61fd1dc2 P 6505fcf874c440788e377dac9aa99209 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5ee4739615124e308720b4aeddfe29cb) I0703 01:42:32.203052 30642 raft_consensus.cc:2730] T b4acd291b108434da9b8791c61fd1dc2 P 6505fcf874c440788e377dac9aa99209: Snoozing failure detection for 1.917s (starting election) I0703 01:42:32.203110 30642 raft_consensus.cc:2782] T b4acd291b108434da9b8791c61fd1dc2 P 6505fcf874c440788e377dac9aa99209 [term 1 FOLLOWER]: Advancing to term 2 I0703 01:42:32.204478 30642 raft_consensus.cc:479] T b4acd291b108434da9b8791c61fd1dc2 P 6505fcf874c440788e377dac9aa99209 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "6505fcf874c440788e377dac9aa99209" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 33903 } attrs { promote: false } } I0703 01:42:32.204751 30642 leader_election.cc:243] T b4acd291b108434da9b8791c61fd1dc2 P 6505fcf874c440788e377dac9aa99209 [CANDIDATE]: Term 2 election: Requesting vote from peer 5ee4739615124e308720b4aeddfe29cb (127.5.240.193:54206) I0703 01:42:32.205346 28377 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "b4acd291b108434da9b8791c61fd1dc2" candidate_uuid: "6505fcf874c440788e377dac9aa99209" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "5ee4739615124e308720b4aeddfe29cb" I0703 01:42:32.205446 28377 raft_consensus.cc:2782] T b4acd291b108434da9b8791c61fd1dc2 P 5ee4739615124e308720b4aeddfe29cb [term 1 FOLLOWER]: Advancing to term 2 I0703 01:42:32.205508 28377 raft_consensus.cc:2730] T b4acd291b108434da9b8791c61fd1dc2 P 5ee4739615124e308720b4aeddfe29cb: Snoozing failure detection for 1.934s (vote granted) I0703 01:42:32.206739 28377 raft_consensus.cc:2243] T b4acd291b108434da9b8791c61fd1dc2 P 5ee4739615124e308720b4aeddfe29cb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6505fcf874c440788e377dac9aa99209 in term 2. I0703 01:42:32.207131 29837 leader_election.cc:379] T b4acd291b108434da9b8791c61fd1dc2 P 6505fcf874c440788e377dac9aa99209 [CANDIDATE]: Term 2 election: Vote granted by peer 5ee4739615124e308720b4aeddfe29cb (127.5.240.193:54206) I0703 01:42:32.207186 29837 leader_election.cc:270] T b4acd291b108434da9b8791c61fd1dc2 P 6505fcf874c440788e377dac9aa99209 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0703 01:42:32.207355 30642 raft_consensus.cc:2730] T b4acd291b108434da9b8791c61fd1dc2 P 6505fcf874c440788e377dac9aa99209: Snoozing failure detection for 2.194s (election complete) I0703 01:42:32.207422 30642 raft_consensus.cc:2537] T b4acd291b108434da9b8791c61fd1dc2 P 6505fcf874c440788e377dac9aa99209 [term 2 FOLLOWER]: Leader election won for term 2 I0703 01:42:32.207710 30642 raft_consensus.cc:593] T b4acd291b108434da9b8791c61fd1dc2 P 6505fcf874c440788e377dac9aa99209 [term 2 LEADER]: Becoming Leader. State: Replica: 6505fcf874c440788e377dac9aa99209, State: Running, Role: LEADER I0703 01:42:32.208006 30642 consensus_queue.cc:204] T b4acd291b108434da9b8791c61fd1dc2 P 6505fcf874c440788e377dac9aa99209 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "6505fcf874c440788e377dac9aa99209" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 33903 } attrs { promote: false } } I0703 01:42:32.210290 28253 catalog_manager.cc:3798] T b4acd291b108434da9b8791c61fd1dc2 P 6505fcf874c440788e377dac9aa99209 reported cstate change: term changed from 1 to 2, leader changed from 5ee4739615124e308720b4aeddfe29cb (127.5.240.193) to 6505fcf874c440788e377dac9aa99209 (127.5.240.199). New cstate: current_term: 2 leader_uuid: "6505fcf874c440788e377dac9aa99209" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6505fcf874c440788e377dac9aa99209" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 33903 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I0703 01:42:32.212287 28253 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet b4acd291b108434da9b8791c61fd1dc2 (attempt 1) I0703 01:42:32.216867 28239 catalog_manager.cc:3369] ChangeConfig:REMOVE_PEER RPC for tablet b4acd291b108434da9b8791c61fd1dc2 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 6505fcf874c440788e377dac9aa99209 (127.5.240.199:33903) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term I0703 01:42:32.216928 28239 catalog_manager.cc:2903] Scheduling retry of ChangeConfig:REMOVE_PEER RPC for tablet b4acd291b108434da9b8791c61fd1dc2 with cas_config_opid_index 5 with a delay of 64 ms (attempt = 1) I0703 01:42:32.281440 28240 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 45b3833ac0664a8d9889ab14b3d0dd69 (attempt 3) I0703 01:42:32.281636 28239 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet b4acd291b108434da9b8791c61fd1dc2 (attempt 2) I0703 01:42:32.282455 28239 catalog_manager.cc:3369] ChangeConfig:REMOVE_PEER RPC for tablet b4acd291b108434da9b8791c61fd1dc2 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 6505fcf874c440788e377dac9aa99209 (127.5.240.199:33903) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term I0703 01:42:32.282510 28240 catalog_manager.cc:3369] ChangeConfig:REMOVE_PEER RPC for tablet 45b3833ac0664a8d9889ab14b3d0dd69 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader e98d29f36f2347f29170e987cb83ced4 (127.5.240.197:41466) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term I0703 01:42:32.282606 28240 catalog_manager.cc:2903] Scheduling retry of ChangeConfig:REMOVE_PEER RPC for tablet 45b3833ac0664a8d9889ab14b3d0dd69 with cas_config_opid_index 5 with a delay of 86 ms (attempt = 3) I0703 01:42:32.282618 28239 catalog_manager.cc:2903] Scheduling retry of ChangeConfig:REMOVE_PEER RPC for tablet b4acd291b108434da9b8791c61fd1dc2 with cas_config_opid_index 5 with a delay of 79 ms (attempt = 2) I0703 01:42:32.361588 30691 raft_consensus.cc:457] T 4d69010513df4b68840b90be3161f402 P 0ae780b86cc348d3a55bbaf5cf5a6217 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ee4739615124e308720b4aeddfe29cb) I0703 01:42:32.361690 30691 raft_consensus.cc:2730] T 4d69010513df4b68840b90be3161f402 P 0ae780b86cc348d3a55bbaf5cf5a6217: Snoozing failure detection for 1.885s (starting election) I0703 01:42:32.361790 30691 raft_consensus.cc:479] T 4d69010513df4b68840b90be3161f402 P 0ae780b86cc348d3a55bbaf5cf5a6217 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "0ae780b86cc348d3a55bbaf5cf5a6217" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46219 } attrs { promote: false } } I0703 01:42:32.362402 28239 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet b4acd291b108434da9b8791c61fd1dc2 (attempt 3) I0703 01:42:32.362458 30691 leader_election.cc:243] T 4d69010513df4b68840b90be3161f402 P 0ae780b86cc348d3a55bbaf5cf5a6217 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 5ee4739615124e308720b4aeddfe29cb (127.5.240.193:54206) I0703 01:42:32.363261 28377 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "4d69010513df4b68840b90be3161f402" candidate_uuid: "0ae780b86cc348d3a55bbaf5cf5a6217" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "5ee4739615124e308720b4aeddfe29cb" is_pre_election: true I0703 01:42:32.363317 28239 catalog_manager.cc:3369] ChangeConfig:REMOVE_PEER RPC for tablet b4acd291b108434da9b8791c61fd1dc2 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 6505fcf874c440788e377dac9aa99209 (127.5.240.199:33903) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term I0703 01:42:32.363370 28239 catalog_manager.cc:2903] Scheduling retry of ChangeConfig:REMOVE_PEER RPC for tablet b4acd291b108434da9b8791c61fd1dc2 with cas_config_opid_index 5 with a delay of 80 ms (attempt = 3) I0703 01:42:32.363402 28377 raft_consensus.cc:2730] T 4d69010513df4b68840b90be3161f402 P 5ee4739615124e308720b4aeddfe29cb: Snoozing failure detection for 1.679s (vote granted) I0703 01:42:32.363469 28377 raft_consensus.cc:2243] T 4d69010513df4b68840b90be3161f402 P 5ee4739615124e308720b4aeddfe29cb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ae780b86cc348d3a55bbaf5cf5a6217 in term 1. I0703 01:42:32.363867 29348 leader_election.cc:379] T 4d69010513df4b68840b90be3161f402 P 0ae780b86cc348d3a55bbaf5cf5a6217 [CANDIDATE]: Term 2 pre-election: Vote granted by peer 5ee4739615124e308720b4aeddfe29cb (127.5.240.193:54206) I0703 01:42:32.363924 29348 leader_election.cc:270] T 4d69010513df4b68840b90be3161f402 P 0ae780b86cc348d3a55bbaf5cf5a6217 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. I0703 01:42:32.364171 30691 raft_consensus.cc:2730] T 4d69010513df4b68840b90be3161f402 P 0ae780b86cc348d3a55bbaf5cf5a6217: Snoozing failure detection for 1.736s (election complete) I0703 01:42:32.364269 30691 raft_consensus.cc:2537] T 4d69010513df4b68840b90be3161f402 P 0ae780b86cc348d3a55bbaf5cf5a6217 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0703 01:42:32.364326 30691 raft_consensus.cc:457] T 4d69010513df4b68840b90be3161f402 P 0ae780b86cc348d3a55bbaf5cf5a6217 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5ee4739615124e308720b4aeddfe29cb) I0703 01:42:32.364392 30691 raft_consensus.cc:2730] T 4d69010513df4b68840b90be3161f402 P 0ae780b86cc348d3a55bbaf5cf5a6217: Snoozing failure detection for 1.901s (starting election) I0703 01:42:32.364460 30691 raft_consensus.cc:2782] T 4d69010513df4b68840b90be3161f402 P 0ae780b86cc348d3a55bbaf5cf5a6217 [term 1 FOLLOWER]: Advancing to term 2 I0703 01:42:32.365989 30691 raft_consensus.cc:479] T 4d69010513df4b68840b90be3161f402 P 0ae780b86cc348d3a55bbaf5cf5a6217 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "0ae780b86cc348d3a55bbaf5cf5a6217" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46219 } attrs { promote: false } } I0703 01:42:32.366230 30691 leader_election.cc:243] T 4d69010513df4b68840b90be3161f402 P 0ae780b86cc348d3a55bbaf5cf5a6217 [CANDIDATE]: Term 2 election: Requesting vote from peer 5ee4739615124e308720b4aeddfe29cb (127.5.240.193:54206) I0703 01:42:32.366814 28377 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "4d69010513df4b68840b90be3161f402" candidate_uuid: "0ae780b86cc348d3a55bbaf5cf5a6217" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "5ee4739615124e308720b4aeddfe29cb" I0703 01:42:32.366911 28377 raft_consensus.cc:2782] T 4d69010513df4b68840b90be3161f402 P 5ee4739615124e308720b4aeddfe29cb [term 1 FOLLOWER]: Advancing to term 2 I0703 01:42:32.366971 28377 raft_consensus.cc:2730] T 4d69010513df4b68840b90be3161f402 P 5ee4739615124e308720b4aeddfe29cb: Snoozing failure detection for 1.873s (vote granted) I0703 01:42:32.368257 28377 raft_consensus.cc:2243] T 4d69010513df4b68840b90be3161f402 P 5ee4739615124e308720b4aeddfe29cb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ae780b86cc348d3a55bbaf5cf5a6217 in term 2. I0703 01:42:32.368604 29348 leader_election.cc:379] T 4d69010513df4b68840b90be3161f402 P 0ae780b86cc348d3a55bbaf5cf5a6217 [CANDIDATE]: Term 2 election: Vote granted by peer 5ee4739615124e308720b4aeddfe29cb (127.5.240.193:54206) I0703 01:42:32.368660 29348 leader_election.cc:270] T 4d69010513df4b68840b90be3161f402 P 0ae780b86cc348d3a55bbaf5cf5a6217 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0703 01:42:32.368826 30691 raft_consensus.cc:2730] T 4d69010513df4b68840b90be3161f402 P 0ae780b86cc348d3a55bbaf5cf5a6217: Snoozing failure detection for 2.247s (election complete) I0703 01:42:32.368890 30691 raft_consensus.cc:2537] T 4d69010513df4b68840b90be3161f402 P 0ae780b86cc348d3a55bbaf5cf5a6217 [term 2 FOLLOWER]: Leader election won for term 2 I0703 01:42:32.369202 30691 raft_consensus.cc:593] T 4d69010513df4b68840b90be3161f402 P 0ae780b86cc348d3a55bbaf5cf5a6217 [term 2 LEADER]: Becoming Leader. State: Replica: 0ae780b86cc348d3a55bbaf5cf5a6217, State: Running, Role: LEADER I0703 01:42:32.369488 30691 consensus_queue.cc:204] T 4d69010513df4b68840b90be3161f402 P 0ae780b86cc348d3a55bbaf5cf5a6217 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "0ae780b86cc348d3a55bbaf5cf5a6217" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46219 } attrs { promote: false } } I0703 01:42:32.369904 28240 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 45b3833ac0664a8d9889ab14b3d0dd69 (attempt 4) I0703 01:42:32.370733 28240 catalog_manager.cc:3369] ChangeConfig:REMOVE_PEER RPC for tablet 45b3833ac0664a8d9889ab14b3d0dd69 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader e98d29f36f2347f29170e987cb83ced4 (127.5.240.197:41466) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term I0703 01:42:32.370807 28240 catalog_manager.cc:2903] Scheduling retry of ChangeConfig:REMOVE_PEER RPC for tablet 45b3833ac0664a8d9889ab14b3d0dd69 with cas_config_opid_index 5 with a delay of 164 ms (attempt = 4) I0703 01:42:32.371889 28253 catalog_manager.cc:3798] T 4d69010513df4b68840b90be3161f402 P 0ae780b86cc348d3a55bbaf5cf5a6217 reported cstate change: term changed from 1 to 2, leader changed from 5ee4739615124e308720b4aeddfe29cb (127.5.240.193) to 0ae780b86cc348d3a55bbaf5cf5a6217 (127.5.240.195). New cstate: current_term: 2 leader_uuid: "0ae780b86cc348d3a55bbaf5cf5a6217" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ae780b86cc348d3a55bbaf5cf5a6217" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46219 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I0703 01:42:32.374164 28253 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 4d69010513df4b68840b90be3161f402 (attempt 1) I0703 01:42:32.380296 28239 catalog_manager.cc:3369] ChangeConfig:REMOVE_PEER RPC for tablet 4d69010513df4b68840b90be3161f402 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 0ae780b86cc348d3a55bbaf5cf5a6217 (127.5.240.195:46219) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term I0703 01:42:32.380359 28239 catalog_manager.cc:2903] Scheduling retry of ChangeConfig:REMOVE_PEER RPC for tablet 4d69010513df4b68840b90be3161f402 with cas_config_opid_index 5 with a delay of 28 ms (attempt = 1) I0703 01:42:32.408684 28239 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 4d69010513df4b68840b90be3161f402 (attempt 2) I0703 01:42:32.409286 28239 catalog_manager.cc:3369] ChangeConfig:REMOVE_PEER RPC for tablet 4d69010513df4b68840b90be3161f402 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 0ae780b86cc348d3a55bbaf5cf5a6217 (127.5.240.195:46219) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term I0703 01:42:32.409355 28239 catalog_manager.cc:2903] Scheduling retry of ChangeConfig:REMOVE_PEER RPC for tablet 4d69010513df4b68840b90be3161f402 with cas_config_opid_index 5 with a delay of 41 ms (attempt = 2) I0703 01:42:32.419775 30695 raft_consensus.cc:457] T 74b62768046f4b8690aab7d2ec0f803a P 575620f4889c4704b7a7c00da34b6a53 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ee4739615124e308720b4aeddfe29cb) I0703 01:42:32.419870 30695 raft_consensus.cc:2730] T 74b62768046f4b8690aab7d2ec0f803a P 575620f4889c4704b7a7c00da34b6a53: Snoozing failure detection for 1.751s (starting election) I0703 01:42:32.419977 30695 raft_consensus.cc:479] T 74b62768046f4b8690aab7d2ec0f803a P 575620f4889c4704b7a7c00da34b6a53 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "575620f4889c4704b7a7c00da34b6a53" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39152 } attrs { promote: false } } I0703 01:42:32.420555 30695 leader_election.cc:243] T 74b62768046f4b8690aab7d2ec0f803a P 575620f4889c4704b7a7c00da34b6a53 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 5ee4739615124e308720b4aeddfe29cb (127.5.240.193:54206) I0703 01:42:32.421381 28377 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "74b62768046f4b8690aab7d2ec0f803a" candidate_uuid: "575620f4889c4704b7a7c00da34b6a53" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "5ee4739615124e308720b4aeddfe29cb" is_pre_election: true I0703 01:42:32.421501 28377 raft_consensus.cc:2730] T 74b62768046f4b8690aab7d2ec0f803a P 5ee4739615124e308720b4aeddfe29cb: Snoozing failure detection for 2.193s (vote granted) I0703 01:42:32.421564 28377 raft_consensus.cc:2243] T 74b62768046f4b8690aab7d2ec0f803a P 5ee4739615124e308720b4aeddfe29cb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 575620f4889c4704b7a7c00da34b6a53 in term 1. I0703 01:42:32.422060 29715 leader_election.cc:379] T 74b62768046f4b8690aab7d2ec0f803a P 575620f4889c4704b7a7c00da34b6a53 [CANDIDATE]: Term 2 pre-election: Vote granted by peer 5ee4739615124e308720b4aeddfe29cb (127.5.240.193:54206) I0703 01:42:32.422125 29715 leader_election.cc:270] T 74b62768046f4b8690aab7d2ec0f803a P 575620f4889c4704b7a7c00da34b6a53 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. I0703 01:42:32.422441 30695 raft_consensus.cc:2730] T 74b62768046f4b8690aab7d2ec0f803a P 575620f4889c4704b7a7c00da34b6a53: Snoozing failure detection for 1.647s (election complete) I0703 01:42:32.422560 30695 raft_consensus.cc:2537] T 74b62768046f4b8690aab7d2ec0f803a P 575620f4889c4704b7a7c00da34b6a53 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0703 01:42:32.422659 30695 raft_consensus.cc:457] T 74b62768046f4b8690aab7d2ec0f803a P 575620f4889c4704b7a7c00da34b6a53 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5ee4739615124e308720b4aeddfe29cb) I0703 01:42:32.422709 30695 raft_consensus.cc:2730] T 74b62768046f4b8690aab7d2ec0f803a P 575620f4889c4704b7a7c00da34b6a53: Snoozing failure detection for 1.744s (starting election) I0703 01:42:32.422789 30695 raft_consensus.cc:2782] T 74b62768046f4b8690aab7d2ec0f803a P 575620f4889c4704b7a7c00da34b6a53 [term 1 FOLLOWER]: Advancing to term 2 I0703 01:42:32.424198 30695 raft_consensus.cc:479] T 74b62768046f4b8690aab7d2ec0f803a P 575620f4889c4704b7a7c00da34b6a53 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "575620f4889c4704b7a7c00da34b6a53" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39152 } attrs { promote: false } } I0703 01:42:32.424458 30695 leader_election.cc:243] T 74b62768046f4b8690aab7d2ec0f803a P 575620f4889c4704b7a7c00da34b6a53 [CANDIDATE]: Term 2 election: Requesting vote from peer 5ee4739615124e308720b4aeddfe29cb (127.5.240.193:54206) I0703 01:42:32.425119 28377 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "74b62768046f4b8690aab7d2ec0f803a" candidate_uuid: "575620f4889c4704b7a7c00da34b6a53" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "5ee4739615124e308720b4aeddfe29cb" I0703 01:42:32.425215 28377 raft_consensus.cc:2782] T 74b62768046f4b8690aab7d2ec0f803a P 5ee4739615124e308720b4aeddfe29cb [term 1 FOLLOWER]: Advancing to term 2 I0703 01:42:32.425268 28377 raft_consensus.cc:2730] T 74b62768046f4b8690aab7d2ec0f803a P 5ee4739615124e308720b4aeddfe29cb: Snoozing failure detection for 1.975s (vote granted) I0703 01:42:32.426544 28377 raft_consensus.cc:2243] T 74b62768046f4b8690aab7d2ec0f803a P 5ee4739615124e308720b4aeddfe29cb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 575620f4889c4704b7a7c00da34b6a53 in term 2. I0703 01:42:32.426940 29715 leader_election.cc:379] T 74b62768046f4b8690aab7d2ec0f803a P 575620f4889c4704b7a7c00da34b6a53 [CANDIDATE]: Term 2 election: Vote granted by peer 5ee4739615124e308720b4aeddfe29cb (127.5.240.193:54206) I0703 01:42:32.426996 29715 leader_election.cc:270] T 74b62768046f4b8690aab7d2ec0f803a P 575620f4889c4704b7a7c00da34b6a53 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0703 01:42:32.427183 30695 raft_consensus.cc:2730] T 74b62768046f4b8690aab7d2ec0f803a P 575620f4889c4704b7a7c00da34b6a53: Snoozing failure detection for 2.240s (election complete) I0703 01:42:32.427254 30695 raft_consensus.cc:2537] T 74b62768046f4b8690aab7d2ec0f803a P 575620f4889c4704b7a7c00da34b6a53 [term 2 FOLLOWER]: Leader election won for term 2 I0703 01:42:32.427537 30695 raft_consensus.cc:593] T 74b62768046f4b8690aab7d2ec0f803a P 575620f4889c4704b7a7c00da34b6a53 [term 2 LEADER]: Becoming Leader. State: Replica: 575620f4889c4704b7a7c00da34b6a53, State: Running, Role: LEADER I0703 01:42:32.427867 30695 consensus_queue.cc:204] T 74b62768046f4b8690aab7d2ec0f803a P 575620f4889c4704b7a7c00da34b6a53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "575620f4889c4704b7a7c00da34b6a53" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39152 } attrs { promote: false } } I0703 01:42:32.430212 28253 catalog_manager.cc:3798] T 74b62768046f4b8690aab7d2ec0f803a P 575620f4889c4704b7a7c00da34b6a53 reported cstate change: term changed from 1 to 2, leader changed from 5ee4739615124e308720b4aeddfe29cb (127.5.240.193) to 575620f4889c4704b7a7c00da34b6a53 (127.5.240.198). New cstate: current_term: 2 leader_uuid: "575620f4889c4704b7a7c00da34b6a53" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "575620f4889c4704b7a7c00da34b6a53" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39152 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I0703 01:42:32.432446 28253 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 74b62768046f4b8690aab7d2ec0f803a (attempt 1) I0703 01:42:32.437330 28240 catalog_manager.cc:3369] ChangeConfig:REMOVE_PEER RPC for tablet 74b62768046f4b8690aab7d2ec0f803a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 575620f4889c4704b7a7c00da34b6a53 (127.5.240.198:39152) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term I0703 01:42:32.437393 28240 catalog_manager.cc:2903] Scheduling retry of ChangeConfig:REMOVE_PEER RPC for tablet 74b62768046f4b8690aab7d2ec0f803a with cas_config_opid_index 5 with a delay of 43 ms (attempt = 1) I0703 01:42:32.444242 28239 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet b4acd291b108434da9b8791c61fd1dc2 (attempt 4) I0703 01:42:32.445029 28239 catalog_manager.cc:3369] ChangeConfig:REMOVE_PEER RPC for tablet b4acd291b108434da9b8791c61fd1dc2 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 6505fcf874c440788e377dac9aa99209 (127.5.240.199:33903) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term I0703 01:42:32.445091 28239 catalog_manager.cc:2903] Scheduling retry of ChangeConfig:REMOVE_PEER RPC for tablet b4acd291b108434da9b8791c61fd1dc2 with cas_config_opid_index 5 with a delay of 128 ms (attempt = 4) I0703 01:42:32.445447 30642 raft_consensus.cc:457] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 6505fcf874c440788e377dac9aa99209 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 31c252b54241470ba1115181c3672664) I0703 01:42:32.445523 30642 raft_consensus.cc:2730] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 6505fcf874c440788e377dac9aa99209: Snoozing failure detection for 1.749s (starting election) I0703 01:42:32.445617 30642 raft_consensus.cc:479] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 6505fcf874c440788e377dac9aa99209 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } attrs { replace: true } } peers { permanent_uuid: "6505fcf874c440788e377dac9aa99209" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 33903 } attrs { promote: false } } I0703 01:42:32.445910 30642 leader_election.cc:243] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 6505fcf874c440788e377dac9aa99209 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 31c252b54241470ba1115181c3672664 (127.5.240.194:38625) I0703 01:42:32.446692 28500 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "41c52d5ecddf4e5897acf3fe5ee44f34" candidate_uuid: "6505fcf874c440788e377dac9aa99209" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "31c252b54241470ba1115181c3672664" is_pre_election: true I0703 01:42:32.446862 28500 raft_consensus.cc:2730] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 31c252b54241470ba1115181c3672664: Snoozing failure detection for 2.140s (vote granted) I0703 01:42:32.446943 28500 raft_consensus.cc:2243] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 31c252b54241470ba1115181c3672664 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6505fcf874c440788e377dac9aa99209 in term 1. I0703 01:42:32.447382 29837 leader_election.cc:379] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 6505fcf874c440788e377dac9aa99209 [CANDIDATE]: Term 2 pre-election: Vote granted by peer 31c252b54241470ba1115181c3672664 (127.5.240.194:38625) I0703 01:42:32.447446 29837 leader_election.cc:270] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 6505fcf874c440788e377dac9aa99209 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. I0703 01:42:32.447644 30642 raft_consensus.cc:2730] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 6505fcf874c440788e377dac9aa99209: Snoozing failure detection for 1.506s (election complete) I0703 01:42:32.447719 30642 raft_consensus.cc:2537] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 6505fcf874c440788e377dac9aa99209 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0703 01:42:32.447765 30642 raft_consensus.cc:457] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 6505fcf874c440788e377dac9aa99209 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 31c252b54241470ba1115181c3672664) I0703 01:42:32.447821 30642 raft_consensus.cc:2730] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 6505fcf874c440788e377dac9aa99209: Snoozing failure detection for 2.092s (starting election) I0703 01:42:32.447873 30642 raft_consensus.cc:2782] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 6505fcf874c440788e377dac9aa99209 [term 1 FOLLOWER]: Advancing to term 2 I0703 01:42:32.449220 30642 raft_consensus.cc:479] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 6505fcf874c440788e377dac9aa99209 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } attrs { replace: true } } peers { permanent_uuid: "6505fcf874c440788e377dac9aa99209" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 33903 } attrs { promote: false } } I0703 01:42:32.449470 30642 leader_election.cc:243] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 6505fcf874c440788e377dac9aa99209 [CANDIDATE]: Term 2 election: Requesting vote from peer 31c252b54241470ba1115181c3672664 (127.5.240.194:38625) I0703 01:42:32.450088 28500 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "41c52d5ecddf4e5897acf3fe5ee44f34" candidate_uuid: "6505fcf874c440788e377dac9aa99209" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "31c252b54241470ba1115181c3672664" I0703 01:42:32.450198 28500 raft_consensus.cc:2782] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 31c252b54241470ba1115181c3672664 [term 1 FOLLOWER]: Advancing to term 2 I0703 01:42:32.450299 28500 raft_consensus.cc:2730] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 31c252b54241470ba1115181c3672664: Snoozing failure detection for 2.055s (vote granted) I0703 01:42:32.451524 28239 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 4d69010513df4b68840b90be3161f402 (attempt 3) I0703 01:42:32.451881 28500 raft_consensus.cc:2243] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 31c252b54241470ba1115181c3672664 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6505fcf874c440788e377dac9aa99209 in term 2. I0703 01:42:32.452306 29837 leader_election.cc:379] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 6505fcf874c440788e377dac9aa99209 [CANDIDATE]: Term 2 election: Vote granted by peer 31c252b54241470ba1115181c3672664 (127.5.240.194:38625) I0703 01:42:32.452364 29837 leader_election.cc:270] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 6505fcf874c440788e377dac9aa99209 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0703 01:42:32.452407 28239 catalog_manager.cc:3369] ChangeConfig:REMOVE_PEER RPC for tablet 4d69010513df4b68840b90be3161f402 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 0ae780b86cc348d3a55bbaf5cf5a6217 (127.5.240.195:46219) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term I0703 01:42:32.452488 28239 catalog_manager.cc:2903] Scheduling retry of ChangeConfig:REMOVE_PEER RPC for tablet 4d69010513df4b68840b90be3161f402 with cas_config_opid_index 5 with a delay of 67 ms (attempt = 3) I0703 01:42:32.452635 30642 raft_consensus.cc:2730] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 6505fcf874c440788e377dac9aa99209: Snoozing failure detection for 1.749s (election complete) I0703 01:42:32.452697 30642 raft_consensus.cc:2537] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 6505fcf874c440788e377dac9aa99209 [term 2 FOLLOWER]: Leader election won for term 2 I0703 01:42:32.452810 30642 raft_consensus.cc:593] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 6505fcf874c440788e377dac9aa99209 [term 2 LEADER]: Becoming Leader. State: Replica: 6505fcf874c440788e377dac9aa99209, State: Running, Role: LEADER I0703 01:42:32.452989 30642 consensus_queue.cc:204] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 6505fcf874c440788e377dac9aa99209 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } attrs { replace: true } } peers { permanent_uuid: "6505fcf874c440788e377dac9aa99209" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 33903 } attrs { promote: false } } I0703 01:42:32.454749 28253 catalog_manager.cc:3798] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 6505fcf874c440788e377dac9aa99209 reported cstate change: term changed from 1 to 2, leader changed from 31c252b54241470ba1115181c3672664 (127.5.240.194) to 6505fcf874c440788e377dac9aa99209 (127.5.240.199). New cstate: current_term: 2 leader_uuid: "6505fcf874c440788e377dac9aa99209" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6505fcf874c440788e377dac9aa99209" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 33903 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I0703 01:42:32.456446 28253 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 41c52d5ecddf4e5897acf3fe5ee44f34 (attempt 1) I0703 01:42:32.457219 28239 catalog_manager.cc:3369] ChangeConfig:REMOVE_PEER RPC for tablet 41c52d5ecddf4e5897acf3fe5ee44f34 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 6505fcf874c440788e377dac9aa99209 (127.5.240.199:33903) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term I0703 01:42:32.457299 28239 catalog_manager.cc:2903] Scheduling retry of ChangeConfig:REMOVE_PEER RPC for tablet 41c52d5ecddf4e5897acf3fe5ee44f34 with cas_config_opid_index 5 with a delay of 63 ms (attempt = 1) I0703 01:42:32.458102 30700 raft_consensus.cc:457] T 107baa5504214d5f8362cef92ed80291 P 35f0266742fc4707a8cc10c6437b84ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ee4739615124e308720b4aeddfe29cb) I0703 01:42:32.458182 30700 raft_consensus.cc:2730] T 107baa5504214d5f8362cef92ed80291 P 35f0266742fc4707a8cc10c6437b84ed: Snoozing failure detection for 1.694s (starting election) I0703 01:42:32.458267 30700 raft_consensus.cc:479] T 107baa5504214d5f8362cef92ed80291 P 35f0266742fc4707a8cc10c6437b84ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "35f0266742fc4707a8cc10c6437b84ed" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 50368 } attrs { promote: false } } I0703 01:42:32.458791 30700 leader_election.cc:243] T 107baa5504214d5f8362cef92ed80291 P 35f0266742fc4707a8cc10c6437b84ed [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 5ee4739615124e308720b4aeddfe29cb (127.5.240.193:54206) I0703 01:42:32.459513 28377 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "107baa5504214d5f8362cef92ed80291" candidate_uuid: "35f0266742fc4707a8cc10c6437b84ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "5ee4739615124e308720b4aeddfe29cb" is_pre_election: true I0703 01:42:32.459641 28377 raft_consensus.cc:2730] T 107baa5504214d5f8362cef92ed80291 P 5ee4739615124e308720b4aeddfe29cb: Snoozing failure detection for 1.676s (vote granted) I0703 01:42:32.459692 28377 raft_consensus.cc:2243] T 107baa5504214d5f8362cef92ed80291 P 5ee4739615124e308720b4aeddfe29cb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35f0266742fc4707a8cc10c6437b84ed in term 1. I0703 01:42:32.460057 29471 leader_election.cc:379] T 107baa5504214d5f8362cef92ed80291 P 35f0266742fc4707a8cc10c6437b84ed [CANDIDATE]: Term 2 pre-election: Vote granted by peer 5ee4739615124e308720b4aeddfe29cb (127.5.240.193:54206) I0703 01:42:32.460115 29471 leader_election.cc:270] T 107baa5504214d5f8362cef92ed80291 P 35f0266742fc4707a8cc10c6437b84ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. I0703 01:42:32.460357 30700 raft_consensus.cc:2730] T 107baa5504214d5f8362cef92ed80291 P 35f0266742fc4707a8cc10c6437b84ed: Snoozing failure detection for 1.716s (election complete) I0703 01:42:32.460454 30700 raft_consensus.cc:2537] T 107baa5504214d5f8362cef92ed80291 P 35f0266742fc4707a8cc10c6437b84ed [term 1 FOLLOWER]: Leader pre-election won for term 2 I0703 01:42:32.460511 30700 raft_consensus.cc:457] T 107baa5504214d5f8362cef92ed80291 P 35f0266742fc4707a8cc10c6437b84ed [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5ee4739615124e308720b4aeddfe29cb) I0703 01:42:32.460556 30700 raft_consensus.cc:2730] T 107baa5504214d5f8362cef92ed80291 P 35f0266742fc4707a8cc10c6437b84ed: Snoozing failure detection for 2.112s (starting election) I0703 01:42:32.460613 30700 raft_consensus.cc:2782] T 107baa5504214d5f8362cef92ed80291 P 35f0266742fc4707a8cc10c6437b84ed [term 1 FOLLOWER]: Advancing to term 2 I0703 01:42:32.461953 30700 raft_consensus.cc:479] T 107baa5504214d5f8362cef92ed80291 P 35f0266742fc4707a8cc10c6437b84ed [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "35f0266742fc4707a8cc10c6437b84ed" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 50368 } attrs { promote: false } } I0703 01:42:32.462196 30700 leader_election.cc:243] T 107baa5504214d5f8362cef92ed80291 P 35f0266742fc4707a8cc10c6437b84ed [CANDIDATE]: Term 2 election: Requesting vote from peer 5ee4739615124e308720b4aeddfe29cb (127.5.240.193:54206) I0703 01:42:32.462785 28377 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "107baa5504214d5f8362cef92ed80291" candidate_uuid: "35f0266742fc4707a8cc10c6437b84ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "5ee4739615124e308720b4aeddfe29cb" I0703 01:42:32.462882 28377 raft_consensus.cc:2782] T 107baa5504214d5f8362cef92ed80291 P 5ee4739615124e308720b4aeddfe29cb [term 1 FOLLOWER]: Advancing to term 2 I0703 01:42:32.462939 28377 raft_consensus.cc:2730] T 107baa5504214d5f8362cef92ed80291 P 5ee4739615124e308720b4aeddfe29cb: Snoozing failure detection for 1.971s (vote granted) I0703 01:42:32.464162 28377 raft_consensus.cc:2243] T 107baa5504214d5f8362cef92ed80291 P 5ee4739615124e308720b4aeddfe29cb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35f0266742fc4707a8cc10c6437b84ed in term 2. I0703 01:42:32.464545 29471 leader_election.cc:379] T 107baa5504214d5f8362cef92ed80291 P 35f0266742fc4707a8cc10c6437b84ed [CANDIDATE]: Term 2 election: Vote granted by peer 5ee4739615124e308720b4aeddfe29cb (127.5.240.193:54206) I0703 01:42:32.464610 29471 leader_election.cc:270] T 107baa5504214d5f8362cef92ed80291 P 35f0266742fc4707a8cc10c6437b84ed [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0703 01:42:32.464776 30700 raft_consensus.cc:2730] T 107baa5504214d5f8362cef92ed80291 P 35f0266742fc4707a8cc10c6437b84ed: Snoozing failure detection for 1.957s (election complete) I0703 01:42:32.464859 30700 raft_consensus.cc:2537] T 107baa5504214d5f8362cef92ed80291 P 35f0266742fc4707a8cc10c6437b84ed [term 2 FOLLOWER]: Leader election won for term 2 I0703 01:42:32.465133 30700 raft_consensus.cc:593] T 107baa5504214d5f8362cef92ed80291 P 35f0266742fc4707a8cc10c6437b84ed [term 2 LEADER]: Becoming Leader. State: Replica: 35f0266742fc4707a8cc10c6437b84ed, State: Running, Role: LEADER I0703 01:42:32.465400 30700 consensus_queue.cc:204] T 107baa5504214d5f8362cef92ed80291 P 35f0266742fc4707a8cc10c6437b84ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "35f0266742fc4707a8cc10c6437b84ed" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 50368 } attrs { promote: false } } I0703 01:42:32.467326 28253 catalog_manager.cc:3798] T 107baa5504214d5f8362cef92ed80291 P 35f0266742fc4707a8cc10c6437b84ed reported cstate change: term changed from 1 to 2, leader changed from 5ee4739615124e308720b4aeddfe29cb (127.5.240.193) to 35f0266742fc4707a8cc10c6437b84ed (127.5.240.196). New cstate: current_term: 2 leader_uuid: "35f0266742fc4707a8cc10c6437b84ed" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35f0266742fc4707a8cc10c6437b84ed" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 50368 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I0703 01:42:32.469161 28253 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 107baa5504214d5f8362cef92ed80291 (attempt 1) I0703 01:42:32.473598 28240 catalog_manager.cc:3369] ChangeConfig:REMOVE_PEER RPC for tablet 107baa5504214d5f8362cef92ed80291 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 35f0266742fc4707a8cc10c6437b84ed (127.5.240.196:50368) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term I0703 01:42:32.473660 28240 catalog_manager.cc:2903] Scheduling retry of ChangeConfig:REMOVE_PEER RPC for tablet 107baa5504214d5f8362cef92ed80291 with cas_config_opid_index 5 with a delay of 48 ms (attempt = 1) I0703 01:42:32.475518 30695 raft_consensus.cc:457] T ef65b31bc12e40c2b3f1e4714cc4218e P 575620f4889c4704b7a7c00da34b6a53 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 31c252b54241470ba1115181c3672664) I0703 01:42:32.475606 30695 raft_consensus.cc:2730] T ef65b31bc12e40c2b3f1e4714cc4218e P 575620f4889c4704b7a7c00da34b6a53: Snoozing failure detection for 2.225s (starting election) I0703 01:42:32.475682 30695 raft_consensus.cc:479] T ef65b31bc12e40c2b3f1e4714cc4218e P 575620f4889c4704b7a7c00da34b6a53 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } attrs { replace: true } } peers { permanent_uuid: "575620f4889c4704b7a7c00da34b6a53" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39152 } attrs { promote: false } } I0703 01:42:32.475929 30695 leader_election.cc:243] T ef65b31bc12e40c2b3f1e4714cc4218e P 575620f4889c4704b7a7c00da34b6a53 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 31c252b54241470ba1115181c3672664 (127.5.240.194:38625) I0703 01:42:32.476516 28500 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "ef65b31bc12e40c2b3f1e4714cc4218e" candidate_uuid: "575620f4889c4704b7a7c00da34b6a53" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "31c252b54241470ba1115181c3672664" is_pre_election: true I0703 01:42:32.476644 28500 raft_consensus.cc:2730] T ef65b31bc12e40c2b3f1e4714cc4218e P 31c252b54241470ba1115181c3672664: Snoozing failure detection for 1.763s (vote granted) I0703 01:42:32.476704 28500 raft_consensus.cc:2243] T ef65b31bc12e40c2b3f1e4714cc4218e P 31c252b54241470ba1115181c3672664 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 575620f4889c4704b7a7c00da34b6a53 in term 1. I0703 01:42:32.477126 29715 leader_election.cc:379] T ef65b31bc12e40c2b3f1e4714cc4218e P 575620f4889c4704b7a7c00da34b6a53 [CANDIDATE]: Term 2 pre-election: Vote granted by peer 31c252b54241470ba1115181c3672664 (127.5.240.194:38625) I0703 01:42:32.477180 29715 leader_election.cc:270] T ef65b31bc12e40c2b3f1e4714cc4218e P 575620f4889c4704b7a7c00da34b6a53 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. I0703 01:42:32.477334 30695 raft_consensus.cc:2730] T ef65b31bc12e40c2b3f1e4714cc4218e P 575620f4889c4704b7a7c00da34b6a53: Snoozing failure detection for 1.589s (election complete) I0703 01:42:32.477401 30695 raft_consensus.cc:2537] T ef65b31bc12e40c2b3f1e4714cc4218e P 575620f4889c4704b7a7c00da34b6a53 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0703 01:42:32.477452 30695 raft_consensus.cc:457] T ef65b31bc12e40c2b3f1e4714cc4218e P 575620f4889c4704b7a7c00da34b6a53 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 31c252b54241470ba1115181c3672664) I0703 01:42:32.477515 30695 raft_consensus.cc:2730] T ef65b31bc12e40c2b3f1e4714cc4218e P 575620f4889c4704b7a7c00da34b6a53: Snoozing failure detection for 1.822s (starting election) I0703 01:42:32.477573 30695 raft_consensus.cc:2782] T ef65b31bc12e40c2b3f1e4714cc4218e P 575620f4889c4704b7a7c00da34b6a53 [term 1 FOLLOWER]: Advancing to term 2 I0703 01:42:32.478838 30695 raft_consensus.cc:479] T ef65b31bc12e40c2b3f1e4714cc4218e P 575620f4889c4704b7a7c00da34b6a53 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } attrs { replace: true } } peers { permanent_uuid: "575620f4889c4704b7a7c00da34b6a53" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39152 } attrs { promote: false } } I0703 01:42:32.479086 30695 leader_election.cc:243] T ef65b31bc12e40c2b3f1e4714cc4218e P 575620f4889c4704b7a7c00da34b6a53 [CANDIDATE]: Term 2 election: Requesting vote from peer 31c252b54241470ba1115181c3672664 (127.5.240.194:38625) I0703 01:42:32.479701 28500 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "ef65b31bc12e40c2b3f1e4714cc4218e" candidate_uuid: "575620f4889c4704b7a7c00da34b6a53" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "31c252b54241470ba1115181c3672664" I0703 01:42:32.479804 28500 raft_consensus.cc:2782] T ef65b31bc12e40c2b3f1e4714cc4218e P 31c252b54241470ba1115181c3672664 [term 1 FOLLOWER]: Advancing to term 2 I0703 01:42:32.479873 28500 raft_consensus.cc:2730] T ef65b31bc12e40c2b3f1e4714cc4218e P 31c252b54241470ba1115181c3672664: Snoozing failure detection for 1.696s (vote granted) I0703 01:42:32.481094 28240 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 74b62768046f4b8690aab7d2ec0f803a (attempt 2) I0703 01:42:32.481182 28500 raft_consensus.cc:2243] T ef65b31bc12e40c2b3f1e4714cc4218e P 31c252b54241470ba1115181c3672664 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 575620f4889c4704b7a7c00da34b6a53 in term 2. I0703 01:42:32.481657 29715 leader_election.cc:379] T ef65b31bc12e40c2b3f1e4714cc4218e P 575620f4889c4704b7a7c00da34b6a53 [CANDIDATE]: Term 2 election: Vote granted by peer 31c252b54241470ba1115181c3672664 (127.5.240.194:38625) I0703 01:42:32.481705 29715 leader_election.cc:270] T ef65b31bc12e40c2b3f1e4714cc4218e P 575620f4889c4704b7a7c00da34b6a53 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0703 01:42:32.481878 30695 raft_consensus.cc:2730] T ef65b31bc12e40c2b3f1e4714cc4218e P 575620f4889c4704b7a7c00da34b6a53: Snoozing failure detection for 2.080s (election complete) I0703 01:42:32.481977 30695 raft_consensus.cc:2537] T ef65b31bc12e40c2b3f1e4714cc4218e P 575620f4889c4704b7a7c00da34b6a53 [term 2 FOLLOWER]: Leader election won for term 2 I0703 01:42:32.481959 28240 catalog_manager.cc:3369] ChangeConfig:REMOVE_PEER RPC for tablet 74b62768046f4b8690aab7d2ec0f803a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 575620f4889c4704b7a7c00da34b6a53 (127.5.240.198:39152) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term I0703 01:42:32.482035 28240 catalog_manager.cc:2903] Scheduling retry of ChangeConfig:REMOVE_PEER RPC for tablet 74b62768046f4b8690aab7d2ec0f803a with cas_config_opid_index 5 with a delay of 35 ms (attempt = 2) I0703 01:42:32.482112 30695 raft_consensus.cc:593] T ef65b31bc12e40c2b3f1e4714cc4218e P 575620f4889c4704b7a7c00da34b6a53 [term 2 LEADER]: Becoming Leader. State: Replica: 575620f4889c4704b7a7c00da34b6a53, State: Running, Role: LEADER I0703 01:42:32.482273 30695 consensus_queue.cc:204] T ef65b31bc12e40c2b3f1e4714cc4218e P 575620f4889c4704b7a7c00da34b6a53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } attrs { replace: true } } peers { permanent_uuid: "575620f4889c4704b7a7c00da34b6a53" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39152 } attrs { promote: false } } I0703 01:42:32.483835 28253 catalog_manager.cc:3798] T ef65b31bc12e40c2b3f1e4714cc4218e P 575620f4889c4704b7a7c00da34b6a53 reported cstate change: term changed from 1 to 2, leader changed from 31c252b54241470ba1115181c3672664 (127.5.240.194) to 575620f4889c4704b7a7c00da34b6a53 (127.5.240.198). New cstate: current_term: 2 leader_uuid: "575620f4889c4704b7a7c00da34b6a53" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "575620f4889c4704b7a7c00da34b6a53" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39152 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I0703 01:42:32.485486 28253 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet ef65b31bc12e40c2b3f1e4714cc4218e (attempt 1) I0703 01:42:32.486418 28240 catalog_manager.cc:3369] ChangeConfig:REMOVE_PEER RPC for tablet ef65b31bc12e40c2b3f1e4714cc4218e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 575620f4889c4704b7a7c00da34b6a53 (127.5.240.198:39152) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term I0703 01:42:32.486488 28240 catalog_manager.cc:2903] Scheduling retry of ChangeConfig:REMOVE_PEER RPC for tablet ef65b31bc12e40c2b3f1e4714cc4218e with cas_config_opid_index 5 with a delay of 51 ms (attempt = 1) I0703 01:42:32.488878 30700 raft_consensus.cc:457] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 35f0266742fc4707a8cc10c6437b84ed [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 31c252b54241470ba1115181c3672664) I0703 01:42:32.488936 30700 raft_consensus.cc:2730] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 35f0266742fc4707a8cc10c6437b84ed: Snoozing failure detection for 1.878s (starting election) I0703 01:42:32.488992 30700 raft_consensus.cc:479] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 35f0266742fc4707a8cc10c6437b84ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } attrs { replace: true } } peers { permanent_uuid: "35f0266742fc4707a8cc10c6437b84ed" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 50368 } attrs { promote: false } } I0703 01:42:32.489210 30700 leader_election.cc:243] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 35f0266742fc4707a8cc10c6437b84ed [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 31c252b54241470ba1115181c3672664 (127.5.240.194:38625) I0703 01:42:32.489823 28500 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "1cd5cbfe70ec4f8a957fa074ae08fcb1" candidate_uuid: "35f0266742fc4707a8cc10c6437b84ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "31c252b54241470ba1115181c3672664" is_pre_election: true I0703 01:42:32.489943 28500 raft_consensus.cc:2730] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 31c252b54241470ba1115181c3672664: Snoozing failure detection for 1.737s (vote granted) I0703 01:42:32.490005 28500 raft_consensus.cc:2243] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 31c252b54241470ba1115181c3672664 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35f0266742fc4707a8cc10c6437b84ed in term 1. I0703 01:42:32.490406 29471 leader_election.cc:379] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 35f0266742fc4707a8cc10c6437b84ed [CANDIDATE]: Term 2 pre-election: Vote granted by peer 31c252b54241470ba1115181c3672664 (127.5.240.194:38625) I0703 01:42:32.490461 29471 leader_election.cc:270] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 35f0266742fc4707a8cc10c6437b84ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. I0703 01:42:32.490635 30700 raft_consensus.cc:2730] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 35f0266742fc4707a8cc10c6437b84ed: Snoozing failure detection for 1.690s (election complete) I0703 01:42:32.490696 30700 raft_consensus.cc:2537] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 35f0266742fc4707a8cc10c6437b84ed [term 1 FOLLOWER]: Leader pre-election won for term 2 I0703 01:42:32.490733 30700 raft_consensus.cc:457] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 35f0266742fc4707a8cc10c6437b84ed [term 1 FOLLOWER]: Starting leader election (detected failure of leader 31c252b54241470ba1115181c3672664) I0703 01:42:32.490775 30700 raft_consensus.cc:2730] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 35f0266742fc4707a8cc10c6437b84ed: Snoozing failure detection for 1.767s (starting election) I0703 01:42:32.490813 30700 raft_consensus.cc:2782] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 35f0266742fc4707a8cc10c6437b84ed [term 1 FOLLOWER]: Advancing to term 2 I0703 01:42:32.492054 30700 raft_consensus.cc:479] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 35f0266742fc4707a8cc10c6437b84ed [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } attrs { replace: true } } peers { permanent_uuid: "35f0266742fc4707a8cc10c6437b84ed" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 50368 } attrs { promote: false } } I0703 01:42:32.492287 30700 leader_election.cc:243] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 35f0266742fc4707a8cc10c6437b84ed [CANDIDATE]: Term 2 election: Requesting vote from peer 31c252b54241470ba1115181c3672664 (127.5.240.194:38625) I0703 01:42:32.492943 28500 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "1cd5cbfe70ec4f8a957fa074ae08fcb1" candidate_uuid: "35f0266742fc4707a8cc10c6437b84ed" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "31c252b54241470ba1115181c3672664" I0703 01:42:32.493047 28500 raft_consensus.cc:2782] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 31c252b54241470ba1115181c3672664 [term 1 FOLLOWER]: Advancing to term 2 I0703 01:42:32.493108 28500 raft_consensus.cc:2730] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 31c252b54241470ba1115181c3672664: Snoozing failure detection for 1.535s (vote granted) I0703 01:42:32.494385 28500 raft_consensus.cc:2243] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 31c252b54241470ba1115181c3672664 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35f0266742fc4707a8cc10c6437b84ed in term 2. I0703 01:42:32.494858 29471 leader_election.cc:379] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 35f0266742fc4707a8cc10c6437b84ed [CANDIDATE]: Term 2 election: Vote granted by peer 31c252b54241470ba1115181c3672664 (127.5.240.194:38625) I0703 01:42:32.494931 29471 leader_election.cc:270] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 35f0266742fc4707a8cc10c6437b84ed [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0703 01:42:32.495131 30700 raft_consensus.cc:2730] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 35f0266742fc4707a8cc10c6437b84ed: Snoozing failure detection for 1.575s (election complete) I0703 01:42:32.495185 30700 raft_consensus.cc:2537] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 35f0266742fc4707a8cc10c6437b84ed [term 2 FOLLOWER]: Leader election won for term 2 I0703 01:42:32.495278 30700 raft_consensus.cc:593] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 35f0266742fc4707a8cc10c6437b84ed [term 2 LEADER]: Becoming Leader. State: Replica: 35f0266742fc4707a8cc10c6437b84ed, State: Running, Role: LEADER I0703 01:42:32.495424 30700 consensus_queue.cc:204] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 35f0266742fc4707a8cc10c6437b84ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } attrs { replace: true } } peers { permanent_uuid: "35f0266742fc4707a8cc10c6437b84ed" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 50368 } attrs { promote: false } } I0703 01:42:32.497032 28253 catalog_manager.cc:3798] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 35f0266742fc4707a8cc10c6437b84ed reported cstate change: term changed from 1 to 2, leader changed from 31c252b54241470ba1115181c3672664 (127.5.240.194) to 35f0266742fc4707a8cc10c6437b84ed (127.5.240.196). New cstate: current_term: 2 leader_uuid: "35f0266742fc4707a8cc10c6437b84ed" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35f0266742fc4707a8cc10c6437b84ed" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 50368 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I0703 01:42:32.498661 28253 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 1cd5cbfe70ec4f8a957fa074ae08fcb1 (attempt 1) I0703 01:42:32.499393 28240 catalog_manager.cc:3369] ChangeConfig:REMOVE_PEER RPC for tablet 1cd5cbfe70ec4f8a957fa074ae08fcb1 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 35f0266742fc4707a8cc10c6437b84ed (127.5.240.196:50368) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term I0703 01:42:32.499464 28240 catalog_manager.cc:2903] Scheduling retry of ChangeConfig:REMOVE_PEER RPC for tablet 1cd5cbfe70ec4f8a957fa074ae08fcb1 with cas_config_opid_index 5 with a delay of 17 ms (attempt = 1) I0703 01:42:32.516852 28240 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 1cd5cbfe70ec4f8a957fa074ae08fcb1 (attempt 2) I0703 01:42:32.517426 28240 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 74b62768046f4b8690aab7d2ec0f803a (attempt 3) I0703 01:42:32.517570 28240 catalog_manager.cc:3369] ChangeConfig:REMOVE_PEER RPC for tablet 1cd5cbfe70ec4f8a957fa074ae08fcb1 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 35f0266742fc4707a8cc10c6437b84ed (127.5.240.196:50368) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term I0703 01:42:32.517657 28240 catalog_manager.cc:2903] Scheduling retry of ChangeConfig:REMOVE_PEER RPC for tablet 1cd5cbfe70ec4f8a957fa074ae08fcb1 with cas_config_opid_index 5 with a delay of 39 ms (attempt = 2) I0703 01:42:32.518256 28240 catalog_manager.cc:3369] ChangeConfig:REMOVE_PEER RPC for tablet 74b62768046f4b8690aab7d2ec0f803a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 575620f4889c4704b7a7c00da34b6a53 (127.5.240.198:39152) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term I0703 01:42:32.518326 28240 catalog_manager.cc:2903] Scheduling retry of ChangeConfig:REMOVE_PEER RPC for tablet 74b62768046f4b8690aab7d2ec0f803a with cas_config_opid_index 5 with a delay of 112 ms (attempt = 3) I0703 01:42:32.520460 28239 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 4d69010513df4b68840b90be3161f402 (attempt 4) I0703 01:42:32.520619 28239 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 41c52d5ecddf4e5897acf3fe5ee44f34 (attempt 2) I0703 01:42:32.521436 28239 catalog_manager.cc:3369] ChangeConfig:REMOVE_PEER RPC for tablet 41c52d5ecddf4e5897acf3fe5ee44f34 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 6505fcf874c440788e377dac9aa99209 (127.5.240.199:33903) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term I0703 01:42:32.521562 28239 catalog_manager.cc:2903] Scheduling retry of ChangeConfig:REMOVE_PEER RPC for tablet 41c52d5ecddf4e5897acf3fe5ee44f34 with cas_config_opid_index 5 with a delay of 81 ms (attempt = 2) I0703 01:42:32.521692 28239 catalog_manager.cc:3369] ChangeConfig:REMOVE_PEER RPC for tablet 4d69010513df4b68840b90be3161f402 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 0ae780b86cc348d3a55bbaf5cf5a6217 (127.5.240.195:46219) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term I0703 01:42:32.521793 28239 catalog_manager.cc:2903] Scheduling retry of ChangeConfig:REMOVE_PEER RPC for tablet 4d69010513df4b68840b90be3161f402 with cas_config_opid_index 5 with a delay of 132 ms (attempt = 4) I0703 01:42:32.522665 28240 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 107baa5504214d5f8362cef92ed80291 (attempt 2) I0703 01:42:32.523277 28240 catalog_manager.cc:3369] ChangeConfig:REMOVE_PEER RPC for tablet 107baa5504214d5f8362cef92ed80291 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 35f0266742fc4707a8cc10c6437b84ed (127.5.240.196:50368) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term I0703 01:42:32.523342 28240 catalog_manager.cc:2903] Scheduling retry of ChangeConfig:REMOVE_PEER RPC for tablet 107baa5504214d5f8362cef92ed80291 with cas_config_opid_index 5 with a delay of 43 ms (attempt = 2) I0703 01:42:32.535851 28240 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 45b3833ac0664a8d9889ab14b3d0dd69 (attempt 5) I0703 01:42:32.536762 28240 catalog_manager.cc:3369] ChangeConfig:REMOVE_PEER RPC for tablet 45b3833ac0664a8d9889ab14b3d0dd69 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader e98d29f36f2347f29170e987cb83ced4 (127.5.240.197:41466) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term I0703 01:42:32.536833 28240 catalog_manager.cc:2903] Scheduling retry of ChangeConfig:REMOVE_PEER RPC for tablet 45b3833ac0664a8d9889ab14b3d0dd69 with cas_config_opid_index 5 with a delay of 262 ms (attempt = 5) I0703 01:42:32.538131 28240 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet ef65b31bc12e40c2b3f1e4714cc4218e (attempt 2) I0703 01:42:32.538745 28240 catalog_manager.cc:3369] ChangeConfig:REMOVE_PEER RPC for tablet ef65b31bc12e40c2b3f1e4714cc4218e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 575620f4889c4704b7a7c00da34b6a53 (127.5.240.198:39152) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term I0703 01:42:32.538810 28240 catalog_manager.cc:2903] Scheduling retry of ChangeConfig:REMOVE_PEER RPC for tablet ef65b31bc12e40c2b3f1e4714cc4218e with cas_config_opid_index 5 with a delay of 66 ms (attempt = 2) I0703 01:42:32.557174 28240 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 1cd5cbfe70ec4f8a957fa074ae08fcb1 (attempt 3) I0703 01:42:32.558012 28240 catalog_manager.cc:3369] ChangeConfig:REMOVE_PEER RPC for tablet 1cd5cbfe70ec4f8a957fa074ae08fcb1 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 35f0266742fc4707a8cc10c6437b84ed (127.5.240.196:50368) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term I0703 01:42:32.558097 28240 catalog_manager.cc:2903] Scheduling retry of ChangeConfig:REMOVE_PEER RPC for tablet 1cd5cbfe70ec4f8a957fa074ae08fcb1 with cas_config_opid_index 5 with a delay of 67 ms (attempt = 3) I0703 01:42:32.566673 28240 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 107baa5504214d5f8362cef92ed80291 (attempt 3) I0703 01:42:32.567548 28240 catalog_manager.cc:3369] ChangeConfig:REMOVE_PEER RPC for tablet 107baa5504214d5f8362cef92ed80291 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 35f0266742fc4707a8cc10c6437b84ed (127.5.240.196:50368) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term I0703 01:42:32.567646 28240 catalog_manager.cc:2903] Scheduling retry of ChangeConfig:REMOVE_PEER RPC for tablet 107baa5504214d5f8362cef92ed80291 with cas_config_opid_index 5 with a delay of 89 ms (attempt = 3) I0703 01:42:32.574273 28239 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet b4acd291b108434da9b8791c61fd1dc2 (attempt 5) I0703 01:42:32.575125 28239 catalog_manager.cc:3369] ChangeConfig:REMOVE_PEER RPC for tablet b4acd291b108434da9b8791c61fd1dc2 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 6505fcf874c440788e377dac9aa99209 (127.5.240.199:33903) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term I0703 01:42:32.575217 28239 catalog_manager.cc:2903] Scheduling retry of ChangeConfig:REMOVE_PEER RPC for tablet b4acd291b108434da9b8791c61fd1dc2 with cas_config_opid_index 5 with a delay of 290 ms (attempt = 5) I0703 01:42:32.603646 28239 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 41c52d5ecddf4e5897acf3fe5ee44f34 (attempt 3) I0703 01:42:32.604249 28239 catalog_manager.cc:3369] ChangeConfig:REMOVE_PEER RPC for tablet 41c52d5ecddf4e5897acf3fe5ee44f34 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 6505fcf874c440788e377dac9aa99209 (127.5.240.199:33903) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term I0703 01:42:32.604316 28239 catalog_manager.cc:2903] Scheduling retry of ChangeConfig:REMOVE_PEER RPC for tablet 41c52d5ecddf4e5897acf3fe5ee44f34 with cas_config_opid_index 5 with a delay of 92 ms (attempt = 3) I0703 01:42:32.605686 28240 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet ef65b31bc12e40c2b3f1e4714cc4218e (attempt 3) I0703 01:42:32.606334 28240 catalog_manager.cc:3369] ChangeConfig:REMOVE_PEER RPC for tablet ef65b31bc12e40c2b3f1e4714cc4218e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 575620f4889c4704b7a7c00da34b6a53 (127.5.240.198:39152) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term I0703 01:42:32.606398 28240 catalog_manager.cc:2903] Scheduling retry of ChangeConfig:REMOVE_PEER RPC for tablet ef65b31bc12e40c2b3f1e4714cc4218e with cas_config_opid_index 5 with a delay of 111 ms (attempt = 3) I0703 01:42:32.617357 30633 raft_consensus.cc:457] T 775bd6dae9ed48ebbbb9df088d833772 P e98d29f36f2347f29170e987cb83ced4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 31c252b54241470ba1115181c3672664) I0703 01:42:32.617434 30633 raft_consensus.cc:2730] T 775bd6dae9ed48ebbbb9df088d833772 P e98d29f36f2347f29170e987cb83ced4: Snoozing failure detection for 2.063s (starting election) I0703 01:42:32.617514 30633 raft_consensus.cc:479] T 775bd6dae9ed48ebbbb9df088d833772 P e98d29f36f2347f29170e987cb83ced4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } attrs { replace: true } } peers { permanent_uuid: "e98d29f36f2347f29170e987cb83ced4" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 41466 } attrs { promote: false } } I0703 01:42:32.617837 30633 leader_election.cc:243] T 775bd6dae9ed48ebbbb9df088d833772 P e98d29f36f2347f29170e987cb83ced4 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 31c252b54241470ba1115181c3672664 (127.5.240.194:38625) I0703 01:42:32.618543 28500 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "775bd6dae9ed48ebbbb9df088d833772" candidate_uuid: "e98d29f36f2347f29170e987cb83ced4" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "31c252b54241470ba1115181c3672664" is_pre_election: true I0703 01:42:32.618700 28500 raft_consensus.cc:2730] T 775bd6dae9ed48ebbbb9df088d833772 P 31c252b54241470ba1115181c3672664: Snoozing failure detection for 1.502s (vote granted) I0703 01:42:32.618769 28500 raft_consensus.cc:2243] T 775bd6dae9ed48ebbbb9df088d833772 P 31c252b54241470ba1115181c3672664 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e98d29f36f2347f29170e987cb83ced4 in term 1. I0703 01:42:32.619176 29593 leader_election.cc:379] T 775bd6dae9ed48ebbbb9df088d833772 P e98d29f36f2347f29170e987cb83ced4 [CANDIDATE]: Term 2 pre-election: Vote granted by peer 31c252b54241470ba1115181c3672664 (127.5.240.194:38625) I0703 01:42:32.619231 29593 leader_election.cc:270] T 775bd6dae9ed48ebbbb9df088d833772 P e98d29f36f2347f29170e987cb83ced4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. I0703 01:42:32.619395 30633 raft_consensus.cc:2730] T 775bd6dae9ed48ebbbb9df088d833772 P e98d29f36f2347f29170e987cb83ced4: Snoozing failure detection for 2.055s (election complete) I0703 01:42:32.619488 30633 raft_consensus.cc:2537] T 775bd6dae9ed48ebbbb9df088d833772 P e98d29f36f2347f29170e987cb83ced4 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0703 01:42:32.619536 30633 raft_consensus.cc:457] T 775bd6dae9ed48ebbbb9df088d833772 P e98d29f36f2347f29170e987cb83ced4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 31c252b54241470ba1115181c3672664) I0703 01:42:32.619616 30633 raft_consensus.cc:2730] T 775bd6dae9ed48ebbbb9df088d833772 P e98d29f36f2347f29170e987cb83ced4: Snoozing failure detection for 1.994s (starting election) I0703 01:42:32.619675 30633 raft_consensus.cc:2782] T 775bd6dae9ed48ebbbb9df088d833772 P e98d29f36f2347f29170e987cb83ced4 [term 1 FOLLOWER]: Advancing to term 2 I0703 01:42:32.621196 30633 raft_consensus.cc:479] T 775bd6dae9ed48ebbbb9df088d833772 P e98d29f36f2347f29170e987cb83ced4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } attrs { replace: true } } peers { permanent_uuid: "e98d29f36f2347f29170e987cb83ced4" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 41466 } attrs { promote: false } } I0703 01:42:32.621467 30633 leader_election.cc:243] T 775bd6dae9ed48ebbbb9df088d833772 P e98d29f36f2347f29170e987cb83ced4 [CANDIDATE]: Term 2 election: Requesting vote from peer 31c252b54241470ba1115181c3672664 (127.5.240.194:38625) I0703 01:42:32.622081 28500 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "775bd6dae9ed48ebbbb9df088d833772" candidate_uuid: "e98d29f36f2347f29170e987cb83ced4" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "31c252b54241470ba1115181c3672664" I0703 01:42:32.622185 28500 raft_consensus.cc:2782] T 775bd6dae9ed48ebbbb9df088d833772 P 31c252b54241470ba1115181c3672664 [term 1 FOLLOWER]: Advancing to term 2 I0703 01:42:32.622253 28500 raft_consensus.cc:2730] T 775bd6dae9ed48ebbbb9df088d833772 P 31c252b54241470ba1115181c3672664: Snoozing failure detection for 1.599s (vote granted) I0703 01:42:32.623697 28500 raft_consensus.cc:2243] T 775bd6dae9ed48ebbbb9df088d833772 P 31c252b54241470ba1115181c3672664 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e98d29f36f2347f29170e987cb83ced4 in term 2. I0703 01:42:32.624078 29593 leader_election.cc:379] T 775bd6dae9ed48ebbbb9df088d833772 P e98d29f36f2347f29170e987cb83ced4 [CANDIDATE]: Term 2 election: Vote granted by peer 31c252b54241470ba1115181c3672664 (127.5.240.194:38625) I0703 01:42:32.624130 29593 leader_election.cc:270] T 775bd6dae9ed48ebbbb9df088d833772 P e98d29f36f2347f29170e987cb83ced4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0703 01:42:32.624301 30633 raft_consensus.cc:2730] T 775bd6dae9ed48ebbbb9df088d833772 P e98d29f36f2347f29170e987cb83ced4: Snoozing failure detection for 1.960s (election complete) I0703 01:42:32.624362 30633 raft_consensus.cc:2537] T 775bd6dae9ed48ebbbb9df088d833772 P e98d29f36f2347f29170e987cb83ced4 [term 2 FOLLOWER]: Leader election won for term 2 I0703 01:42:32.624476 30633 raft_consensus.cc:593] T 775bd6dae9ed48ebbbb9df088d833772 P e98d29f36f2347f29170e987cb83ced4 [term 2 LEADER]: Becoming Leader. State: Replica: e98d29f36f2347f29170e987cb83ced4, State: Running, Role: LEADER I0703 01:42:32.624667 30633 consensus_queue.cc:204] T 775bd6dae9ed48ebbbb9df088d833772 P e98d29f36f2347f29170e987cb83ced4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } attrs { replace: true } } peers { permanent_uuid: "e98d29f36f2347f29170e987cb83ced4" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 41466 } attrs { promote: false } } I0703 01:42:32.626154 28240 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 1cd5cbfe70ec4f8a957fa074ae08fcb1 (attempt 4) I0703 01:42:32.626703 28253 catalog_manager.cc:3798] T 775bd6dae9ed48ebbbb9df088d833772 P e98d29f36f2347f29170e987cb83ced4 reported cstate change: term changed from 1 to 2, leader changed from 31c252b54241470ba1115181c3672664 (127.5.240.194) to e98d29f36f2347f29170e987cb83ced4 (127.5.240.197). New cstate: current_term: 2 leader_uuid: "e98d29f36f2347f29170e987cb83ced4" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e98d29f36f2347f29170e987cb83ced4" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 41466 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I0703 01:42:32.627018 28240 catalog_manager.cc:3369] ChangeConfig:REMOVE_PEER RPC for tablet 1cd5cbfe70ec4f8a957fa074ae08fcb1 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 35f0266742fc4707a8cc10c6437b84ed (127.5.240.196:50368) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term I0703 01:42:32.627110 28240 catalog_manager.cc:2903] Scheduling retry of ChangeConfig:REMOVE_PEER RPC for tablet 1cd5cbfe70ec4f8a957fa074ae08fcb1 with cas_config_opid_index 5 with a delay of 141 ms (attempt = 4) I0703 01:42:32.628968 28253 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 775bd6dae9ed48ebbbb9df088d833772 (attempt 1) I0703 01:42:32.629827 28240 catalog_manager.cc:3369] ChangeConfig:REMOVE_PEER RPC for tablet 775bd6dae9ed48ebbbb9df088d833772 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader e98d29f36f2347f29170e987cb83ced4 (127.5.240.197:41466) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term I0703 01:42:32.629900 28240 catalog_manager.cc:2903] Scheduling retry of ChangeConfig:REMOVE_PEER RPC for tablet 775bd6dae9ed48ebbbb9df088d833772 with cas_config_opid_index 5 with a delay of 49 ms (attempt = 1) I0703 01:42:32.631161 28240 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 74b62768046f4b8690aab7d2ec0f803a (attempt 4) I0703 01:42:32.631887 28240 catalog_manager.cc:3369] ChangeConfig:REMOVE_PEER RPC for tablet 74b62768046f4b8690aab7d2ec0f803a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 575620f4889c4704b7a7c00da34b6a53 (127.5.240.198:39152) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term I0703 01:42:32.631955 28240 catalog_manager.cc:2903] Scheduling retry of ChangeConfig:REMOVE_PEER RPC for tablet 74b62768046f4b8690aab7d2ec0f803a with cas_config_opid_index 5 with a delay of 143 ms (attempt = 4) I0703 01:42:32.655011 28239 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 4d69010513df4b68840b90be3161f402 (attempt 5) I0703 01:42:32.655838 28239 catalog_manager.cc:3369] ChangeConfig:REMOVE_PEER RPC for tablet 4d69010513df4b68840b90be3161f402 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 0ae780b86cc348d3a55bbaf5cf5a6217 (127.5.240.195:46219) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term I0703 01:42:32.655900 28239 catalog_manager.cc:2903] Scheduling retry of ChangeConfig:REMOVE_PEER RPC for tablet 4d69010513df4b68840b90be3161f402 with cas_config_opid_index 5 with a delay of 266 ms (attempt = 5) I0703 01:42:32.657367 28240 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 107baa5504214d5f8362cef92ed80291 (attempt 4) I0703 01:42:32.658011 28240 catalog_manager.cc:3369] ChangeConfig:REMOVE_PEER RPC for tablet 107baa5504214d5f8362cef92ed80291 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 35f0266742fc4707a8cc10c6437b84ed (127.5.240.196:50368) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term I0703 01:42:32.658072 28240 catalog_manager.cc:2903] Scheduling retry of ChangeConfig:REMOVE_PEER RPC for tablet 107baa5504214d5f8362cef92ed80291 with cas_config_opid_index 5 with a delay of 159 ms (attempt = 4) I0703 01:42:32.666793 28377 raft_consensus.cc:1081] T 45b3833ac0664a8d9889ab14b3d0dd69 P 5ee4739615124e308720b4aeddfe29cb [term 2 FOLLOWER]: Refusing update from remote peer e98d29f36f2347f29170e987cb83ced4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch) I0703 01:42:32.667577 30633 consensus_queue.cc:962] T 45b3833ac0664a8d9889ab14b3d0dd69 P e98d29f36f2347f29170e987cb83ced4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I0703 01:42:32.672143 28253 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 45b3833ac0664a8d9889ab14b3d0dd69 (attempt 1) I0703 01:42:32.673252 29662 consensus_peers.cc:455] T 45b3833ac0664a8d9889ab14b3d0dd69 P e98d29f36f2347f29170e987cb83ced4 -> Peer 5ee4739615124e308720b4aeddfe29cb (127.5.240.193:54206): Closing peer: 5ee4739615124e308720b4aeddfe29cb I0703 01:42:32.673614 29662 consensus_queue.cc:204] T 45b3833ac0664a8d9889ab14b3d0dd69 P e98d29f36f2347f29170e987cb83ced4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e98d29f36f2347f29170e987cb83ced4" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 41466 } attrs { promote: false } } I0703 01:42:32.674173 30633 raft_consensus.cc:2684] T 45b3833ac0664a8d9889ab14b3d0dd69 P e98d29f36f2347f29170e987cb83ced4 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 5ee4739615124e308720b4aeddfe29cb (127.5.240.193) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e98d29f36f2347f29170e987cb83ced4" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 41466 } attrs { promote: false } } } I0703 01:42:32.676337 28240 catalog_manager.cc:3352] ChangeConfig:REMOVE_PEER RPC for tablet 45b3833ac0664a8d9889ab14b3d0dd69 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0703 01:42:32.677139 28253 catalog_manager.cc:3798] T 45b3833ac0664a8d9889ab14b3d0dd69 P e98d29f36f2347f29170e987cb83ced4 reported cstate change: config changed from index 5 to 7, VOTER 5ee4739615124e308720b4aeddfe29cb (127.5.240.193) evicted. New cstate: current_term: 2 leader_uuid: "e98d29f36f2347f29170e987cb83ced4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e98d29f36f2347f29170e987cb83ced4" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 41466 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I0703 01:42:32.679682 28253 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 45b3833ac0664a8d9889ab14b3d0dd69 on 5ee4739615124e308720b4aeddfe29cb (127.5.240.193:54206) (TS 5ee4739615124e308720b4aeddfe29cb not found in new config with opid_index 7) I0703 01:42:32.680070 28240 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 775bd6dae9ed48ebbbb9df088d833772 (attempt 2) I0703 01:42:32.680193 28356 tablet_service.cc:800] Processing DeleteTablet for tablet 45b3833ac0664a8d9889ab14b3d0dd69 with delete_type TABLET_DATA_TOMBSTONED (TS 5ee4739615124e308720b4aeddfe29cb not found in new config with opid_index 7) from {username='jenkins'} at 127.5.240.193:54680 I0703 01:42:32.680805 30754 tablet_replica.cc:261] T 45b3833ac0664a8d9889ab14b3d0dd69 P 5ee4739615124e308720b4aeddfe29cb: stopping tablet replica I0703 01:42:32.680845 28240 catalog_manager.cc:3369] ChangeConfig:REMOVE_PEER RPC for tablet 775bd6dae9ed48ebbbb9df088d833772 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader e98d29f36f2347f29170e987cb83ced4 (127.5.240.197:41466) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term I0703 01:42:32.680929 28240 catalog_manager.cc:2903] Scheduling retry of ChangeConfig:REMOVE_PEER RPC for tablet 775bd6dae9ed48ebbbb9df088d833772 with cas_config_opid_index 5 with a delay of 46 ms (attempt = 2) I0703 01:42:32.681042 30754 maintenance_manager.cc:237] P 5ee4739615124e308720b4aeddfe29cb: Unregistered op CompactRowSetsOp(45b3833ac0664a8d9889ab14b3d0dd69) I0703 01:42:32.681120 30754 maintenance_manager.cc:237] P 5ee4739615124e308720b4aeddfe29cb: Unregistered op MinorDeltaCompactionOp(45b3833ac0664a8d9889ab14b3d0dd69) I0703 01:42:32.681175 30754 maintenance_manager.cc:237] P 5ee4739615124e308720b4aeddfe29cb: Unregistered op MajorDeltaCompactionOp(45b3833ac0664a8d9889ab14b3d0dd69) I0703 01:42:32.681236 30754 maintenance_manager.cc:237] P 5ee4739615124e308720b4aeddfe29cb: Unregistered op UndoDeltaBlockGCOp(45b3833ac0664a8d9889ab14b3d0dd69) I0703 01:42:32.681319 30754 maintenance_manager.cc:237] P 5ee4739615124e308720b4aeddfe29cb: Unregistered op FlushMRSOp(45b3833ac0664a8d9889ab14b3d0dd69) I0703 01:42:32.681390 30754 maintenance_manager.cc:237] P 5ee4739615124e308720b4aeddfe29cb: Unregistered op FlushDeltaMemStoresOp(45b3833ac0664a8d9889ab14b3d0dd69) I0703 01:42:32.681433 30754 maintenance_manager.cc:237] P 5ee4739615124e308720b4aeddfe29cb: Unregistered op LogGCOp(45b3833ac0664a8d9889ab14b3d0dd69) I0703 01:42:32.681496 30754 raft_consensus.cc:2040] T 45b3833ac0664a8d9889ab14b3d0dd69 P 5ee4739615124e308720b4aeddfe29cb [term 2 FOLLOWER]: Raft consensus shutting down. I0703 01:42:32.681679 30754 raft_consensus.cc:2067] T 45b3833ac0664a8d9889ab14b3d0dd69 P 5ee4739615124e308720b4aeddfe29cb [term 2 FOLLOWER]: Raft consensus is shut down! I0703 01:42:32.682837 30754 ts_tablet_manager.cc:1369] T 45b3833ac0664a8d9889ab14b3d0dd69 P 5ee4739615124e308720b4aeddfe29cb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:42:32.685701 30754 ts_tablet_manager.cc:1382] T 45b3833ac0664a8d9889ab14b3d0dd69 P 5ee4739615124e308720b4aeddfe29cb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6 I0703 01:42:32.685786 30754 log.cc:986] T 45b3833ac0664a8d9889ab14b3d0dd69 P 5ee4739615124e308720b4aeddfe29cb: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal/wals/45b3833ac0664a8d9889ab14b3d0dd69 I0703 01:42:32.686379 28238 catalog_manager.cc:3103] TS 5ee4739615124e308720b4aeddfe29cb (127.5.240.193:54206): tablet 45b3833ac0664a8d9889ab14b3d0dd69 (table rebalance_test_table_0 [id=2fa00f90e0e84298b5fa673ff8ab7e37]) successfully deleted I0703 01:42:32.697273 28239 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 41c52d5ecddf4e5897acf3fe5ee44f34 (attempt 4) I0703 01:42:32.698038 28239 catalog_manager.cc:3369] ChangeConfig:REMOVE_PEER RPC for tablet 41c52d5ecddf4e5897acf3fe5ee44f34 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 6505fcf874c440788e377dac9aa99209 (127.5.240.199:33903) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term I0703 01:42:32.698109 28239 catalog_manager.cc:2903] Scheduling retry of ChangeConfig:REMOVE_PEER RPC for tablet 41c52d5ecddf4e5897acf3fe5ee44f34 with cas_config_opid_index 5 with a delay of 163 ms (attempt = 4) I0703 01:42:32.718361 28240 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet ef65b31bc12e40c2b3f1e4714cc4218e (attempt 4) I0703 01:42:32.719095 28240 catalog_manager.cc:3369] ChangeConfig:REMOVE_PEER RPC for tablet ef65b31bc12e40c2b3f1e4714cc4218e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 575620f4889c4704b7a7c00da34b6a53 (127.5.240.198:39152) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term I0703 01:42:32.719158 28240 catalog_manager.cc:2903] Scheduling retry of ChangeConfig:REMOVE_PEER RPC for tablet ef65b31bc12e40c2b3f1e4714cc4218e with cas_config_opid_index 5 with a delay of 175 ms (attempt = 4) I0703 01:42:32.727768 28240 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 775bd6dae9ed48ebbbb9df088d833772 (attempt 3) I0703 01:42:32.728446 28240 catalog_manager.cc:3369] ChangeConfig:REMOVE_PEER RPC for tablet 775bd6dae9ed48ebbbb9df088d833772 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader e98d29f36f2347f29170e987cb83ced4 (127.5.240.197:41466) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term I0703 01:42:32.728504 28240 catalog_manager.cc:2903] Scheduling retry of ChangeConfig:REMOVE_PEER RPC for tablet 775bd6dae9ed48ebbbb9df088d833772 with cas_config_opid_index 5 with a delay of 64 ms (attempt = 3) I0703 01:42:32.743165 28377 raft_consensus.cc:1081] T b4acd291b108434da9b8791c61fd1dc2 P 5ee4739615124e308720b4aeddfe29cb [term 2 FOLLOWER]: Refusing update from remote peer 6505fcf874c440788e377dac9aa99209: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch) I0703 01:42:32.743733 30642 consensus_queue.cc:962] T b4acd291b108434da9b8791c61fd1dc2 P 6505fcf874c440788e377dac9aa99209 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I0703 01:42:32.748152 28253 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet b4acd291b108434da9b8791c61fd1dc2 (attempt 1) I0703 01:42:32.749002 29906 consensus_peers.cc:455] T b4acd291b108434da9b8791c61fd1dc2 P 6505fcf874c440788e377dac9aa99209 -> Peer 5ee4739615124e308720b4aeddfe29cb (127.5.240.193:54206): Closing peer: 5ee4739615124e308720b4aeddfe29cb I0703 01:42:32.749274 29906 consensus_queue.cc:204] T b4acd291b108434da9b8791c61fd1dc2 P 6505fcf874c440788e377dac9aa99209 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6505fcf874c440788e377dac9aa99209" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 33903 } attrs { promote: false } } I0703 01:42:32.749830 30642 raft_consensus.cc:2684] T b4acd291b108434da9b8791c61fd1dc2 P 6505fcf874c440788e377dac9aa99209 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 5ee4739615124e308720b4aeddfe29cb (127.5.240.193) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6505fcf874c440788e377dac9aa99209" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 33903 } attrs { promote: false } } } I0703 01:42:32.751749 28239 catalog_manager.cc:3352] ChangeConfig:REMOVE_PEER RPC for tablet b4acd291b108434da9b8791c61fd1dc2 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0703 01:42:32.752462 28253 catalog_manager.cc:3798] T b4acd291b108434da9b8791c61fd1dc2 P 6505fcf874c440788e377dac9aa99209 reported cstate change: config changed from index 5 to 7, VOTER 5ee4739615124e308720b4aeddfe29cb (127.5.240.193) evicted. New cstate: current_term: 2 leader_uuid: "6505fcf874c440788e377dac9aa99209" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6505fcf874c440788e377dac9aa99209" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 33903 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I0703 01:42:32.754222 28253 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet b4acd291b108434da9b8791c61fd1dc2 on 5ee4739615124e308720b4aeddfe29cb (127.5.240.193:54206) (TS 5ee4739615124e308720b4aeddfe29cb not found in new config with opid_index 7) I0703 01:42:32.754679 28356 tablet_service.cc:800] Processing DeleteTablet for tablet b4acd291b108434da9b8791c61fd1dc2 with delete_type TABLET_DATA_TOMBSTONED (TS 5ee4739615124e308720b4aeddfe29cb not found in new config with opid_index 7) from {username='jenkins'} at 127.5.240.193:54680 I0703 01:42:32.754897 30754 tablet_replica.cc:261] T b4acd291b108434da9b8791c61fd1dc2 P 5ee4739615124e308720b4aeddfe29cb: stopping tablet replica I0703 01:42:32.754981 30754 maintenance_manager.cc:237] P 5ee4739615124e308720b4aeddfe29cb: Unregistered op CompactRowSetsOp(b4acd291b108434da9b8791c61fd1dc2) I0703 01:42:32.755039 30754 maintenance_manager.cc:237] P 5ee4739615124e308720b4aeddfe29cb: Unregistered op MinorDeltaCompactionOp(b4acd291b108434da9b8791c61fd1dc2) I0703 01:42:32.755102 30754 maintenance_manager.cc:237] P 5ee4739615124e308720b4aeddfe29cb: Unregistered op MajorDeltaCompactionOp(b4acd291b108434da9b8791c61fd1dc2) I0703 01:42:32.755156 30754 maintenance_manager.cc:237] P 5ee4739615124e308720b4aeddfe29cb: Unregistered op UndoDeltaBlockGCOp(b4acd291b108434da9b8791c61fd1dc2) I0703 01:42:32.755216 30754 maintenance_manager.cc:237] P 5ee4739615124e308720b4aeddfe29cb: Unregistered op FlushMRSOp(b4acd291b108434da9b8791c61fd1dc2) I0703 01:42:32.755261 30754 maintenance_manager.cc:237] P 5ee4739615124e308720b4aeddfe29cb: Unregistered op FlushDeltaMemStoresOp(b4acd291b108434da9b8791c61fd1dc2) I0703 01:42:32.755309 30754 maintenance_manager.cc:237] P 5ee4739615124e308720b4aeddfe29cb: Unregistered op LogGCOp(b4acd291b108434da9b8791c61fd1dc2) I0703 01:42:32.755357 30754 raft_consensus.cc:2040] T b4acd291b108434da9b8791c61fd1dc2 P 5ee4739615124e308720b4aeddfe29cb [term 2 FOLLOWER]: Raft consensus shutting down. I0703 01:42:32.755445 30754 raft_consensus.cc:2067] T b4acd291b108434da9b8791c61fd1dc2 P 5ee4739615124e308720b4aeddfe29cb [term 2 FOLLOWER]: Raft consensus is shut down! I0703 01:42:32.756032 30754 ts_tablet_manager.cc:1369] T b4acd291b108434da9b8791c61fd1dc2 P 5ee4739615124e308720b4aeddfe29cb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:42:32.758571 30754 ts_tablet_manager.cc:1382] T b4acd291b108434da9b8791c61fd1dc2 P 5ee4739615124e308720b4aeddfe29cb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6 I0703 01:42:32.758652 30754 log.cc:986] T b4acd291b108434da9b8791c61fd1dc2 P 5ee4739615124e308720b4aeddfe29cb: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal/wals/b4acd291b108434da9b8791c61fd1dc2 I0703 01:42:32.759105 28238 catalog_manager.cc:3103] TS 5ee4739615124e308720b4aeddfe29cb (127.5.240.193:54206): tablet b4acd291b108434da9b8791c61fd1dc2 (table rebalance_test_table_0 [id=2fa00f90e0e84298b5fa673ff8ab7e37]) successfully deleted I0703 01:42:32.766484 28377 raft_consensus.cc:1081] T 4d69010513df4b68840b90be3161f402 P 5ee4739615124e308720b4aeddfe29cb [term 2 FOLLOWER]: Refusing update from remote peer 0ae780b86cc348d3a55bbaf5cf5a6217: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch) I0703 01:42:32.767040 30691 consensus_queue.cc:962] T 4d69010513df4b68840b90be3161f402 P 0ae780b86cc348d3a55bbaf5cf5a6217 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I0703 01:42:32.769071 28240 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 1cd5cbfe70ec4f8a957fa074ae08fcb1 (attempt 5) I0703 01:42:32.770311 28240 catalog_manager.cc:3369] ChangeConfig:REMOVE_PEER RPC for tablet 1cd5cbfe70ec4f8a957fa074ae08fcb1 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 35f0266742fc4707a8cc10c6437b84ed (127.5.240.196:50368) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term I0703 01:42:32.770395 28240 catalog_manager.cc:2903] Scheduling retry of ChangeConfig:REMOVE_PEER RPC for tablet 1cd5cbfe70ec4f8a957fa074ae08fcb1 with cas_config_opid_index 5 with a delay of 303 ms (attempt = 5) I0703 01:42:32.771306 28253 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 4d69010513df4b68840b90be3161f402 (attempt 1) I0703 01:42:32.772228 29417 consensus_peers.cc:455] T 4d69010513df4b68840b90be3161f402 P 0ae780b86cc348d3a55bbaf5cf5a6217 -> Peer 5ee4739615124e308720b4aeddfe29cb (127.5.240.193:54206): Closing peer: 5ee4739615124e308720b4aeddfe29cb I0703 01:42:32.772568 29417 consensus_queue.cc:204] T 4d69010513df4b68840b90be3161f402 P 0ae780b86cc348d3a55bbaf5cf5a6217 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0ae780b86cc348d3a55bbaf5cf5a6217" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46219 } attrs { promote: false } } I0703 01:42:32.773244 30691 raft_consensus.cc:2684] T 4d69010513df4b68840b90be3161f402 P 0ae780b86cc348d3a55bbaf5cf5a6217 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 5ee4739615124e308720b4aeddfe29cb (127.5.240.193) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0ae780b86cc348d3a55bbaf5cf5a6217" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46219 } attrs { promote: false } } } I0703 01:42:32.775218 28239 catalog_manager.cc:3352] ChangeConfig:REMOVE_PEER RPC for tablet 4d69010513df4b68840b90be3161f402 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0703 01:42:32.775902 28253 catalog_manager.cc:3798] T 4d69010513df4b68840b90be3161f402 P 0ae780b86cc348d3a55bbaf5cf5a6217 reported cstate change: config changed from index 5 to 7, VOTER 5ee4739615124e308720b4aeddfe29cb (127.5.240.193) evicted. New cstate: current_term: 2 leader_uuid: "0ae780b86cc348d3a55bbaf5cf5a6217" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0ae780b86cc348d3a55bbaf5cf5a6217" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46219 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I0703 01:42:32.776108 28240 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 74b62768046f4b8690aab7d2ec0f803a (attempt 5) I0703 01:42:32.777235 28240 catalog_manager.cc:3369] ChangeConfig:REMOVE_PEER RPC for tablet 74b62768046f4b8690aab7d2ec0f803a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 575620f4889c4704b7a7c00da34b6a53 (127.5.240.198:39152) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term I0703 01:42:32.777349 28240 catalog_manager.cc:2903] Scheduling retry of ChangeConfig:REMOVE_PEER RPC for tablet 74b62768046f4b8690aab7d2ec0f803a with cas_config_opid_index 5 with a delay of 264 ms (attempt = 5) I0703 01:42:32.777765 28253 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 4d69010513df4b68840b90be3161f402 on 5ee4739615124e308720b4aeddfe29cb (127.5.240.193:54206) (TS 5ee4739615124e308720b4aeddfe29cb not found in new config with opid_index 7) I0703 01:42:32.778175 28356 tablet_service.cc:800] Processing DeleteTablet for tablet 4d69010513df4b68840b90be3161f402 with delete_type TABLET_DATA_TOMBSTONED (TS 5ee4739615124e308720b4aeddfe29cb not found in new config with opid_index 7) from {username='jenkins'} at 127.5.240.193:54680 I0703 01:42:32.778400 30754 tablet_replica.cc:261] T 4d69010513df4b68840b90be3161f402 P 5ee4739615124e308720b4aeddfe29cb: stopping tablet replica I0703 01:42:32.778465 30754 maintenance_manager.cc:237] P 5ee4739615124e308720b4aeddfe29cb: Unregistered op CompactRowSetsOp(4d69010513df4b68840b90be3161f402) I0703 01:42:32.778547 30754 maintenance_manager.cc:237] P 5ee4739615124e308720b4aeddfe29cb: Unregistered op MinorDeltaCompactionOp(4d69010513df4b68840b90be3161f402) I0703 01:42:32.778601 30754 maintenance_manager.cc:237] P 5ee4739615124e308720b4aeddfe29cb: Unregistered op MajorDeltaCompactionOp(4d69010513df4b68840b90be3161f402) I0703 01:42:32.778676 30754 maintenance_manager.cc:237] P 5ee4739615124e308720b4aeddfe29cb: Unregistered op UndoDeltaBlockGCOp(4d69010513df4b68840b90be3161f402) I0703 01:42:32.778731 30754 maintenance_manager.cc:237] P 5ee4739615124e308720b4aeddfe29cb: Unregistered op FlushMRSOp(4d69010513df4b68840b90be3161f402) I0703 01:42:32.778789 30754 maintenance_manager.cc:237] P 5ee4739615124e308720b4aeddfe29cb: Unregistered op FlushDeltaMemStoresOp(4d69010513df4b68840b90be3161f402) I0703 01:42:32.778836 30754 maintenance_manager.cc:237] P 5ee4739615124e308720b4aeddfe29cb: Unregistered op LogGCOp(4d69010513df4b68840b90be3161f402) I0703 01:42:32.778895 30754 raft_consensus.cc:2040] T 4d69010513df4b68840b90be3161f402 P 5ee4739615124e308720b4aeddfe29cb [term 2 FOLLOWER]: Raft consensus shutting down. I0703 01:42:32.778983 30754 raft_consensus.cc:2067] T 4d69010513df4b68840b90be3161f402 P 5ee4739615124e308720b4aeddfe29cb [term 2 FOLLOWER]: Raft consensus is shut down! I0703 01:42:32.779570 30754 ts_tablet_manager.cc:1369] T 4d69010513df4b68840b90be3161f402 P 5ee4739615124e308720b4aeddfe29cb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:42:32.781977 30754 ts_tablet_manager.cc:1382] T 4d69010513df4b68840b90be3161f402 P 5ee4739615124e308720b4aeddfe29cb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6 I0703 01:42:32.782050 30754 log.cc:986] T 4d69010513df4b68840b90be3161f402 P 5ee4739615124e308720b4aeddfe29cb: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal/wals/4d69010513df4b68840b90be3161f402 I0703 01:42:32.782483 28238 catalog_manager.cc:3103] TS 5ee4739615124e308720b4aeddfe29cb (127.5.240.193:54206): tablet 4d69010513df4b68840b90be3161f402 (table rebalance_test_table_4 [id=991632a32296482d80382591992c76aa]) successfully deleted I0703 01:42:32.792970 28240 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 775bd6dae9ed48ebbbb9df088d833772 (attempt 4) I0703 01:42:32.793736 28240 catalog_manager.cc:3369] ChangeConfig:REMOVE_PEER RPC for tablet 775bd6dae9ed48ebbbb9df088d833772 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader e98d29f36f2347f29170e987cb83ced4 (127.5.240.197:41466) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term I0703 01:42:32.793802 28240 catalog_manager.cc:2903] Scheduling retry of ChangeConfig:REMOVE_PEER RPC for tablet 775bd6dae9ed48ebbbb9df088d833772 with cas_config_opid_index 5 with a delay of 157 ms (attempt = 4) I0703 01:42:32.799067 28240 catalog_manager.cc:3385] ChangeConfig:REMOVE_PEER RPC for tablet 45b3833ac0664a8d9889ab14b3d0dd69 with cas_config_opid_index 5: aborting the task: latest config opid_index 7; task opid_index 5 I0703 01:42:32.817608 28240 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 107baa5504214d5f8362cef92ed80291 (attempt 5) I0703 01:42:32.818802 28240 catalog_manager.cc:3369] ChangeConfig:REMOVE_PEER RPC for tablet 107baa5504214d5f8362cef92ed80291 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 35f0266742fc4707a8cc10c6437b84ed (127.5.240.196:50368) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term I0703 01:42:32.818899 28240 catalog_manager.cc:2903] Scheduling retry of ChangeConfig:REMOVE_PEER RPC for tablet 107baa5504214d5f8362cef92ed80291 with cas_config_opid_index 5 with a delay of 305 ms (attempt = 5) I0703 01:42:32.855221 28377 raft_consensus.cc:1081] T 74b62768046f4b8690aab7d2ec0f803a P 5ee4739615124e308720b4aeddfe29cb [term 2 FOLLOWER]: Refusing update from remote peer 575620f4889c4704b7a7c00da34b6a53: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch) I0703 01:42:32.855901 30695 consensus_queue.cc:962] T 74b62768046f4b8690aab7d2ec0f803a P 575620f4889c4704b7a7c00da34b6a53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I0703 01:42:32.860285 28253 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 74b62768046f4b8690aab7d2ec0f803a (attempt 1) I0703 01:42:32.861564 29783 consensus_peers.cc:455] T 74b62768046f4b8690aab7d2ec0f803a P 575620f4889c4704b7a7c00da34b6a53 -> Peer 5ee4739615124e308720b4aeddfe29cb (127.5.240.193:54206): Closing peer: 5ee4739615124e308720b4aeddfe29cb I0703 01:42:32.861843 29783 consensus_queue.cc:204] T 74b62768046f4b8690aab7d2ec0f803a P 575620f4889c4704b7a7c00da34b6a53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "575620f4889c4704b7a7c00da34b6a53" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39152 } attrs { promote: false } } I0703 01:42:32.862004 28239 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 41c52d5ecddf4e5897acf3fe5ee44f34 (attempt 5) I0703 01:42:32.862424 30695 raft_consensus.cc:2684] T 74b62768046f4b8690aab7d2ec0f803a P 575620f4889c4704b7a7c00da34b6a53 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 5ee4739615124e308720b4aeddfe29cb (127.5.240.193) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "575620f4889c4704b7a7c00da34b6a53" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39152 } attrs { promote: false } } } I0703 01:42:32.862715 28500 raft_consensus.cc:1081] T ef65b31bc12e40c2b3f1e4714cc4218e P 31c252b54241470ba1115181c3672664 [term 2 FOLLOWER]: Refusing update from remote peer 575620f4889c4704b7a7c00da34b6a53: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch) I0703 01:42:32.862802 28239 catalog_manager.cc:3369] ChangeConfig:REMOVE_PEER RPC for tablet 41c52d5ecddf4e5897acf3fe5ee44f34 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 6505fcf874c440788e377dac9aa99209 (127.5.240.199:33903) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term I0703 01:42:32.862884 28239 catalog_manager.cc:2903] Scheduling retry of ChangeConfig:REMOVE_PEER RPC for tablet 41c52d5ecddf4e5897acf3fe5ee44f34 with cas_config_opid_index 5 with a delay of 269 ms (attempt = 5) I0703 01:42:32.863657 30696 consensus_queue.cc:962] T ef65b31bc12e40c2b3f1e4714cc4218e P 575620f4889c4704b7a7c00da34b6a53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I0703 01:42:32.865443 28240 catalog_manager.cc:3352] ChangeConfig:REMOVE_PEER RPC for tablet 74b62768046f4b8690aab7d2ec0f803a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0703 01:42:32.865741 28239 catalog_manager.cc:3385] ChangeConfig:REMOVE_PEER RPC for tablet b4acd291b108434da9b8791c61fd1dc2 with cas_config_opid_index 5: aborting the task: latest config opid_index 7; task opid_index 5 I0703 01:42:32.866303 28251 catalog_manager.cc:3798] T 74b62768046f4b8690aab7d2ec0f803a P 575620f4889c4704b7a7c00da34b6a53 reported cstate change: config changed from index 5 to 7, VOTER 5ee4739615124e308720b4aeddfe29cb (127.5.240.193) evicted. New cstate: current_term: 2 leader_uuid: "575620f4889c4704b7a7c00da34b6a53" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "575620f4889c4704b7a7c00da34b6a53" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39152 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I0703 01:42:32.866457 28377 raft_consensus.cc:1081] T 107baa5504214d5f8362cef92ed80291 P 5ee4739615124e308720b4aeddfe29cb [term 2 FOLLOWER]: Refusing update from remote peer 35f0266742fc4707a8cc10c6437b84ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch) I0703 01:42:32.867146 30700 consensus_queue.cc:962] T 107baa5504214d5f8362cef92ed80291 P 35f0266742fc4707a8cc10c6437b84ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I0703 01:42:32.869247 28251 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 74b62768046f4b8690aab7d2ec0f803a on 5ee4739615124e308720b4aeddfe29cb (127.5.240.193:54206) (TS 5ee4739615124e308720b4aeddfe29cb not found in new config with opid_index 7) I0703 01:42:32.869778 28356 tablet_service.cc:800] Processing DeleteTablet for tablet 74b62768046f4b8690aab7d2ec0f803a with delete_type TABLET_DATA_TOMBSTONED (TS 5ee4739615124e308720b4aeddfe29cb not found in new config with opid_index 7) from {username='jenkins'} at 127.5.240.193:54680 I0703 01:42:32.870029 30754 tablet_replica.cc:261] T 74b62768046f4b8690aab7d2ec0f803a P 5ee4739615124e308720b4aeddfe29cb: stopping tablet replica I0703 01:42:32.870115 30754 maintenance_manager.cc:237] P 5ee4739615124e308720b4aeddfe29cb: Unregistered op CompactRowSetsOp(74b62768046f4b8690aab7d2ec0f803a) I0703 01:42:32.870162 30754 maintenance_manager.cc:237] P 5ee4739615124e308720b4aeddfe29cb: Unregistered op MinorDeltaCompactionOp(74b62768046f4b8690aab7d2ec0f803a) I0703 01:42:32.870219 30754 maintenance_manager.cc:237] P 5ee4739615124e308720b4aeddfe29cb: Unregistered op MajorDeltaCompactionOp(74b62768046f4b8690aab7d2ec0f803a) I0703 01:42:32.870292 30754 maintenance_manager.cc:237] P 5ee4739615124e308720b4aeddfe29cb: Unregistered op UndoDeltaBlockGCOp(74b62768046f4b8690aab7d2ec0f803a) I0703 01:42:32.870395 30754 maintenance_manager.cc:237] P 5ee4739615124e308720b4aeddfe29cb: Unregistered op FlushMRSOp(74b62768046f4b8690aab7d2ec0f803a) I0703 01:42:32.870491 30754 maintenance_manager.cc:237] P 5ee4739615124e308720b4aeddfe29cb: Unregistered op FlushDeltaMemStoresOp(74b62768046f4b8690aab7d2ec0f803a) I0703 01:42:32.870570 30754 maintenance_manager.cc:237] P 5ee4739615124e308720b4aeddfe29cb: Unregistered op LogGCOp(74b62768046f4b8690aab7d2ec0f803a) I0703 01:42:32.870652 30754 raft_consensus.cc:2040] T 74b62768046f4b8690aab7d2ec0f803a P 5ee4739615124e308720b4aeddfe29cb [term 2 FOLLOWER]: Raft consensus shutting down. I0703 01:42:32.870795 30754 raft_consensus.cc:2067] T 74b62768046f4b8690aab7d2ec0f803a P 5ee4739615124e308720b4aeddfe29cb [term 2 FOLLOWER]: Raft consensus is shut down! I0703 01:42:32.871743 28251 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet ef65b31bc12e40c2b3f1e4714cc4218e (attempt 1) I0703 01:42:32.872422 30754 ts_tablet_manager.cc:1369] T 74b62768046f4b8690aab7d2ec0f803a P 5ee4739615124e308720b4aeddfe29cb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:42:32.873955 29783 consensus_peers.cc:455] T ef65b31bc12e40c2b3f1e4714cc4218e P 575620f4889c4704b7a7c00da34b6a53 -> Peer 31c252b54241470ba1115181c3672664 (127.5.240.194:38625): Closing peer: 31c252b54241470ba1115181c3672664 I0703 01:42:32.874127 29783 consensus_queue.cc:204] T ef65b31bc12e40c2b3f1e4714cc4218e P 575620f4889c4704b7a7c00da34b6a53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "575620f4889c4704b7a7c00da34b6a53" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39152 } attrs { promote: false } } I0703 01:42:32.874661 30805 raft_consensus.cc:2684] T ef65b31bc12e40c2b3f1e4714cc4218e P 575620f4889c4704b7a7c00da34b6a53 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 31c252b54241470ba1115181c3672664 (127.5.240.194) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "575620f4889c4704b7a7c00da34b6a53" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39152 } attrs { promote: false } } } I0703 01:42:32.877255 28240 catalog_manager.cc:3352] ChangeConfig:REMOVE_PEER RPC for tablet ef65b31bc12e40c2b3f1e4714cc4218e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0703 01:42:32.877689 30754 ts_tablet_manager.cc:1382] T 74b62768046f4b8690aab7d2ec0f803a P 5ee4739615124e308720b4aeddfe29cb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6 I0703 01:42:32.877776 30754 log.cc:986] T 74b62768046f4b8690aab7d2ec0f803a P 5ee4739615124e308720b4aeddfe29cb: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal/wals/74b62768046f4b8690aab7d2ec0f803a I0703 01:42:32.878326 28251 catalog_manager.cc:3798] T ef65b31bc12e40c2b3f1e4714cc4218e P 575620f4889c4704b7a7c00da34b6a53 reported cstate change: config changed from index 5 to 7, VOTER 31c252b54241470ba1115181c3672664 (127.5.240.194) evicted. New cstate: current_term: 2 leader_uuid: "575620f4889c4704b7a7c00da34b6a53" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "575620f4889c4704b7a7c00da34b6a53" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 39152 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I0703 01:42:32.878491 28238 catalog_manager.cc:3103] TS 5ee4739615124e308720b4aeddfe29cb (127.5.240.193:54206): tablet 74b62768046f4b8690aab7d2ec0f803a (table rebalance_test_table_4 [id=991632a32296482d80382591992c76aa]) successfully deleted I0703 01:42:32.880704 28251 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet ef65b31bc12e40c2b3f1e4714cc4218e on 31c252b54241470ba1115181c3672664 (127.5.240.194:38625) (TS 31c252b54241470ba1115181c3672664 not found in new config with opid_index 7) I0703 01:42:32.881420 28479 tablet_service.cc:800] Processing DeleteTablet for tablet ef65b31bc12e40c2b3f1e4714cc4218e with delete_type TABLET_DATA_TOMBSTONED (TS 31c252b54241470ba1115181c3672664 not found in new config with opid_index 7) from {username='jenkins'} at 127.5.240.194:40764 I0703 01:42:32.881808 28253 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 107baa5504214d5f8362cef92ed80291 (attempt 1) I0703 01:42:32.882758 30807 tablet_replica.cc:261] T ef65b31bc12e40c2b3f1e4714cc4218e P 31c252b54241470ba1115181c3672664: stopping tablet replica I0703 01:42:32.882901 29540 consensus_peers.cc:455] T 107baa5504214d5f8362cef92ed80291 P 35f0266742fc4707a8cc10c6437b84ed -> Peer 5ee4739615124e308720b4aeddfe29cb (127.5.240.193:54206): Closing peer: 5ee4739615124e308720b4aeddfe29cb I0703 01:42:32.882948 30807 maintenance_manager.cc:237] P 31c252b54241470ba1115181c3672664: Unregistered op CompactRowSetsOp(ef65b31bc12e40c2b3f1e4714cc4218e) I0703 01:42:32.883045 30807 maintenance_manager.cc:237] P 31c252b54241470ba1115181c3672664: Unregistered op MinorDeltaCompactionOp(ef65b31bc12e40c2b3f1e4714cc4218e) I0703 01:42:32.883107 30807 maintenance_manager.cc:237] P 31c252b54241470ba1115181c3672664: Unregistered op MajorDeltaCompactionOp(ef65b31bc12e40c2b3f1e4714cc4218e) I0703 01:42:32.883173 30807 maintenance_manager.cc:237] P 31c252b54241470ba1115181c3672664: Unregistered op UndoDeltaBlockGCOp(ef65b31bc12e40c2b3f1e4714cc4218e) I0703 01:42:32.883143 29540 consensus_queue.cc:204] T 107baa5504214d5f8362cef92ed80291 P 35f0266742fc4707a8cc10c6437b84ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "35f0266742fc4707a8cc10c6437b84ed" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 50368 } attrs { promote: false } } I0703 01:42:32.883258 30807 maintenance_manager.cc:237] P 31c252b54241470ba1115181c3672664: Unregistered op FlushMRSOp(ef65b31bc12e40c2b3f1e4714cc4218e) I0703 01:42:32.883304 30807 maintenance_manager.cc:237] P 31c252b54241470ba1115181c3672664: Unregistered op FlushDeltaMemStoresOp(ef65b31bc12e40c2b3f1e4714cc4218e) I0703 01:42:32.883360 30807 maintenance_manager.cc:237] P 31c252b54241470ba1115181c3672664: Unregistered op LogGCOp(ef65b31bc12e40c2b3f1e4714cc4218e) I0703 01:42:32.883430 30807 raft_consensus.cc:2040] T ef65b31bc12e40c2b3f1e4714cc4218e P 31c252b54241470ba1115181c3672664 [term 2 FOLLOWER]: Raft consensus shutting down. I0703 01:42:32.883594 30807 raft_consensus.cc:2067] T ef65b31bc12e40c2b3f1e4714cc4218e P 31c252b54241470ba1115181c3672664 [term 2 FOLLOWER]: Raft consensus is shut down! I0703 01:42:32.883713 30700 raft_consensus.cc:2684] T 107baa5504214d5f8362cef92ed80291 P 35f0266742fc4707a8cc10c6437b84ed [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 5ee4739615124e308720b4aeddfe29cb (127.5.240.193) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "35f0266742fc4707a8cc10c6437b84ed" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 50368 } attrs { promote: false } } } I0703 01:42:32.885064 30807 ts_tablet_manager.cc:1369] T ef65b31bc12e40c2b3f1e4714cc4218e P 31c252b54241470ba1115181c3672664: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:42:32.886337 28240 catalog_manager.cc:3352] ChangeConfig:REMOVE_PEER RPC for tablet 107baa5504214d5f8362cef92ed80291 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0703 01:42:32.887282 28251 catalog_manager.cc:3798] T 107baa5504214d5f8362cef92ed80291 P 35f0266742fc4707a8cc10c6437b84ed reported cstate change: config changed from index 5 to 7, VOTER 5ee4739615124e308720b4aeddfe29cb (127.5.240.193) evicted. New cstate: current_term: 2 leader_uuid: "35f0266742fc4707a8cc10c6437b84ed" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "35f0266742fc4707a8cc10c6437b84ed" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 50368 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I0703 01:42:32.888801 30807 ts_tablet_manager.cc:1382] T ef65b31bc12e40c2b3f1e4714cc4218e P 31c252b54241470ba1115181c3672664: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6 I0703 01:42:32.888890 30807 log.cc:986] T ef65b31bc12e40c2b3f1e4714cc4218e P 31c252b54241470ba1115181c3672664: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal/wals/ef65b31bc12e40c2b3f1e4714cc4218e I0703 01:42:32.889077 28251 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 107baa5504214d5f8362cef92ed80291 on 5ee4739615124e308720b4aeddfe29cb (127.5.240.193:54206) (TS 5ee4739615124e308720b4aeddfe29cb not found in new config with opid_index 7) I0703 01:42:32.889509 28238 catalog_manager.cc:3103] TS 31c252b54241470ba1115181c3672664 (127.5.240.194:38625): tablet ef65b31bc12e40c2b3f1e4714cc4218e (table rebalance_test_table_3 [id=75241f23aa654e04a8d5a26fb0d09c07]) successfully deleted I0703 01:42:32.889600 28356 tablet_service.cc:800] Processing DeleteTablet for tablet 107baa5504214d5f8362cef92ed80291 with delete_type TABLET_DATA_TOMBSTONED (TS 5ee4739615124e308720b4aeddfe29cb not found in new config with opid_index 7) from {username='jenkins'} at 127.5.240.193:54680 I0703 01:42:32.889812 30754 tablet_replica.cc:261] T 107baa5504214d5f8362cef92ed80291 P 5ee4739615124e308720b4aeddfe29cb: stopping tablet replica I0703 01:42:32.889871 30754 maintenance_manager.cc:237] P 5ee4739615124e308720b4aeddfe29cb: Unregistered op CompactRowSetsOp(107baa5504214d5f8362cef92ed80291) I0703 01:42:32.889917 30754 maintenance_manager.cc:237] P 5ee4739615124e308720b4aeddfe29cb: Unregistered op MinorDeltaCompactionOp(107baa5504214d5f8362cef92ed80291) I0703 01:42:32.889958 30754 maintenance_manager.cc:237] P 5ee4739615124e308720b4aeddfe29cb: Unregistered op MajorDeltaCompactionOp(107baa5504214d5f8362cef92ed80291) I0703 01:42:32.889997 30754 maintenance_manager.cc:237] P 5ee4739615124e308720b4aeddfe29cb: Unregistered op UndoDeltaBlockGCOp(107baa5504214d5f8362cef92ed80291) I0703 01:42:32.890034 30754 maintenance_manager.cc:237] P 5ee4739615124e308720b4aeddfe29cb: Unregistered op FlushMRSOp(107baa5504214d5f8362cef92ed80291) I0703 01:42:32.890069 30754 maintenance_manager.cc:237] P 5ee4739615124e308720b4aeddfe29cb: Unregistered op FlushDeltaMemStoresOp(107baa5504214d5f8362cef92ed80291) I0703 01:42:32.890097 30754 maintenance_manager.cc:237] P 5ee4739615124e308720b4aeddfe29cb: Unregistered op LogGCOp(107baa5504214d5f8362cef92ed80291) I0703 01:42:32.890133 30754 raft_consensus.cc:2040] T 107baa5504214d5f8362cef92ed80291 P 5ee4739615124e308720b4aeddfe29cb [term 2 FOLLOWER]: Raft consensus shutting down. I0703 01:42:32.890202 30754 raft_consensus.cc:2067] T 107baa5504214d5f8362cef92ed80291 P 5ee4739615124e308720b4aeddfe29cb [term 2 FOLLOWER]: Raft consensus is shut down! I0703 01:42:32.890803 30754 ts_tablet_manager.cc:1369] T 107baa5504214d5f8362cef92ed80291 P 5ee4739615124e308720b4aeddfe29cb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:42:32.893368 30754 ts_tablet_manager.cc:1382] T 107baa5504214d5f8362cef92ed80291 P 5ee4739615124e308720b4aeddfe29cb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6 I0703 01:42:32.893429 30754 log.cc:986] T 107baa5504214d5f8362cef92ed80291 P 5ee4739615124e308720b4aeddfe29cb: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal/wals/107baa5504214d5f8362cef92ed80291 I0703 01:42:32.893937 28238 catalog_manager.cc:3103] TS 5ee4739615124e308720b4aeddfe29cb (127.5.240.193:54206): tablet 107baa5504214d5f8362cef92ed80291 (table rebalance_test_table_2 [id=b5ac6ea59801477a8e4cca2b80c8beec]) successfully deleted I0703 01:42:32.894815 28240 catalog_manager.cc:3385] ChangeConfig:REMOVE_PEER RPC for tablet ef65b31bc12e40c2b3f1e4714cc4218e with cas_config_opid_index 5: aborting the task: latest config opid_index 7; task opid_index 5 I0703 01:42:32.901049 28500 raft_consensus.cc:1081] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 31c252b54241470ba1115181c3672664 [term 2 FOLLOWER]: Refusing update from remote peer 35f0266742fc4707a8cc10c6437b84ed: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch) I0703 01:42:32.901605 30700 consensus_queue.cc:962] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 35f0266742fc4707a8cc10c6437b84ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I0703 01:42:32.905328 28251 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 1cd5cbfe70ec4f8a957fa074ae08fcb1 (attempt 1) I0703 01:42:32.906021 29540 consensus_peers.cc:455] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 35f0266742fc4707a8cc10c6437b84ed -> Peer 31c252b54241470ba1115181c3672664 (127.5.240.194:38625): Closing peer: 31c252b54241470ba1115181c3672664 I0703 01:42:32.906251 29540 consensus_queue.cc:204] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 35f0266742fc4707a8cc10c6437b84ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "35f0266742fc4707a8cc10c6437b84ed" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 50368 } attrs { promote: false } } I0703 01:42:32.906769 30701 raft_consensus.cc:2684] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 35f0266742fc4707a8cc10c6437b84ed [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 31c252b54241470ba1115181c3672664 (127.5.240.194) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "35f0266742fc4707a8cc10c6437b84ed" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 50368 } attrs { promote: false } } } I0703 01:42:32.908531 28240 catalog_manager.cc:3352] ChangeConfig:REMOVE_PEER RPC for tablet 1cd5cbfe70ec4f8a957fa074ae08fcb1 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0703 01:42:32.909173 28253 catalog_manager.cc:3798] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 35f0266742fc4707a8cc10c6437b84ed reported cstate change: config changed from index 5 to 7, VOTER 31c252b54241470ba1115181c3672664 (127.5.240.194) evicted. New cstate: current_term: 2 leader_uuid: "35f0266742fc4707a8cc10c6437b84ed" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "35f0266742fc4707a8cc10c6437b84ed" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 50368 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I0703 01:42:32.910904 28253 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 1cd5cbfe70ec4f8a957fa074ae08fcb1 on 31c252b54241470ba1115181c3672664 (127.5.240.194:38625) (TS 31c252b54241470ba1115181c3672664 not found in new config with opid_index 7) I0703 01:42:32.911381 28479 tablet_service.cc:800] Processing DeleteTablet for tablet 1cd5cbfe70ec4f8a957fa074ae08fcb1 with delete_type TABLET_DATA_TOMBSTONED (TS 31c252b54241470ba1115181c3672664 not found in new config with opid_index 7) from {username='jenkins'} at 127.5.240.194:40764 I0703 01:42:32.911607 30807 tablet_replica.cc:261] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 31c252b54241470ba1115181c3672664: stopping tablet replica I0703 01:42:32.911679 30807 maintenance_manager.cc:237] P 31c252b54241470ba1115181c3672664: Unregistered op CompactRowSetsOp(1cd5cbfe70ec4f8a957fa074ae08fcb1) I0703 01:42:32.911721 30807 maintenance_manager.cc:237] P 31c252b54241470ba1115181c3672664: Unregistered op MinorDeltaCompactionOp(1cd5cbfe70ec4f8a957fa074ae08fcb1) I0703 01:42:32.911762 30807 maintenance_manager.cc:237] P 31c252b54241470ba1115181c3672664: Unregistered op MajorDeltaCompactionOp(1cd5cbfe70ec4f8a957fa074ae08fcb1) I0703 01:42:32.911793 30807 maintenance_manager.cc:237] P 31c252b54241470ba1115181c3672664: Unregistered op UndoDeltaBlockGCOp(1cd5cbfe70ec4f8a957fa074ae08fcb1) I0703 01:42:32.911828 30807 maintenance_manager.cc:237] P 31c252b54241470ba1115181c3672664: Unregistered op FlushMRSOp(1cd5cbfe70ec4f8a957fa074ae08fcb1) I0703 01:42:32.911860 30807 maintenance_manager.cc:237] P 31c252b54241470ba1115181c3672664: Unregistered op FlushDeltaMemStoresOp(1cd5cbfe70ec4f8a957fa074ae08fcb1) I0703 01:42:32.911888 30807 maintenance_manager.cc:237] P 31c252b54241470ba1115181c3672664: Unregistered op LogGCOp(1cd5cbfe70ec4f8a957fa074ae08fcb1) I0703 01:42:32.911919 30807 raft_consensus.cc:2040] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 31c252b54241470ba1115181c3672664 [term 2 FOLLOWER]: Raft consensus shutting down. I0703 01:42:32.911999 30807 raft_consensus.cc:2067] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 31c252b54241470ba1115181c3672664 [term 2 FOLLOWER]: Raft consensus is shut down! I0703 01:42:32.912664 30807 ts_tablet_manager.cc:1369] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 31c252b54241470ba1115181c3672664: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:42:32.915201 30807 ts_tablet_manager.cc:1382] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 31c252b54241470ba1115181c3672664: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6 I0703 01:42:32.915258 30807 log.cc:986] T 1cd5cbfe70ec4f8a957fa074ae08fcb1 P 31c252b54241470ba1115181c3672664: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal/wals/1cd5cbfe70ec4f8a957fa074ae08fcb1 I0703 01:42:32.915702 28238 catalog_manager.cc:3103] TS 31c252b54241470ba1115181c3672664 (127.5.240.194:38625): tablet 1cd5cbfe70ec4f8a957fa074ae08fcb1 (table rebalance_test_table_1 [id=14bf1112832a4171b3d41a08ed95d54e]) successfully deleted I0703 01:42:32.922611 28239 catalog_manager.cc:3385] ChangeConfig:REMOVE_PEER RPC for tablet 4d69010513df4b68840b90be3161f402 with cas_config_opid_index 5: aborting the task: latest config opid_index 7; task opid_index 5 I0703 01:42:32.951918 28240 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 775bd6dae9ed48ebbbb9df088d833772 (attempt 5) I0703 01:42:32.952709 28240 catalog_manager.cc:3369] ChangeConfig:REMOVE_PEER RPC for tablet 775bd6dae9ed48ebbbb9df088d833772 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader e98d29f36f2347f29170e987cb83ced4 (127.5.240.197:41466) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term I0703 01:42:32.952776 28240 catalog_manager.cc:2903] Scheduling retry of ChangeConfig:REMOVE_PEER RPC for tablet 775bd6dae9ed48ebbbb9df088d833772 with cas_config_opid_index 5 with a delay of 285 ms (attempt = 5) I0703 01:42:33.042328 28240 catalog_manager.cc:3385] ChangeConfig:REMOVE_PEER RPC for tablet 74b62768046f4b8690aab7d2ec0f803a with cas_config_opid_index 5: aborting the task: latest config opid_index 7; task opid_index 5 I0703 01:42:33.052912 28500 raft_consensus.cc:1081] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 31c252b54241470ba1115181c3672664 [term 2 FOLLOWER]: Refusing update from remote peer 6505fcf874c440788e377dac9aa99209: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch) I0703 01:42:33.053488 30642 consensus_queue.cc:962] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 6505fcf874c440788e377dac9aa99209 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I0703 01:42:33.057796 28253 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 41c52d5ecddf4e5897acf3fe5ee44f34 (attempt 1) I0703 01:42:33.058568 29906 consensus_peers.cc:455] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 6505fcf874c440788e377dac9aa99209 -> Peer 31c252b54241470ba1115181c3672664 (127.5.240.194:38625): Closing peer: 31c252b54241470ba1115181c3672664 I0703 01:42:33.058799 29906 consensus_queue.cc:204] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 6505fcf874c440788e377dac9aa99209 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6505fcf874c440788e377dac9aa99209" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 33903 } attrs { promote: false } } I0703 01:42:33.059387 30642 raft_consensus.cc:2684] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 6505fcf874c440788e377dac9aa99209 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 31c252b54241470ba1115181c3672664 (127.5.240.194) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6505fcf874c440788e377dac9aa99209" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 33903 } attrs { promote: false } } } I0703 01:42:33.061971 28239 catalog_manager.cc:3352] ChangeConfig:REMOVE_PEER RPC for tablet 41c52d5ecddf4e5897acf3fe5ee44f34 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0703 01:42:33.062841 28253 catalog_manager.cc:3798] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 6505fcf874c440788e377dac9aa99209 reported cstate change: config changed from index 5 to 7, VOTER 31c252b54241470ba1115181c3672664 (127.5.240.194) evicted. New cstate: current_term: 2 leader_uuid: "6505fcf874c440788e377dac9aa99209" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6505fcf874c440788e377dac9aa99209" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 33903 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I0703 01:42:33.064978 28253 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 41c52d5ecddf4e5897acf3fe5ee44f34 on 31c252b54241470ba1115181c3672664 (127.5.240.194:38625) (TS 31c252b54241470ba1115181c3672664 not found in new config with opid_index 7) I0703 01:42:33.065675 28479 tablet_service.cc:800] Processing DeleteTablet for tablet 41c52d5ecddf4e5897acf3fe5ee44f34 with delete_type TABLET_DATA_TOMBSTONED (TS 31c252b54241470ba1115181c3672664 not found in new config with opid_index 7) from {username='jenkins'} at 127.5.240.194:40764 I0703 01:42:33.065948 30807 tablet_replica.cc:261] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 31c252b54241470ba1115181c3672664: stopping tablet replica I0703 01:42:33.066040 30807 maintenance_manager.cc:237] P 31c252b54241470ba1115181c3672664: Unregistered op CompactRowSetsOp(41c52d5ecddf4e5897acf3fe5ee44f34) I0703 01:42:33.066125 30807 maintenance_manager.cc:237] P 31c252b54241470ba1115181c3672664: Unregistered op MinorDeltaCompactionOp(41c52d5ecddf4e5897acf3fe5ee44f34) I0703 01:42:33.066175 30807 maintenance_manager.cc:237] P 31c252b54241470ba1115181c3672664: Unregistered op MajorDeltaCompactionOp(41c52d5ecddf4e5897acf3fe5ee44f34) I0703 01:42:33.066212 30807 maintenance_manager.cc:237] P 31c252b54241470ba1115181c3672664: Unregistered op UndoDeltaBlockGCOp(41c52d5ecddf4e5897acf3fe5ee44f34) I0703 01:42:33.066282 30807 maintenance_manager.cc:237] P 31c252b54241470ba1115181c3672664: Unregistered op FlushMRSOp(41c52d5ecddf4e5897acf3fe5ee44f34) I0703 01:42:33.066337 30807 maintenance_manager.cc:237] P 31c252b54241470ba1115181c3672664: Unregistered op FlushDeltaMemStoresOp(41c52d5ecddf4e5897acf3fe5ee44f34) I0703 01:42:33.066370 30807 maintenance_manager.cc:237] P 31c252b54241470ba1115181c3672664: Unregistered op LogGCOp(41c52d5ecddf4e5897acf3fe5ee44f34) I0703 01:42:33.066406 30807 raft_consensus.cc:2040] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 31c252b54241470ba1115181c3672664 [term 2 FOLLOWER]: Raft consensus shutting down. I0703 01:42:33.066490 30807 raft_consensus.cc:2067] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 31c252b54241470ba1115181c3672664 [term 2 FOLLOWER]: Raft consensus is shut down! I0703 01:42:33.067194 30807 ts_tablet_manager.cc:1369] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 31c252b54241470ba1115181c3672664: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:42:33.068850 28500 raft_consensus.cc:1081] T 775bd6dae9ed48ebbbb9df088d833772 P 31c252b54241470ba1115181c3672664 [term 2 FOLLOWER]: Refusing update from remote peer e98d29f36f2347f29170e987cb83ced4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch) I0703 01:42:33.069504 30634 consensus_queue.cc:962] T 775bd6dae9ed48ebbbb9df088d833772 P e98d29f36f2347f29170e987cb83ced4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31c252b54241470ba1115181c3672664" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 38625 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I0703 01:42:33.070988 28377 consensus_queue.cc:204] T 12020f47e415490ebd506a2685335058 P 5ee4739615124e308720b4aeddfe29cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "0ae780b86cc348d3a55bbaf5cf5a6217" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 46219 } attrs { promote: true } } I0703 01:42:33.072757 30752 raft_consensus.cc:2684] T 12020f47e415490ebd506a2685335058 P 5ee4739615124e308720b4aeddfe29cb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 0ae780b86cc348d3a55bbaf5cf5a6217 (127.5.240.195) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "0ae780b86cc348d3a55bbaf5cf5a6217" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 46219 } attrs { promote: true } } } I0703 01:42:33.074795 28240 catalog_manager.cc:3385] ChangeConfig:REMOVE_PEER RPC for tablet 1cd5cbfe70ec4f8a957fa074ae08fcb1 with cas_config_opid_index 5: aborting the task: latest config opid_index 7; task opid_index 5 I0703 01:42:33.074949 30807 ts_tablet_manager.cc:1382] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 31c252b54241470ba1115181c3672664: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6 I0703 01:42:33.075049 30807 log.cc:986] T 41c52d5ecddf4e5897acf3fe5ee44f34 P 31c252b54241470ba1115181c3672664: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal/wals/41c52d5ecddf4e5897acf3fe5ee44f34 I0703 01:42:33.075794 28238 catalog_manager.cc:3103] TS 31c252b54241470ba1115181c3672664 (127.5.240.194:38625): tablet 41c52d5ecddf4e5897acf3fe5ee44f34 (table rebalance_test_table_1 [id=14bf1112832a4171b3d41a08ed95d54e]) successfully deleted I0703 01:42:33.076272 28253 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 775bd6dae9ed48ebbbb9df088d833772 (attempt 1) I0703 01:42:33.077101 29662 consensus_peers.cc:455] T 775bd6dae9ed48ebbbb9df088d833772 P e98d29f36f2347f29170e987cb83ced4 -> Peer 31c252b54241470ba1115181c3672664 (127.5.240.194:38625): Closing peer: 31c252b54241470ba1115181c3672664 I0703 01:42:33.077311 29662 consensus_queue.cc:204] T 775bd6dae9ed48ebbbb9df088d833772 P e98d29f36f2347f29170e987cb83ced4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e98d29f36f2347f29170e987cb83ced4" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 41466 } attrs { promote: false } } I0703 01:42:33.077958 30634 raft_consensus.cc:2684] T 775bd6dae9ed48ebbbb9df088d833772 P e98d29f36f2347f29170e987cb83ced4 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 31c252b54241470ba1115181c3672664 (127.5.240.194) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e98d29f36f2347f29170e987cb83ced4" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 41466 } attrs { promote: false } } } W0703 01:42:33.078539 28309 consensus_peers.cc:433] T 12020f47e415490ebd506a2685335058 P 5ee4739615124e308720b4aeddfe29cb -> Peer 0ae780b86cc348d3a55bbaf5cf5a6217 (127.5.240.195:46219): Couldn't send request to peer 0ae780b86cc348d3a55bbaf5cf5a6217 for tablet 12020f47e415490ebd506a2685335058. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 12020f47e415490ebd506a2685335058. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:42:33.078652 28253 catalog_manager.cc:3798] T 12020f47e415490ebd506a2685335058 P 5ee4739615124e308720b4aeddfe29cb reported cstate change: config changed from index -1 to 4, NON_VOTER 0ae780b86cc348d3a55bbaf5cf5a6217 (127.5.240.195) added. New cstate: current_term: 1 leader_uuid: "5ee4739615124e308720b4aeddfe29cb" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ae780b86cc348d3a55bbaf5cf5a6217" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 46219 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I0703 01:42:33.080462 28240 catalog_manager.cc:3352] ChangeConfig:REMOVE_PEER RPC for tablet 775bd6dae9ed48ebbbb9df088d833772 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0703 01:42:33.081523 28253 catalog_manager.cc:3798] T 775bd6dae9ed48ebbbb9df088d833772 P e98d29f36f2347f29170e987cb83ced4 reported cstate change: config changed from index 5 to 7, VOTER 31c252b54241470ba1115181c3672664 (127.5.240.194) evicted. New cstate: current_term: 2 leader_uuid: "e98d29f36f2347f29170e987cb83ced4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e98d29f36f2347f29170e987cb83ced4" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 41466 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I0703 01:42:33.083371 28253 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 775bd6dae9ed48ebbbb9df088d833772 on 31c252b54241470ba1115181c3672664 (127.5.240.194:38625) (TS 31c252b54241470ba1115181c3672664 not found in new config with opid_index 7) I0703 01:42:33.083837 28479 tablet_service.cc:800] Processing DeleteTablet for tablet 775bd6dae9ed48ebbbb9df088d833772 with delete_type TABLET_DATA_TOMBSTONED (TS 31c252b54241470ba1115181c3672664 not found in new config with opid_index 7) from {username='jenkins'} at 127.5.240.194:40764 I0703 01:42:33.084048 30807 tablet_replica.cc:261] T 775bd6dae9ed48ebbbb9df088d833772 P 31c252b54241470ba1115181c3672664: stopping tablet replica I0703 01:42:33.084113 30807 maintenance_manager.cc:237] P 31c252b54241470ba1115181c3672664: Unregistered op CompactRowSetsOp(775bd6dae9ed48ebbbb9df088d833772) I0703 01:42:33.084163 30807 maintenance_manager.cc:237] P 31c252b54241470ba1115181c3672664: Unregistered op MinorDeltaCompactionOp(775bd6dae9ed48ebbbb9df088d833772) I0703 01:42:33.084203 30807 maintenance_manager.cc:237] P 31c252b54241470ba1115181c3672664: Unregistered op MajorDeltaCompactionOp(775bd6dae9ed48ebbbb9df088d833772) I0703 01:42:33.084240 30807 maintenance_manager.cc:237] P 31c252b54241470ba1115181c3672664: Unregistered op UndoDeltaBlockGCOp(775bd6dae9ed48ebbbb9df088d833772) I0703 01:42:33.084283 30807 maintenance_manager.cc:237] P 31c252b54241470ba1115181c3672664: Unregistered op FlushMRSOp(775bd6dae9ed48ebbbb9df088d833772) I0703 01:42:33.084314 30807 maintenance_manager.cc:237] P 31c252b54241470ba1115181c3672664: Unregistered op FlushDeltaMemStoresOp(775bd6dae9ed48ebbbb9df088d833772) I0703 01:42:33.084349 30807 maintenance_manager.cc:237] P 31c252b54241470ba1115181c3672664: Unregistered op LogGCOp(775bd6dae9ed48ebbbb9df088d833772) I0703 01:42:33.084383 30807 raft_consensus.cc:2040] T 775bd6dae9ed48ebbbb9df088d833772 P 31c252b54241470ba1115181c3672664 [term 2 FOLLOWER]: Raft consensus shutting down. I0703 01:42:33.084460 30807 raft_consensus.cc:2067] T 775bd6dae9ed48ebbbb9df088d833772 P 31c252b54241470ba1115181c3672664 [term 2 FOLLOWER]: Raft consensus is shut down! I0703 01:42:33.085126 30807 ts_tablet_manager.cc:1369] T 775bd6dae9ed48ebbbb9df088d833772 P 31c252b54241470ba1115181c3672664: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:42:33.088394 30807 ts_tablet_manager.cc:1382] T 775bd6dae9ed48ebbbb9df088d833772 P 31c252b54241470ba1115181c3672664: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6 I0703 01:42:33.088452 30807 log.cc:986] T 775bd6dae9ed48ebbbb9df088d833772 P 31c252b54241470ba1115181c3672664: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal/wals/775bd6dae9ed48ebbbb9df088d833772 I0703 01:42:33.089042 28238 catalog_manager.cc:3103] TS 31c252b54241470ba1115181c3672664 (127.5.240.194:38625): tablet 775bd6dae9ed48ebbbb9df088d833772 (table rebalance_test_table_3 [id=75241f23aa654e04a8d5a26fb0d09c07]) successfully deleted I0703 01:42:33.124152 28240 catalog_manager.cc:3385] ChangeConfig:REMOVE_PEER RPC for tablet 107baa5504214d5f8362cef92ed80291 with cas_config_opid_index 5: aborting the task: latest config opid_index 7; task opid_index 5 I0703 01:42:33.132800 28239 catalog_manager.cc:3385] ChangeConfig:REMOVE_PEER RPC for tablet 41c52d5ecddf4e5897acf3fe5ee44f34 with cas_config_opid_index 5: aborting the task: latest config opid_index 7; task opid_index 5 I0703 01:42:33.238917 28240 catalog_manager.cc:3385] ChangeConfig:REMOVE_PEER RPC for tablet 775bd6dae9ed48ebbbb9df088d833772 with cas_config_opid_index 5: aborting the task: latest config opid_index 7; task opid_index 5 I0703 01:42:33.500450 30861 ts_tablet_manager.cc:669] T 12020f47e415490ebd506a2685335058 P 0ae780b86cc348d3a55bbaf5cf5a6217: Initiating tablet copy from peer 5ee4739615124e308720b4aeddfe29cb (127.5.240.193:54206) I0703 01:42:33.500602 30861 tablet_copy_client.cc:239] T 12020f47e415490ebd506a2685335058 P 0ae780b86cc348d3a55bbaf5cf5a6217: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.193:54206 I0703 01:42:33.501276 28397 tablet_copy_service.cc:135] P 5ee4739615124e308720b4aeddfe29cb: Received BeginTabletCopySession request for tablet 12020f47e415490ebd506a2685335058 from peer 0ae780b86cc348d3a55bbaf5cf5a6217 ({username='jenkins'} at 127.5.240.195:54164) I0703 01:42:33.501399 28397 tablet_copy_service.cc:156] P 5ee4739615124e308720b4aeddfe29cb: Beginning new tablet copy session on tablet 12020f47e415490ebd506a2685335058 from peer 0ae780b86cc348d3a55bbaf5cf5a6217 at {username='jenkins'} at 127.5.240.195:54164: session id = 0ae780b86cc348d3a55bbaf5cf5a6217-12020f47e415490ebd506a2685335058 I0703 01:42:33.503042 28397 tablet_copy_source_session.cc:202] T 12020f47e415490ebd506a2685335058 P 5ee4739615124e308720b4aeddfe29cb: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:42:33.504082 30861 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 12020f47e415490ebd506a2685335058. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:33.506978 30861 ts_tablet_manager.cc:1173] T 12020f47e415490ebd506a2685335058 P 0ae780b86cc348d3a55bbaf5cf5a6217: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:42:33.507323 30861 tablet_copy_client.cc:566] T 12020f47e415490ebd506a2685335058 P 0ae780b86cc348d3a55bbaf5cf5a6217: tablet copy: Starting download of 0 data blocks... I0703 01:42:33.507469 30861 tablet_copy_client.cc:529] T 12020f47e415490ebd506a2685335058 P 0ae780b86cc348d3a55bbaf5cf5a6217: tablet copy: Starting download of 1 WAL segments... I0703 01:42:33.509074 30861 tablet_copy_client.cc:405] T 12020f47e415490ebd506a2685335058 P 0ae780b86cc348d3a55bbaf5cf5a6217: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:42:33.511162 30861 ts_tablet_manager.cc:1028] T 12020f47e415490ebd506a2685335058 P 0ae780b86cc348d3a55bbaf5cf5a6217: Bootstrapping tablet I0703 01:42:33.511322 30861 tablet_bootstrap.cc:427] T 12020f47e415490ebd506a2685335058 P 0ae780b86cc348d3a55bbaf5cf5a6217: Bootstrap starting. I0703 01:42:33.553840 30861 tablet_bootstrap.cc:427] T 12020f47e415490ebd506a2685335058 P 0ae780b86cc348d3a55bbaf5cf5a6217: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=656 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:42:33.554157 30861 tablet_bootstrap.cc:427] T 12020f47e415490ebd506a2685335058 P 0ae780b86cc348d3a55bbaf5cf5a6217: Bootstrap complete. I0703 01:42:33.554320 30861 ts_tablet_manager.cc:1045] T 12020f47e415490ebd506a2685335058 P 0ae780b86cc348d3a55bbaf5cf5a6217: Time spent bootstrapping tablet: real 0.043s user 0.039s sys 0.002s I0703 01:42:33.554965 30861 raft_consensus.cc:325] T 12020f47e415490ebd506a2685335058 P 0ae780b86cc348d3a55bbaf5cf5a6217 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "0ae780b86cc348d3a55bbaf5cf5a6217" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 46219 } attrs { promote: true } } I0703 01:42:33.555135 30861 raft_consensus.cc:628] T 12020f47e415490ebd506a2685335058 P 0ae780b86cc348d3a55bbaf5cf5a6217 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0ae780b86cc348d3a55bbaf5cf5a6217, State: Initialized, Role: LEARNER I0703 01:42:33.555264 30861 consensus_queue.cc:227] T 12020f47e415490ebd506a2685335058 P 0ae780b86cc348d3a55bbaf5cf5a6217 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "0ae780b86cc348d3a55bbaf5cf5a6217" member_type: NON_VOTER last_known_addr { host: "127.5.240.195" port: 46219 } attrs { promote: true } } I0703 01:42:33.556047 30861 ts_tablet_manager.cc:1073] T 12020f47e415490ebd506a2685335058 P 0ae780b86cc348d3a55bbaf5cf5a6217: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s I0703 01:42:33.556879 28397 tablet_copy_service.cc:325] P 5ee4739615124e308720b4aeddfe29cb: Request end of tablet copy session 0ae780b86cc348d3a55bbaf5cf5a6217-12020f47e415490ebd506a2685335058 received from {username='jenkins'} at 127.5.240.195:54164 I0703 01:42:33.556943 28397 tablet_copy_service.cc:413] P 5ee4739615124e308720b4aeddfe29cb: ending tablet copy session 0ae780b86cc348d3a55bbaf5cf5a6217-12020f47e415490ebd506a2685335058 on tablet 12020f47e415490ebd506a2685335058 with peer 0ae780b86cc348d3a55bbaf5cf5a6217 I0703 01:42:34.014041 29417 raft_consensus.cc:1026] T 12020f47e415490ebd506a2685335058 P 0ae780b86cc348d3a55bbaf5cf5a6217 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[] I0703 01:42:34.560564 30892 raft_consensus.cc:890] T 12020f47e415490ebd506a2685335058 P 5ee4739615124e308720b4aeddfe29cb: attempting to promote NON_VOTER 0ae780b86cc348d3a55bbaf5cf5a6217 to VOTER I0703 01:42:34.560843 30892 consensus_peers.cc:455] T 12020f47e415490ebd506a2685335058 P 5ee4739615124e308720b4aeddfe29cb -> Peer 0ae780b86cc348d3a55bbaf5cf5a6217 (127.5.240.195:46219): Closing peer: 0ae780b86cc348d3a55bbaf5cf5a6217 I0703 01:42:34.561044 30892 consensus_queue.cc:204] T 12020f47e415490ebd506a2685335058 P 5ee4739615124e308720b4aeddfe29cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "0ae780b86cc348d3a55bbaf5cf5a6217" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46219 } attrs { promote: false } } I0703 01:42:34.562500 29417 raft_consensus.cc:1081] T 12020f47e415490ebd506a2685335058 P 0ae780b86cc348d3a55bbaf5cf5a6217 [term 1 LEARNER]: Refusing update from remote peer 5ee4739615124e308720b4aeddfe29cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch) I0703 01:42:34.563112 30893 consensus_queue.cc:962] T 12020f47e415490ebd506a2685335058 P 5ee4739615124e308720b4aeddfe29cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ae780b86cc348d3a55bbaf5cf5a6217" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46219 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I0703 01:42:34.565948 29417 raft_consensus.cc:2684] T 12020f47e415490ebd506a2685335058 P 0ae780b86cc348d3a55bbaf5cf5a6217 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0ae780b86cc348d3a55bbaf5cf5a6217 (127.5.240.195) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "0ae780b86cc348d3a55bbaf5cf5a6217" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46219 } attrs { promote: false } } } I0703 01:42:34.565284 30892 raft_consensus.cc:2684] T 12020f47e415490ebd506a2685335058 P 5ee4739615124e308720b4aeddfe29cb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 0ae780b86cc348d3a55bbaf5cf5a6217 (127.5.240.195) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "0ae780b86cc348d3a55bbaf5cf5a6217" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46219 } attrs { promote: false } } } I0703 01:42:34.569536 28251 catalog_manager.cc:3798] T 12020f47e415490ebd506a2685335058 P 0ae780b86cc348d3a55bbaf5cf5a6217 reported cstate change: config changed from index 4 to 5, 0ae780b86cc348d3a55bbaf5cf5a6217 (127.5.240.195) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5ee4739615124e308720b4aeddfe29cb" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "0ae780b86cc348d3a55bbaf5cf5a6217" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46219 } attrs { promote: false } } } I0703 01:42:34.583930 28377 tablet_service.cc:1140] Received LeaderStepDown RPC: tablet_id: "12020f47e415490ebd506a2685335058" dest_uuid: "5ee4739615124e308720b4aeddfe29cb" from {username='jenkins'} at 127.5.240.193:54867 I0703 01:42:34.584053 28377 raft_consensus.cc:552] T 12020f47e415490ebd506a2685335058 P 5ee4739615124e308720b4aeddfe29cb [term 1 LEADER]: Received request to step down I0703 01:42:34.584108 28377 raft_consensus.cc:628] T 12020f47e415490ebd506a2685335058 P 5ee4739615124e308720b4aeddfe29cb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5ee4739615124e308720b4aeddfe29cb, State: Running, Role: LEADER I0703 01:42:34.584296 28377 consensus_queue.cc:227] T 12020f47e415490ebd506a2685335058 P 5ee4739615124e308720b4aeddfe29cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "0ae780b86cc348d3a55bbaf5cf5a6217" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46219 } attrs { promote: false } } I0703 01:42:34.584431 28377 consensus_peers.cc:455] T 12020f47e415490ebd506a2685335058 P 5ee4739615124e308720b4aeddfe29cb -> Peer 0ae780b86cc348d3a55bbaf5cf5a6217 (127.5.240.195:46219): Closing peer: 0ae780b86cc348d3a55bbaf5cf5a6217 I0703 01:42:34.584537 28377 raft_consensus.cc:2730] T 12020f47e415490ebd506a2685335058 P 5ee4739615124e308720b4aeddfe29cb: Snoozing failure detection for 3.000s (explicit stepdown request) I0703 01:42:36.066649 30947 raft_consensus.cc:457] T 12020f47e415490ebd506a2685335058 P 0ae780b86cc348d3a55bbaf5cf5a6217 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5ee4739615124e308720b4aeddfe29cb) I0703 01:42:36.066730 30947 raft_consensus.cc:2730] T 12020f47e415490ebd506a2685335058 P 0ae780b86cc348d3a55bbaf5cf5a6217: Snoozing failure detection for 1.587s (starting election) I0703 01:42:36.066843 30947 raft_consensus.cc:479] T 12020f47e415490ebd506a2685335058 P 0ae780b86cc348d3a55bbaf5cf5a6217 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "0ae780b86cc348d3a55bbaf5cf5a6217" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46219 } attrs { promote: false } } I0703 01:42:36.067199 30947 leader_election.cc:243] T 12020f47e415490ebd506a2685335058 P 0ae780b86cc348d3a55bbaf5cf5a6217 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 5ee4739615124e308720b4aeddfe29cb (127.5.240.193:54206) I0703 01:42:36.068004 28377 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "12020f47e415490ebd506a2685335058" candidate_uuid: "0ae780b86cc348d3a55bbaf5cf5a6217" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "5ee4739615124e308720b4aeddfe29cb" is_pre_election: true I0703 01:42:36.068151 28377 raft_consensus.cc:2730] T 12020f47e415490ebd506a2685335058 P 5ee4739615124e308720b4aeddfe29cb: Snoozing failure detection for 1.980s (vote granted) I0703 01:42:36.068228 28377 raft_consensus.cc:2243] T 12020f47e415490ebd506a2685335058 P 5ee4739615124e308720b4aeddfe29cb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ae780b86cc348d3a55bbaf5cf5a6217 in term 1. I0703 01:42:36.068696 29348 leader_election.cc:379] T 12020f47e415490ebd506a2685335058 P 0ae780b86cc348d3a55bbaf5cf5a6217 [CANDIDATE]: Term 2 pre-election: Vote granted by peer 5ee4739615124e308720b4aeddfe29cb (127.5.240.193:54206) I0703 01:42:36.068750 29348 leader_election.cc:270] T 12020f47e415490ebd506a2685335058 P 0ae780b86cc348d3a55bbaf5cf5a6217 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. I0703 01:42:36.068917 30947 raft_consensus.cc:2730] T 12020f47e415490ebd506a2685335058 P 0ae780b86cc348d3a55bbaf5cf5a6217: Snoozing failure detection for 1.715s (election complete) I0703 01:42:36.069006 30947 raft_consensus.cc:2537] T 12020f47e415490ebd506a2685335058 P 0ae780b86cc348d3a55bbaf5cf5a6217 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0703 01:42:36.069061 30947 raft_consensus.cc:457] T 12020f47e415490ebd506a2685335058 P 0ae780b86cc348d3a55bbaf5cf5a6217 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5ee4739615124e308720b4aeddfe29cb) I0703 01:42:36.069116 30947 raft_consensus.cc:2730] T 12020f47e415490ebd506a2685335058 P 0ae780b86cc348d3a55bbaf5cf5a6217: Snoozing failure detection for 1.583s (starting election) I0703 01:42:36.069175 30947 raft_consensus.cc:2782] T 12020f47e415490ebd506a2685335058 P 0ae780b86cc348d3a55bbaf5cf5a6217 [term 1 FOLLOWER]: Advancing to term 2 I0703 01:42:36.070729 30947 raft_consensus.cc:479] T 12020f47e415490ebd506a2685335058 P 0ae780b86cc348d3a55bbaf5cf5a6217 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "0ae780b86cc348d3a55bbaf5cf5a6217" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46219 } attrs { promote: false } } I0703 01:42:36.070973 30947 leader_election.cc:243] T 12020f47e415490ebd506a2685335058 P 0ae780b86cc348d3a55bbaf5cf5a6217 [CANDIDATE]: Term 2 election: Requesting vote from peer 5ee4739615124e308720b4aeddfe29cb (127.5.240.193:54206) I0703 01:42:36.071528 28377 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "12020f47e415490ebd506a2685335058" candidate_uuid: "0ae780b86cc348d3a55bbaf5cf5a6217" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "5ee4739615124e308720b4aeddfe29cb" I0703 01:42:36.071655 28377 raft_consensus.cc:2782] T 12020f47e415490ebd506a2685335058 P 5ee4739615124e308720b4aeddfe29cb [term 1 FOLLOWER]: Advancing to term 2 I0703 01:42:36.071719 28377 raft_consensus.cc:2730] T 12020f47e415490ebd506a2685335058 P 5ee4739615124e308720b4aeddfe29cb: Snoozing failure detection for 2.192s (vote granted) I0703 01:42:36.073026 28377 raft_consensus.cc:2243] T 12020f47e415490ebd506a2685335058 P 5ee4739615124e308720b4aeddfe29cb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ae780b86cc348d3a55bbaf5cf5a6217 in term 2. I0703 01:42:36.073398 29348 leader_election.cc:379] T 12020f47e415490ebd506a2685335058 P 0ae780b86cc348d3a55bbaf5cf5a6217 [CANDIDATE]: Term 2 election: Vote granted by peer 5ee4739615124e308720b4aeddfe29cb (127.5.240.193:54206) I0703 01:42:36.073454 29348 leader_election.cc:270] T 12020f47e415490ebd506a2685335058 P 0ae780b86cc348d3a55bbaf5cf5a6217 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0703 01:42:36.073637 30947 raft_consensus.cc:2730] T 12020f47e415490ebd506a2685335058 P 0ae780b86cc348d3a55bbaf5cf5a6217: Snoozing failure detection for 1.832s (election complete) I0703 01:42:36.073711 30947 raft_consensus.cc:2537] T 12020f47e415490ebd506a2685335058 P 0ae780b86cc348d3a55bbaf5cf5a6217 [term 2 FOLLOWER]: Leader election won for term 2 I0703 01:42:36.074007 30947 raft_consensus.cc:593] T 12020f47e415490ebd506a2685335058 P 0ae780b86cc348d3a55bbaf5cf5a6217 [term 2 LEADER]: Becoming Leader. State: Replica: 0ae780b86cc348d3a55bbaf5cf5a6217, State: Running, Role: LEADER I0703 01:42:36.074235 30947 consensus_queue.cc:204] T 12020f47e415490ebd506a2685335058 P 0ae780b86cc348d3a55bbaf5cf5a6217 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } } peers { permanent_uuid: "0ae780b86cc348d3a55bbaf5cf5a6217" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46219 } attrs { promote: false } } I0703 01:42:36.076022 28253 catalog_manager.cc:3798] T 12020f47e415490ebd506a2685335058 P 0ae780b86cc348d3a55bbaf5cf5a6217 reported cstate change: term changed from 1 to 2, leader changed from 5ee4739615124e308720b4aeddfe29cb (127.5.240.193) to 0ae780b86cc348d3a55bbaf5cf5a6217 (127.5.240.195). New cstate: current_term: 2 leader_uuid: "0ae780b86cc348d3a55bbaf5cf5a6217" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ae780b86cc348d3a55bbaf5cf5a6217" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46219 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I0703 01:42:36.079407 28253 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 12020f47e415490ebd506a2685335058 (attempt 1) I0703 01:42:36.080286 28239 catalog_manager.cc:3369] ChangeConfig:REMOVE_PEER RPC for tablet 12020f47e415490ebd506a2685335058 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 0ae780b86cc348d3a55bbaf5cf5a6217 (127.5.240.195:46219) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term I0703 01:42:36.080355 28239 catalog_manager.cc:2903] Scheduling retry of ChangeConfig:REMOVE_PEER RPC for tablet 12020f47e415490ebd506a2685335058 with cas_config_opid_index 5 with a delay of 50 ms (attempt = 1) I0703 01:42:36.131157 28239 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 12020f47e415490ebd506a2685335058 (attempt 2) I0703 01:42:36.132200 28239 catalog_manager.cc:3369] ChangeConfig:REMOVE_PEER RPC for tablet 12020f47e415490ebd506a2685335058 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 0ae780b86cc348d3a55bbaf5cf5a6217 (127.5.240.195:46219) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term I0703 01:42:36.132283 28239 catalog_manager.cc:2903] Scheduling retry of ChangeConfig:REMOVE_PEER RPC for tablet 12020f47e415490ebd506a2685335058 with cas_config_opid_index 5 with a delay of 48 ms (attempt = 2) I0703 01:42:36.180686 28239 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 12020f47e415490ebd506a2685335058 (attempt 3) I0703 01:42:36.181643 28239 catalog_manager.cc:3369] ChangeConfig:REMOVE_PEER RPC for tablet 12020f47e415490ebd506a2685335058 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 0ae780b86cc348d3a55bbaf5cf5a6217 (127.5.240.195:46219) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term I0703 01:42:36.181716 28239 catalog_manager.cc:2903] Scheduling retry of ChangeConfig:REMOVE_PEER RPC for tablet 12020f47e415490ebd506a2685335058 with cas_config_opid_index 5 with a delay of 78 ms (attempt = 3) I0703 01:42:36.260272 28239 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 12020f47e415490ebd506a2685335058 (attempt 4) I0703 01:42:36.260965 28239 catalog_manager.cc:3369] ChangeConfig:REMOVE_PEER RPC for tablet 12020f47e415490ebd506a2685335058 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 0ae780b86cc348d3a55bbaf5cf5a6217 (127.5.240.195:46219) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term I0703 01:42:36.261039 28239 catalog_manager.cc:2903] Scheduling retry of ChangeConfig:REMOVE_PEER RPC for tablet 12020f47e415490ebd506a2685335058 with cas_config_opid_index 5 with a delay of 163 ms (attempt = 4) I0703 01:42:36.425179 28239 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 12020f47e415490ebd506a2685335058 (attempt 5) I0703 01:42:36.426187 28239 catalog_manager.cc:3369] ChangeConfig:REMOVE_PEER RPC for tablet 12020f47e415490ebd506a2685335058 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 0ae780b86cc348d3a55bbaf5cf5a6217 (127.5.240.195:46219) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term I0703 01:42:36.426259 28239 catalog_manager.cc:2903] Scheduling retry of ChangeConfig:REMOVE_PEER RPC for tablet 12020f47e415490ebd506a2685335058 with cas_config_opid_index 5 with a delay of 281 ms (attempt = 5) I0703 01:42:36.483047 28377 raft_consensus.cc:1081] T 12020f47e415490ebd506a2685335058 P 5ee4739615124e308720b4aeddfe29cb [term 2 FOLLOWER]: Refusing update from remote peer 0ae780b86cc348d3a55bbaf5cf5a6217: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch) I0703 01:42:36.483685 30947 consensus_queue.cc:962] T 12020f47e415490ebd506a2685335058 P 0ae780b86cc348d3a55bbaf5cf5a6217 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ee4739615124e308720b4aeddfe29cb" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54206 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I0703 01:42:36.487994 28253 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 12020f47e415490ebd506a2685335058 (attempt 1) I0703 01:42:36.488852 29417 consensus_peers.cc:455] T 12020f47e415490ebd506a2685335058 P 0ae780b86cc348d3a55bbaf5cf5a6217 -> Peer 5ee4739615124e308720b4aeddfe29cb (127.5.240.193:54206): Closing peer: 5ee4739615124e308720b4aeddfe29cb I0703 01:42:36.489032 29417 consensus_queue.cc:204] T 12020f47e415490ebd506a2685335058 P 0ae780b86cc348d3a55bbaf5cf5a6217 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0ae780b86cc348d3a55bbaf5cf5a6217" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46219 } attrs { promote: false } } I0703 01:42:36.489604 30948 raft_consensus.cc:2684] T 12020f47e415490ebd506a2685335058 P 0ae780b86cc348d3a55bbaf5cf5a6217 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 5ee4739615124e308720b4aeddfe29cb (127.5.240.193) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0ae780b86cc348d3a55bbaf5cf5a6217" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46219 } attrs { promote: false } } } I0703 01:42:36.491633 28239 catalog_manager.cc:3352] ChangeConfig:REMOVE_PEER RPC for tablet 12020f47e415490ebd506a2685335058 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0703 01:42:36.492368 28251 catalog_manager.cc:3798] T 12020f47e415490ebd506a2685335058 P 0ae780b86cc348d3a55bbaf5cf5a6217 reported cstate change: config changed from index 5 to 7, VOTER 5ee4739615124e308720b4aeddfe29cb (127.5.240.193) evicted. New cstate: current_term: 2 leader_uuid: "0ae780b86cc348d3a55bbaf5cf5a6217" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0ae780b86cc348d3a55bbaf5cf5a6217" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 46219 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I0703 01:42:36.494516 28251 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 12020f47e415490ebd506a2685335058 on 5ee4739615124e308720b4aeddfe29cb (127.5.240.193:54206) (TS 5ee4739615124e308720b4aeddfe29cb not found in new config with opid_index 7) I0703 01:42:36.495092 28356 tablet_service.cc:800] Processing DeleteTablet for tablet 12020f47e415490ebd506a2685335058 with delete_type TABLET_DATA_TOMBSTONED (TS 5ee4739615124e308720b4aeddfe29cb not found in new config with opid_index 7) from {username='jenkins'} at 127.5.240.193:54680 I0703 01:42:36.495606 30967 tablet_replica.cc:261] T 12020f47e415490ebd506a2685335058 P 5ee4739615124e308720b4aeddfe29cb: stopping tablet replica I0703 01:42:36.495699 30967 maintenance_manager.cc:237] P 5ee4739615124e308720b4aeddfe29cb: Unregistered op CompactRowSetsOp(12020f47e415490ebd506a2685335058) I0703 01:42:36.495769 30967 maintenance_manager.cc:237] P 5ee4739615124e308720b4aeddfe29cb: Unregistered op MinorDeltaCompactionOp(12020f47e415490ebd506a2685335058) I0703 01:42:36.495833 30967 maintenance_manager.cc:237] P 5ee4739615124e308720b4aeddfe29cb: Unregistered op MajorDeltaCompactionOp(12020f47e415490ebd506a2685335058) I0703 01:42:36.495898 30967 maintenance_manager.cc:237] P 5ee4739615124e308720b4aeddfe29cb: Unregistered op UndoDeltaBlockGCOp(12020f47e415490ebd506a2685335058) I0703 01:42:36.495959 30967 maintenance_manager.cc:237] P 5ee4739615124e308720b4aeddfe29cb: Unregistered op FlushMRSOp(12020f47e415490ebd506a2685335058) I0703 01:42:36.496013 30967 maintenance_manager.cc:237] P 5ee4739615124e308720b4aeddfe29cb: Unregistered op FlushDeltaMemStoresOp(12020f47e415490ebd506a2685335058) I0703 01:42:36.496060 30967 maintenance_manager.cc:237] P 5ee4739615124e308720b4aeddfe29cb: Unregistered op LogGCOp(12020f47e415490ebd506a2685335058) I0703 01:42:36.496117 30967 raft_consensus.cc:2040] T 12020f47e415490ebd506a2685335058 P 5ee4739615124e308720b4aeddfe29cb [term 2 FOLLOWER]: Raft consensus shutting down. I0703 01:42:36.496218 30967 raft_consensus.cc:2067] T 12020f47e415490ebd506a2685335058 P 5ee4739615124e308720b4aeddfe29cb [term 2 FOLLOWER]: Raft consensus is shut down! I0703 01:42:36.496992 30967 ts_tablet_manager.cc:1369] T 12020f47e415490ebd506a2685335058 P 5ee4739615124e308720b4aeddfe29cb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:42:36.499864 30967 ts_tablet_manager.cc:1382] T 12020f47e415490ebd506a2685335058 P 5ee4739615124e308720b4aeddfe29cb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6 I0703 01:42:36.499948 30967 log.cc:986] T 12020f47e415490ebd506a2685335058 P 5ee4739615124e308720b4aeddfe29cb: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal/wals/12020f47e415490ebd506a2685335058 I0703 01:42:36.500572 28238 catalog_manager.cc:3103] TS 5ee4739615124e308720b4aeddfe29cb (127.5.240.193:54206): tablet 12020f47e415490ebd506a2685335058 (table rebalance_test_table_2 [id=b5ac6ea59801477a8e4cca2b80c8beec]) successfully deleted I0703 01:42:36.708078 28239 catalog_manager.cc:3385] ChangeConfig:REMOVE_PEER RPC for tablet 12020f47e415490ebd506a2685335058 with cas_config_opid_index 5: aborting the task: latest config opid_index 7; task opid_index 5 Master Summary UUID | Address | Status ----------------------------------+---------------------+--------- 651f357379414cf8959c30179d420b62 | 127.5.240.254:47851 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- allow_unsafe_replication_factor | true | unsafe | all 1 server(s) checked ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status ----------------------------------+---------------------+--------- 0ae780b86cc348d3a55bbaf5cf5a6217 | 127.5.240.195:46219 | HEALTHY 31c252b54241470ba1115181c3672664 | 127.5.240.194:38625 | HEALTHY 35f0266742fc4707a8cc10c6437b84ed | 127.5.240.196:50368 | HEALTHY 575620f4889c4704b7a7c00da34b6a53 | 127.5.240.198:39152 | HEALTHY 5ee4739615124e308720b4aeddfe29cb | 127.5.240.193:54206 | HEALTHY 6505fcf874c440788e377dac9aa99209 | 127.5.240.199:33903 | HEALTHY e98d29f36f2347f29170e987cb83ced4 | 127.5.240.197:41466 | HEALTHY Flag | Value | Tags | Tablet Server ------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_server_key_size | 1024 | experimental | all 7 server(s) checked local_ip_for_outbound_sockets | 127.5.240.193 | experimental | 127.5.240.193:54206 local_ip_for_outbound_sockets | 127.5.240.194 | experimental | 127.5.240.194:38625 local_ip_for_outbound_sockets | 127.5.240.195 | experimental | 127.5.240.195:46219 local_ip_for_outbound_sockets | 127.5.240.196 | experimental | 127.5.240.196:50368 local_ip_for_outbound_sockets | 127.5.240.197 | experimental | 127.5.240.197:41466 local_ip_for_outbound_sockets | 127.5.240.198 | experimental | 127.5.240.198:39152 local_ip_for_outbound_sockets | 127.5.240.199 | experimental | 127.5.240.199:33903 never_fsync | true | unsafe,advanced | all 7 server(s) checked raft_prepare_replacement_before_eviction | true | experimental,advanced | all 7 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked server_dump_info_format | pb | hidden | all 7 server(s) checked server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.5.240.193:54206 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.240.194:38625 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.240.195:46219 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.5.240.196:50368 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.240.197:41466 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.240.198:39152 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_1.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.5.240.199:33903 Version Summary Version | Servers ----------------+------------------------- 1.8.0-SNAPSHOT | all 8 server(s) checked Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ------------------------+----+---------+---------------+---------+------------+------------------+------------- TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0 rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0 rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0 rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0 rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0 rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0 | Total Count ----------------+------------- Masters | 1 Tablet Servers | 7 Tables | 6 Tablets | 16 Replicas | 16 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set OK I0703 01:42:37.724058 6083 log_verifier.cc:121] Checking tablet 107baa5504214d5f8362cef92ed80291 I0703 01:42:37.735606 6083 log_verifier.cc:172] Verified matching terms for 7 ops in tablet 107baa5504214d5f8362cef92ed80291 I0703 01:42:37.735652 6083 log_verifier.cc:121] Checking tablet 12020f47e415490ebd506a2685335058 I0703 01:42:37.746613 6083 log_verifier.cc:172] Verified matching terms for 7 ops in tablet 12020f47e415490ebd506a2685335058 I0703 01:42:37.746650 6083 log_verifier.cc:121] Checking tablet 1cd5cbfe70ec4f8a957fa074ae08fcb1 I0703 01:42:37.757201 6083 log_verifier.cc:172] Verified matching terms for 7 ops in tablet 1cd5cbfe70ec4f8a957fa074ae08fcb1 I0703 01:42:37.757239 6083 log_verifier.cc:121] Checking tablet 41c52d5ecddf4e5897acf3fe5ee44f34 I0703 01:42:37.767629 6083 log_verifier.cc:172] Verified matching terms for 7 ops in tablet 41c52d5ecddf4e5897acf3fe5ee44f34 I0703 01:42:37.767668 6083 log_verifier.cc:121] Checking tablet 45b3833ac0664a8d9889ab14b3d0dd69 I0703 01:42:37.778522 6083 log_verifier.cc:172] Verified matching terms for 7 ops in tablet 45b3833ac0664a8d9889ab14b3d0dd69 I0703 01:42:37.778587 6083 log_verifier.cc:121] Checking tablet 46f9123e80ac47f59d142ca87e99a1d5 I0703 01:42:37.788491 6083 log_verifier.cc:172] Verified matching terms for 3 ops in tablet 46f9123e80ac47f59d142ca87e99a1d5 I0703 01:42:37.788523 6083 log_verifier.cc:121] Checking tablet 4d69010513df4b68840b90be3161f402 I0703 01:42:37.799188 6083 log_verifier.cc:172] Verified matching terms for 7 ops in tablet 4d69010513df4b68840b90be3161f402 I0703 01:42:37.799226 6083 log_verifier.cc:121] Checking tablet 74b62768046f4b8690aab7d2ec0f803a I0703 01:42:37.824671 6083 log_verifier.cc:172] Verified matching terms for 7 ops in tablet 74b62768046f4b8690aab7d2ec0f803a I0703 01:42:37.824709 6083 log_verifier.cc:121] Checking tablet 775bd6dae9ed48ebbbb9df088d833772 I0703 01:42:37.834868 6083 log_verifier.cc:172] Verified matching terms for 7 ops in tablet 775bd6dae9ed48ebbbb9df088d833772 I0703 01:42:37.834903 6083 log_verifier.cc:121] Checking tablet 776d4f14ef3e4b5494f5834ae1676782 I0703 01:42:37.844540 6083 log_verifier.cc:172] Verified matching terms for 3 ops in tablet 776d4f14ef3e4b5494f5834ae1676782 I0703 01:42:37.844594 6083 log_verifier.cc:121] Checking tablet 7f2afbb6827246e1be0babd14a9d3cbd I0703 01:42:37.852705 6083 log_verifier.cc:172] Verified matching terms for 1 ops in tablet 7f2afbb6827246e1be0babd14a9d3cbd I0703 01:42:37.852740 6083 log_verifier.cc:121] Checking tablet b4acd291b108434da9b8791c61fd1dc2 I0703 01:42:37.864629 6083 log_verifier.cc:172] Verified matching terms for 7 ops in tablet b4acd291b108434da9b8791c61fd1dc2 I0703 01:42:37.864663 6083 log_verifier.cc:121] Checking tablet ca3c350378324abcb8b68a2aaa4ba5a7 I0703 01:42:37.874264 6083 log_verifier.cc:172] Verified matching terms for 3 ops in tablet ca3c350378324abcb8b68a2aaa4ba5a7 I0703 01:42:37.874295 6083 log_verifier.cc:121] Checking tablet d0d06a73ceef43ffae542e800f00ff20 I0703 01:42:37.884155 6083 log_verifier.cc:172] Verified matching terms for 3 ops in tablet d0d06a73ceef43ffae542e800f00ff20 I0703 01:42:37.884186 6083 log_verifier.cc:121] Checking tablet ef65b31bc12e40c2b3f1e4714cc4218e I0703 01:42:37.895225 6083 log_verifier.cc:172] Verified matching terms for 7 ops in tablet ef65b31bc12e40c2b3f1e4714cc4218e I0703 01:42:37.895261 6083 log_verifier.cc:121] Checking tablet f6594322fcdd4cd3a1357b7bda51b648 I0703 01:42:37.904958 6083 log_verifier.cc:172] Verified matching terms for 3 ops in tablet f6594322fcdd4cd3a1357b7bda51b648 I0703 01:42:37.905961 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 28291 I0703 01:42:37.908494 28417 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:42:37.981961 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 28291 W0703 01:42:37.995746 29593 connection.cc:512] client connection to 127.5.240.193:54206 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:54206): Connection reset by peer (error 104) W0703 01:42:37.995837 28238 connection.cc:512] server connection from 127.5.240.193:54456 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:54456): Connection reset by peer (error 104) W0703 01:42:37.996282 29596 connection.cc:512] server connection from 127.5.240.193:38623 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:38623): Connection reset by peer (error 104) W0703 01:42:37.996281 29348 connection.cc:512] client connection to 127.5.240.193:54206 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:54206): Connection reset by peer (error 104) W0703 01:42:37.996327 29471 connection.cc:512] server connection from 127.5.240.193:42533 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:42533): Connection reset by peer (error 104) W0703 01:42:37.996732 29350 connection.cc:512] server connection from 127.5.240.193:33904 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:33904): Connection reset by peer (error 104) W0703 01:42:37.996774 29471 connection.cc:512] client connection to 127.5.240.193:54206 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:54206): Connection reset by peer (error 104) W0703 01:42:37.996829 29837 connection.cc:512] server connection from 127.5.240.193:37281 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:37281): Connection reset by peer (error 104) W0703 01:42:37.997066 29715 connection.cc:512] server connection from 127.5.240.193:49674 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:49674): Connection reset by peer (error 104) W0703 01:42:37.997207 29837 connection.cc:512] client connection to 127.5.240.193:54206 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:54206): Connection reset by peer (error 104) I0703 01:42:37.997272 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 28424 W0703 01:42:37.997439 29715 connection.cc:512] client connection to 127.5.240.193:54206 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:54206): Connection reset by peer (error 104) I0703 01:42:37.998000 28540 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:42:38.064087 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 28424 W0703 01:42:38.076704 29837 connection.cc:512] client connection to 127.5.240.194:38625 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:38625): Connection reset by peer (error 104) W0703 01:42:38.076920 28241 connection.cc:512] server connection from 127.5.240.194:55278 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:55278): Connection reset by peer (error 104) W0703 01:42:38.077153 29839 connection.cc:512] server connection from 127.5.240.194:49532 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:49532): Connection reset by peer (error 104) W0703 01:42:38.077222 29593 connection.cc:512] client connection to 127.5.240.194:38625 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:38625): Connection reset by peer (error 104) W0703 01:42:38.077589 29472 connection.cc:512] server connection from 127.5.240.194:57349 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:57349): Connection reset by peer (error 104) W0703 01:42:38.077576 29593 connection.cc:512] server connection from 127.5.240.194:53734 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:53734): Connection reset by peer (error 104) W0703 01:42:38.077965 29471 connection.cc:512] client connection to 127.5.240.194:38625 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:38625): Connection reset by peer (error 104) W0703 01:42:38.077962 29718 connection.cc:512] server connection from 127.5.240.194:47386 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:47386): Connection reset by peer (error 104) W0703 01:42:38.078294 29715 connection.cc:512] client connection to 127.5.240.194:38625 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:38625): Connection reset by peer (error 104) I0703 01:42:38.078384 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 29341 I0703 01:42:38.082185 29457 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:42:38.133298 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 29341 W0703 01:42:38.142967 28239 connection.cc:512] server connection from 127.5.240.195:40747 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.195:40747): Connection reset by peer (error 104) I0703 01:42:38.143111 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 29464 I0703 01:42:38.146934 29580 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:42:38.197806 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 29464 W0703 01:42:38.207557 28240 connection.cc:512] server connection from 127.5.240.196:59868 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.196:59868): Connection reset by peer (error 104) I0703 01:42:38.207614 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 29586 I0703 01:42:38.211266 29702 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:42:38.262488 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 29586 W0703 01:42:38.271358 28241 connection.cc:512] server connection from 127.5.240.197:52096 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.197:52096): Connection reset by peer (error 104) I0703 01:42:38.271466 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 29708 I0703 01:42:38.275631 29824 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:42:38.326807 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 29708 W0703 01:42:38.335935 28239 connection.cc:512] server connection from 127.5.240.198:45849 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.198:45849): Connection reset by peer (error 104) I0703 01:42:38.336143 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 29830 I0703 01:42:38.340003 29946 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:42:38.390374 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 29830 W0703 01:42:38.399451 28240 connection.cc:512] server connection from 127.5.240.199:42019 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.199:42019): Connection reset by peer (error 104) I0703 01:42:38.399730 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master pid 28231 I0703 01:42:38.400420 28287 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:42:38.479408 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master with pid 28231 [ OK ] RebalanceParamTest.Rebalance/1 (20934 ms) [ RUN ] RebalanceParamTest.Rebalance/2 I0703 01:42:38.520259 6083 test_util.cc:208] Using random seed: 1273310138 I0703 01:42:38.521827 6083 ts_itest-base.cc:113] Starting cluster with: I0703 01:42:38.521848 6083 ts_itest-base.cc:114] -------------- I0703 01:42:38.521858 6083 ts_itest-base.cc:115] 7 tablet servers I0703 01:42:38.521867 6083 ts_itest-base.cc:116] 2 replicas per TS I0703 01:42:38.521876 6083 ts_itest-base.cc:117] -------------- I0703 01:42:38.522903 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:36104 --webserver_interface=127.5.240.254 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --allow_unsafe_replication_factor --raft_prepare_replacement_before_eviction=false --tserver_unresponsive_timeout_ms=3000 --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:42:38.708410 31126 flags.cc:406] Enabled unsafe flag: --allow_unsafe_replication_factor=true W0703 01:42:38.709401 31126 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:42:38.709463 31126 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:42:38.711680 31126 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0703 01:42:38.711738 31126 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0703 01:42:38.711773 31126 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:42:38.711799 31126 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0703 01:42:38.711836 31126 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0703 01:42:38.720052 31126 master_main.cc:73] Master server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --allow_unsafe_replication_factor=true --ipki_ca_key_size=1024 --tserver_unresponsive_timeout_ms=3000 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:36104 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.5.240.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:42:38.725764 31126 master_main.cc:80] Initializing master server... I0703 01:42:38.726436 31126 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 232708us I0703 01:42:38.727466 31126 fs_manager.cc:261] Metadata directory not provided I0703 01:42:38.727521 31126 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:42:38.727794 31126 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:42:38.727856 31126 server_base.cc:433] Attempting to create new FS layout instead I0703 01:42:38.731976 31126 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/instance: uuid: "61b07253093846a589ff6e2547007d7d" format_stamp: "Formatted at 2018-07-03 08:42:38 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:42:38.732863 31126 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "61b07253093846a589ff6e2547007d7d" format_stamp: "Formatted at 2018-07-03 08:42:38 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:42:38.740916 31126 fs_manager.cc:501] Time spent creating directory manager: real 0.008s user 0.004s sys 0.003s I0703 01:42:38.742125 31126 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:42:38.742913 31126 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:42:38.746618 31126 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s I0703 01:42:38.746672 31126 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal uuid: "61b07253093846a589ff6e2547007d7d" format_stamp: "Formatted at 2018-07-03 08:42:38 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:42:38.746783 31126 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:42:38.807217 31126 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:42:38.808161 31126 master_main.cc:83] Starting Master server... I0703 01:42:38.844990 31126 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.254:36104 I0703 01:42:38.845459 31126 webserver.cc:175] Starting webserver on 127.5.240.254:0 I0703 01:42:38.845507 31126 webserver.cc:186] Document root disabled I0703 01:42:38.846191 31126 webserver.cc:313] Webserver started. Bound to: http://127.5.240.254:60914/ I0703 01:42:38.846807 31126 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb I0703 01:42:38.849860 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master as pid 31126 I0703 01:42:38.850749 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.5.240.193:0 --local_ip_for_outbound_sockets=127.5.240.193 --webserver_interface=127.5.240.193 --webserver_port=0 --tserver_master_addrs=127.5.240.254:36104 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0703 01:42:38.851445 31185 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:38.865648 31185 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 61b07253093846a589ff6e2547007d7d: Bootstrap starting. I0703 01:42:38.867758 31185 tablet_bootstrap.cc:578] T 00000000000000000000000000000000 P 61b07253093846a589ff6e2547007d7d: No blocks or log segments found. Creating new log. I0703 01:42:38.868501 31185 log.cc:525] T 00000000000000000000000000000000 P 61b07253093846a589ff6e2547007d7d: Log is configured to *not* fsync() on all Append() calls I0703 01:42:38.870364 31185 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P 61b07253093846a589ff6e2547007d7d: No bootstrap required, opened a new log I0703 01:42:38.876555 31185 raft_consensus.cc:325] T 00000000000000000000000000000000 P 61b07253093846a589ff6e2547007d7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61b07253093846a589ff6e2547007d7d" member_type: VOTER } I0703 01:42:38.876776 31185 raft_consensus.cc:351] T 00000000000000000000000000000000 P 61b07253093846a589ff6e2547007d7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:42:38.876838 31185 raft_consensus.cc:628] T 00000000000000000000000000000000 P 61b07253093846a589ff6e2547007d7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61b07253093846a589ff6e2547007d7d, State: Initialized, Role: FOLLOWER I0703 01:42:38.877305 31185 consensus_queue.cc:227] T 00000000000000000000000000000000 P 61b07253093846a589ff6e2547007d7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61b07253093846a589ff6e2547007d7d" member_type: VOTER } I0703 01:42:38.877450 31185 raft_consensus.cc:365] T 00000000000000000000000000000000 P 61b07253093846a589ff6e2547007d7d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:42:38.877532 31185 raft_consensus.cc:457] T 00000000000000000000000000000000 P 61b07253093846a589ff6e2547007d7d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:42:38.877627 31185 raft_consensus.cc:2730] T 00000000000000000000000000000000 P 61b07253093846a589ff6e2547007d7d: Snoozing failure detection for 1.636s (starting election) I0703 01:42:38.877715 31185 raft_consensus.cc:2782] T 00000000000000000000000000000000 P 61b07253093846a589ff6e2547007d7d [term 0 FOLLOWER]: Advancing to term 1 I0703 01:42:38.879055 31185 raft_consensus.cc:479] T 00000000000000000000000000000000 P 61b07253093846a589ff6e2547007d7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61b07253093846a589ff6e2547007d7d" member_type: VOTER } I0703 01:42:38.879410 31185 leader_election.cc:270] T 00000000000000000000000000000000 P 61b07253093846a589ff6e2547007d7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:42:38.880872 31188 raft_consensus.cc:2730] T 00000000000000000000000000000000 P 61b07253093846a589ff6e2547007d7d: Snoozing failure detection for 1.808s (election complete) I0703 01:42:38.881134 31188 raft_consensus.cc:2537] T 00000000000000000000000000000000 P 61b07253093846a589ff6e2547007d7d [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:42:38.882778 31188 raft_consensus.cc:593] T 00000000000000000000000000000000 P 61b07253093846a589ff6e2547007d7d [term 1 LEADER]: Becoming Leader. State: Replica: 61b07253093846a589ff6e2547007d7d, State: Running, Role: LEADER I0703 01:42:38.883139 31188 consensus_queue.cc:204] T 00000000000000000000000000000000 P 61b07253093846a589ff6e2547007d7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61b07253093846a589ff6e2547007d7d" member_type: VOTER } I0703 01:42:38.885836 31185 sys_catalog.cc:337] T 00000000000000000000000000000000 P 61b07253093846a589ff6e2547007d7d [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "61b07253093846a589ff6e2547007d7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61b07253093846a589ff6e2547007d7d" member_type: VOTER } } I0703 01:42:38.885540 31189 sys_catalog.cc:337] T 00000000000000000000000000000000 P 61b07253093846a589ff6e2547007d7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61b07253093846a589ff6e2547007d7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61b07253093846a589ff6e2547007d7d" member_type: VOTER } } I0703 01:42:38.886361 31189 sys_catalog.cc:340] T 00000000000000000000000000000000 P 61b07253093846a589ff6e2547007d7d [sys.catalog]: This master's current role is: LEADER I0703 01:42:38.886750 31185 sys_catalog.cc:340] T 00000000000000000000000000000000 P 61b07253093846a589ff6e2547007d7d [sys.catalog]: This master's current role is: LEADER I0703 01:42:38.885840 31190 sys_catalog.cc:337] T 00000000000000000000000000000000 P 61b07253093846a589ff6e2547007d7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61b07253093846a589ff6e2547007d7d. Latest consensus state: current_term: 1 leader_uuid: "61b07253093846a589ff6e2547007d7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61b07253093846a589ff6e2547007d7d" member_type: VOTER } } I0703 01:42:38.887611 31190 sys_catalog.cc:340] T 00000000000000000000000000000000 P 61b07253093846a589ff6e2547007d7d [sys.catalog]: This master's current role is: LEADER I0703 01:42:38.889160 31185 sys_catalog.cc:424] T 00000000000000000000000000000000 P 61b07253093846a589ff6e2547007d7d [sys.catalog]: configured and running, proceeding with master startup. I0703 01:42:38.892182 31126 master_main.cc:86] Master server successfully started. I0703 01:42:38.892237 31192 catalog_manager.cc:1006] Loading table and tablet metadata into memory... I0703 01:42:38.897476 31192 catalog_manager.cc:1017] Initializing Kudu internal certificate authority... I0703 01:42:38.961757 31192 catalog_manager.cc:904] Generated new certificate authority record I0703 01:42:38.963750 31192 catalog_manager.cc:1026] Loading token signing keys... I0703 01:42:38.976617 31192 catalog_manager.cc:4102] T 00000000000000000000000000000000 P 61b07253093846a589ff6e2547007d7d: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:42:39.025388 31186 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:42:39.026394 31186 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:42:39.028375 31186 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0703 01:42:39.028424 31186 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:42:39.028554 31186 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.193 I0703 01:42:39.034991 31186 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.5.240.193 --webserver_port=0 --tserver_master_addrs=127.5.240.254:36104 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.193 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:42:39.041599 31186 tablet_server_main.cc:85] Initializing tablet server... I0703 01:42:39.043154 31186 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 233208us I0703 01:42:39.043740 31186 fs_manager.cc:261] Metadata directory not provided I0703 01:42:39.043805 31186 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:42:39.044042 31186 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:42:39.044112 31186 server_base.cc:433] Attempting to create new FS layout instead I0703 01:42:39.048302 31186 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "1288ca0989104bbea36e22f38c374218" format_stamp: "Formatted at 2018-07-03 08:42:39 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:42:39.048813 31186 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "1288ca0989104bbea36e22f38c374218" format_stamp: "Formatted at 2018-07-03 08:42:39 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:42:39.057461 31186 fs_manager.cc:501] Time spent creating directory manager: real 0.008s user 0.002s sys 0.004s I0703 01:42:39.058698 31186 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:42:39.059106 31186 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:42:39.062741 31186 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0703 01:42:39.062793 31186 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal uuid: "1288ca0989104bbea36e22f38c374218" format_stamp: "Formatted at 2018-07-03 08:42:39 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:42:39.062929 31186 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:42:39.132160 31186 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:42:39.133745 31186 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:42:39.134129 31186 tablet_server_main.cc:90] Starting tablet server... I0703 01:42:39.252215 31186 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.193:33521 I0703 01:42:39.252652 31186 webserver.cc:175] Starting webserver on 127.5.240.193:0 I0703 01:42:39.252699 31186 webserver.cc:186] Document root disabled I0703 01:42:39.253876 31186 webserver.cc:313] Webserver started. Bound to: http://127.5.240.193:52605/ I0703 01:42:39.254405 31186 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb I0703 01:42:39.255331 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 31186 I0703 01:42:39.256136 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.5.240.194:0 --local_ip_for_outbound_sockets=127.5.240.194 --webserver_interface=127.5.240.194 --webserver_port=0 --tserver_master_addrs=127.5.240.254:36104 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0703 01:42:39.261725 31186 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:42:39.279266 31315 heartbeater.cc:346] Connected to a master server at 127.5.240.254:36104 I0703 01:42:39.279454 31315 heartbeater.cc:426] Registering TS with master... I0703 01:42:39.280758 31315 heartbeater.cc:475] Master 127.5.240.254:36104 requested a full tablet report, sending... I0703 01:42:39.282320 31152 ts_manager.cc:98] Registered new tserver with Master: 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521) I0703 01:42:39.284950 31152 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.193:48823 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:42:39.425309 31319 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:42:39.426306 31319 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:42:39.428278 31319 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0703 01:42:39.428325 31319 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:42:39.428411 31319 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.194 I0703 01:42:39.435413 31319 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.194:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.5.240.194 --webserver_port=0 --tserver_master_addrs=127.5.240.254:36104 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.194 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:42:39.442046 31319 tablet_server_main.cc:85] Initializing tablet server... I0703 01:42:39.443615 31319 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 233208us I0703 01:42:39.444180 31319 fs_manager.cc:261] Metadata directory not provided I0703 01:42:39.444237 31319 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:42:39.444453 31319 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:42:39.444514 31319 server_base.cc:433] Attempting to create new FS layout instead I0703 01:42:39.448616 31319 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "2f3fdb7c089640c0b936a574bc64707e" format_stamp: "Formatted at 2018-07-03 08:42:39 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:42:39.449111 31319 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "2f3fdb7c089640c0b936a574bc64707e" format_stamp: "Formatted at 2018-07-03 08:42:39 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:42:39.456501 31319 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.003s sys 0.003s I0703 01:42:39.457712 31319 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:42:39.458081 31319 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:42:39.461676 31319 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s I0703 01:42:39.461722 31319 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal uuid: "2f3fdb7c089640c0b936a574bc64707e" format_stamp: "Formatted at 2018-07-03 08:42:39 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:42:39.461845 31319 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:42:39.497998 31319 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:42:39.499908 31319 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:42:39.500231 31319 tablet_server_main.cc:90] Starting tablet server... I0703 01:42:39.583714 31319 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.194:39827 I0703 01:42:39.584143 31319 webserver.cc:175] Starting webserver on 127.5.240.194:0 I0703 01:42:39.584189 31319 webserver.cc:186] Document root disabled I0703 01:42:39.584915 31319 webserver.cc:313] Webserver started. Bound to: http://127.5.240.194:47979/ I0703 01:42:39.585455 31319 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb I0703 01:42:39.587196 31319 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:42:39.590255 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 31319 I0703 01:42:39.590837 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.5.240.195:0 --local_ip_for_outbound_sockets=127.5.240.195 --webserver_interface=127.5.240.195 --webserver_port=0 --tserver_master_addrs=127.5.240.254:36104 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0703 01:42:39.598809 31438 heartbeater.cc:346] Connected to a master server at 127.5.240.254:36104 I0703 01:42:39.598960 31438 heartbeater.cc:426] Registering TS with master... I0703 01:42:39.599309 31438 heartbeater.cc:475] Master 127.5.240.254:36104 requested a full tablet report, sending... I0703 01:42:39.600258 31152 ts_manager.cc:98] Registered new tserver with Master: 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194:39827) I0703 01:42:39.601105 31152 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.194:53448 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:42:39.756539 31441 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:42:39.757570 31441 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:42:39.759582 31441 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0703 01:42:39.759647 31441 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:42:39.759753 31441 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.195 I0703 01:42:39.766324 31441 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.195:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.5.240.195 --webserver_port=0 --tserver_master_addrs=127.5.240.254:36104 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.195 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:42:39.773196 31441 tablet_server_main.cc:85] Initializing tablet server... I0703 01:42:39.774732 31441 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 233208us I0703 01:42:39.775274 31441 fs_manager.cc:261] Metadata directory not provided I0703 01:42:39.775343 31441 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:42:39.775970 31441 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:42:39.776034 31441 server_base.cc:433] Attempting to create new FS layout instead I0703 01:42:39.780230 31441 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "de93b58c2d67488dae12285e1806ebac" format_stamp: "Formatted at 2018-07-03 08:42:39 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:42:39.780767 31441 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "de93b58c2d67488dae12285e1806ebac" format_stamp: "Formatted at 2018-07-03 08:42:39 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:42:39.788630 31441 fs_manager.cc:501] Time spent creating directory manager: real 0.008s user 0.001s sys 0.005s I0703 01:42:39.789816 31441 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:42:39.790189 31441 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:42:39.793310 31441 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0703 01:42:39.793361 31441 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal uuid: "de93b58c2d67488dae12285e1806ebac" format_stamp: "Formatted at 2018-07-03 08:42:39 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:42:39.793486 31441 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:42:39.828431 31441 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:42:39.830302 31441 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:42:39.830651 31441 tablet_server_main.cc:90] Starting tablet server... I0703 01:42:39.915055 31441 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.195:59321 I0703 01:42:39.915474 31441 webserver.cc:175] Starting webserver on 127.5.240.195:0 I0703 01:42:39.915520 31441 webserver.cc:186] Document root disabled I0703 01:42:39.916229 31441 webserver.cc:313] Webserver started. Bound to: http://127.5.240.195:55500/ I0703 01:42:39.916776 31441 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb I0703 01:42:39.919184 31441 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:42:39.919859 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 31441 I0703 01:42:39.920485 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.5.240.196:0 --local_ip_for_outbound_sockets=127.5.240.196 --webserver_interface=127.5.240.196 --webserver_port=0 --tserver_master_addrs=127.5.240.254:36104 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0703 01:42:39.932293 31560 heartbeater.cc:346] Connected to a master server at 127.5.240.254:36104 I0703 01:42:39.932432 31560 heartbeater.cc:426] Registering TS with master... I0703 01:42:39.932803 31560 heartbeater.cc:475] Master 127.5.240.254:36104 requested a full tablet report, sending... I0703 01:42:39.933799 31152 ts_manager.cc:98] Registered new tserver with Master: de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321) I0703 01:42:39.934655 31152 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.195:41435 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:42:40.085726 31563 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:42:40.086736 31563 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:42:40.088780 31563 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0703 01:42:40.088843 31563 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:42:40.088945 31563 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.196 I0703 01:42:40.095610 31563 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.196:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb --webserver_interface=127.5.240.196 --webserver_port=0 --tserver_master_addrs=127.5.240.254:36104 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.196 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:42:40.102283 31563 tablet_server_main.cc:85] Initializing tablet server... I0703 01:42:40.103812 31563 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 233708us I0703 01:42:40.104359 31563 fs_manager.cc:261] Metadata directory not provided I0703 01:42:40.104425 31563 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:42:40.104682 31563 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:42:40.104750 31563 server_base.cc:433] Attempting to create new FS layout instead I0703 01:42:40.108809 31563 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/instance: uuid: "d9dc68ace903455d809cb1f715fe36be" format_stamp: "Formatted at 2018-07-03 08:42:40 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:42:40.109346 31563 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal/instance: uuid: "d9dc68ace903455d809cb1f715fe36be" format_stamp: "Formatted at 2018-07-03 08:42:40 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:42:40.116654 31563 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s I0703 01:42:40.117830 31563 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:42:40.118192 31563 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:42:40.121860 31563 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0703 01:42:40.121917 31563 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal uuid: "d9dc68ace903455d809cb1f715fe36be" format_stamp: "Formatted at 2018-07-03 08:42:40 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:42:40.122040 31563 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:42:40.156438 31563 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:42:40.158277 31563 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:42:40.158638 31563 tablet_server_main.cc:90] Starting tablet server... I0703 01:42:40.243361 31563 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.196:56543 I0703 01:42:40.243806 31563 webserver.cc:175] Starting webserver on 127.5.240.196:0 I0703 01:42:40.243852 31563 webserver.cc:186] Document root disabled I0703 01:42:40.244557 31563 webserver.cc:313] Webserver started. Bound to: http://127.5.240.196:39924/ I0703 01:42:40.245091 31563 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb I0703 01:42:40.246829 31563 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:42:40.251726 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 31563 I0703 01:42:40.252492 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data --block_manager=log --rpc_bind_addresses=127.5.240.197:0 --local_ip_for_outbound_sockets=127.5.240.197 --webserver_interface=127.5.240.197 --webserver_port=0 --tserver_master_addrs=127.5.240.254:36104 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0703 01:42:40.260747 31682 heartbeater.cc:346] Connected to a master server at 127.5.240.254:36104 I0703 01:42:40.260891 31682 heartbeater.cc:426] Registering TS with master... I0703 01:42:40.261243 31682 heartbeater.cc:475] Master 127.5.240.254:36104 requested a full tablet report, sending... I0703 01:42:40.262240 31152 ts_manager.cc:98] Registered new tserver with Master: d9dc68ace903455d809cb1f715fe36be (127.5.240.196:56543) I0703 01:42:40.263097 31152 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.196:46359 I0703 01:42:40.287467 31315 heartbeater.cc:467] Master 127.5.240.254:36104 was elected leader, sending a full tablet report... WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:42:40.417114 31685 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:42:40.418119 31685 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:42:40.420187 31685 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0703 01:42:40.420243 31685 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:42:40.420331 31685 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.197 I0703 01:42:40.426951 31685 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.197:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb --webserver_interface=127.5.240.197 --webserver_port=0 --tserver_master_addrs=127.5.240.254:36104 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.197 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:42:40.433567 31685 tablet_server_main.cc:85] Initializing tablet server... I0703 01:42:40.435134 31685 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 233708us I0703 01:42:40.435699 31685 fs_manager.cc:261] Metadata directory not provided I0703 01:42:40.435758 31685 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:42:40.435971 31685 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:42:40.436033 31685 server_base.cc:433] Attempting to create new FS layout instead I0703 01:42:40.440088 31685 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/instance: uuid: "3c1770123dc14bba80e00db5f182b585" format_stamp: "Formatted at 2018-07-03 08:42:40 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:42:40.440650 31685 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal/instance: uuid: "3c1770123dc14bba80e00db5f182b585" format_stamp: "Formatted at 2018-07-03 08:42:40 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:42:40.447938 31685 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.003s sys 0.003s I0703 01:42:40.449120 31685 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:42:40.449496 31685 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:42:40.452651 31685 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s I0703 01:42:40.452702 31685 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal uuid: "3c1770123dc14bba80e00db5f182b585" format_stamp: "Formatted at 2018-07-03 08:42:40 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:42:40.452826 31685 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:42:40.505220 31685 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:42:40.506683 31685 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:42:40.506979 31685 tablet_server_main.cc:90] Starting tablet server... I0703 01:42:40.592948 31685 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.197:44326 I0703 01:42:40.593399 31685 webserver.cc:175] Starting webserver on 127.5.240.197:0 I0703 01:42:40.593443 31685 webserver.cc:186] Document root disabled I0703 01:42:40.594630 31685 webserver.cc:313] Webserver started. Bound to: http://127.5.240.197:41260/ I0703 01:42:40.595146 31685 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb I0703 01:42:40.597059 31685 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:42:40.602918 31438 heartbeater.cc:467] Master 127.5.240.254:36104 was elected leader, sending a full tablet report... I0703 01:42:40.603632 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 31685 I0703 01:42:40.604377 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data --block_manager=log --rpc_bind_addresses=127.5.240.198:0 --local_ip_for_outbound_sockets=127.5.240.198 --webserver_interface=127.5.240.198 --webserver_port=0 --tserver_master_addrs=127.5.240.254:36104 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0703 01:42:40.608852 31804 heartbeater.cc:346] Connected to a master server at 127.5.240.254:36104 I0703 01:42:40.608990 31804 heartbeater.cc:426] Registering TS with master... I0703 01:42:40.609333 31804 heartbeater.cc:475] Master 127.5.240.254:36104 requested a full tablet report, sending... I0703 01:42:40.610172 31152 ts_manager.cc:98] Registered new tserver with Master: 3c1770123dc14bba80e00db5f182b585 (127.5.240.197:44326) I0703 01:42:40.610972 31152 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.197:52557 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:42:40.768378 31807 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:42:40.769417 31807 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:42:40.771528 31807 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0703 01:42:40.771594 31807 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:42:40.771723 31807 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.198 I0703 01:42:40.778692 31807 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.198:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data/info.pb --webserver_interface=127.5.240.198 --webserver_port=0 --tserver_master_addrs=127.5.240.254:36104 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.198 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:42:40.785328 31807 tablet_server_main.cc:85] Initializing tablet server... I0703 01:42:40.786882 31807 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 233708us I0703 01:42:40.787441 31807 fs_manager.cc:261] Metadata directory not provided I0703 01:42:40.787508 31807 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:42:40.787760 31807 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:42:40.787829 31807 server_base.cc:433] Attempting to create new FS layout instead I0703 01:42:40.791985 31807 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data/instance: uuid: "768d8de4b2ba40f180b18194c28be589" format_stamp: "Formatted at 2018-07-03 08:42:40 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:42:40.792491 31807 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/wal/instance: uuid: "768d8de4b2ba40f180b18194c28be589" format_stamp: "Formatted at 2018-07-03 08:42:40 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:42:40.799803 31807 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.004s sys 0.002s I0703 01:42:40.800995 31807 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:42:40.801360 31807 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:42:40.804461 31807 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s I0703 01:42:40.804512 31807 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/wal uuid: "768d8de4b2ba40f180b18194c28be589" format_stamp: "Formatted at 2018-07-03 08:42:40 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:42:40.804664 31807 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:42:40.839552 31807 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:42:40.840931 31807 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:42:40.841223 31807 tablet_server_main.cc:90] Starting tablet server... I0703 01:42:40.926661 31807 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.198:42654 I0703 01:42:40.927055 31807 webserver.cc:175] Starting webserver on 127.5.240.198:0 I0703 01:42:40.927093 31807 webserver.cc:186] Document root disabled I0703 01:42:40.928247 31807 webserver.cc:313] Webserver started. Bound to: http://127.5.240.198:35397/ I0703 01:42:40.928791 31807 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data/info.pb I0703 01:42:40.930642 31807 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:42:40.933364 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 31807 I0703 01:42:40.933982 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data --block_manager=log --rpc_bind_addresses=127.5.240.199:0 --local_ip_for_outbound_sockets=127.5.240.199 --webserver_interface=127.5.240.199 --webserver_port=0 --tserver_master_addrs=127.5.240.254:36104 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0703 01:42:40.936524 31560 heartbeater.cc:467] Master 127.5.240.254:36104 was elected leader, sending a full tablet report... I0703 01:42:40.942934 31926 heartbeater.cc:346] Connected to a master server at 127.5.240.254:36104 I0703 01:42:40.943111 31926 heartbeater.cc:426] Registering TS with master... I0703 01:42:40.943578 31926 heartbeater.cc:475] Master 127.5.240.254:36104 requested a full tablet report, sending... I0703 01:42:40.944566 31152 ts_manager.cc:98] Registered new tserver with Master: 768d8de4b2ba40f180b18194c28be589 (127.5.240.198:42654) I0703 01:42:40.945447 31152 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.198:48241 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:42:41.097863 31929 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:42:41.098862 31929 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:42:41.100873 31929 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0703 01:42:41.100935 31929 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:42:41.101038 31929 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.199 I0703 01:42:41.107803 31929 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.199:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data/info.pb --webserver_interface=127.5.240.199 --webserver_port=0 --tserver_master_addrs=127.5.240.254:36104 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.199 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:42:41.114491 31929 tablet_server_main.cc:85] Initializing tablet server... I0703 01:42:41.116052 31929 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 234208us I0703 01:42:41.116619 31929 fs_manager.cc:261] Metadata directory not provided I0703 01:42:41.116686 31929 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:42:41.117283 31929 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:42:41.117352 31929 server_base.cc:433] Attempting to create new FS layout instead I0703 01:42:41.121460 31929 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data/instance: uuid: "56eaab17f812433a8f7880f0744e1744" format_stamp: "Formatted at 2018-07-03 08:42:41 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:42:41.121992 31929 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/wal/instance: uuid: "56eaab17f812433a8f7880f0744e1744" format_stamp: "Formatted at 2018-07-03 08:42:41 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:42:41.129726 31929 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.003s sys 0.004s I0703 01:42:41.131103 31929 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:42:41.131608 31929 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:42:41.134912 31929 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0703 01:42:41.134963 31929 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/wal uuid: "56eaab17f812433a8f7880f0744e1744" format_stamp: "Formatted at 2018-07-03 08:42:41 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:42:41.135087 31929 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:42:41.171588 31929 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:42:41.173418 31929 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:42:41.173765 31929 tablet_server_main.cc:90] Starting tablet server... I0703 01:42:41.257442 31929 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.199:57714 I0703 01:42:41.257877 31929 webserver.cc:175] Starting webserver on 127.5.240.199:0 I0703 01:42:41.257922 31929 webserver.cc:186] Document root disabled I0703 01:42:41.259083 31929 webserver.cc:313] Webserver started. Bound to: http://127.5.240.199:58133/ I0703 01:42:41.259665 31929 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data/info.pb I0703 01:42:41.262084 31929 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:42:41.264909 31682 heartbeater.cc:467] Master 127.5.240.254:36104 was elected leader, sending a full tablet report... I0703 01:42:41.265538 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 31929 I0703 01:42:41.275012 32048 heartbeater.cc:346] Connected to a master server at 127.5.240.254:36104 I0703 01:42:41.275187 32048 heartbeater.cc:426] Registering TS with master... I0703 01:42:41.275677 32048 heartbeater.cc:475] Master 127.5.240.254:36104 requested a full tablet report, sending... I0703 01:42:41.276676 31152 ts_manager.cc:98] Registered new tserver with Master: 56eaab17f812433a8f7880f0744e1744 (127.5.240.199:57714) I0703 01:42:41.277586 31152 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.199:35813 I0703 01:42:41.278939 6083 external_mini_cluster.cc:490] 7 TS(s) registered with all masters I0703 01:42:41.288725 31152 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.5.240.254:51772: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 2 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0703 01:42:41.311882 31375 tablet_service.cc:759] Processing CreateTablet for tablet 454bab377db147a1a2f8241ecc2241f3 (table=TestTable [id=fc16aaadf7d64bcc93cb3a3e710b1316]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:42:41.312646 31375 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 454bab377db147a1a2f8241ecc2241f3. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:41.313277 31985 tablet_service.cc:759] Processing CreateTablet for tablet 454bab377db147a1a2f8241ecc2241f3 (table=TestTable [id=fc16aaadf7d64bcc93cb3a3e710b1316]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:42:41.313900 31985 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 454bab377db147a1a2f8241ecc2241f3. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:41.322492 31375 ts_tablet_manager.cc:1173] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e: Registered tablet (data state: TABLET_DATA_READY) I0703 01:42:41.323232 32066 ts_tablet_manager.cc:1028] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e: Bootstrapping tablet I0703 01:42:41.323365 31985 ts_tablet_manager.cc:1173] T 454bab377db147a1a2f8241ecc2241f3 P 56eaab17f812433a8f7880f0744e1744: Registered tablet (data state: TABLET_DATA_READY) I0703 01:42:41.323577 32066 tablet_bootstrap.cc:427] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e: Bootstrap starting. I0703 01:42:41.325063 32067 ts_tablet_manager.cc:1028] T 454bab377db147a1a2f8241ecc2241f3 P 56eaab17f812433a8f7880f0744e1744: Bootstrapping tablet I0703 01:42:41.325372 32067 tablet_bootstrap.cc:427] T 454bab377db147a1a2f8241ecc2241f3 P 56eaab17f812433a8f7880f0744e1744: Bootstrap starting. I0703 01:42:41.326345 32066 tablet_bootstrap.cc:578] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e: No blocks or log segments found. Creating new log. I0703 01:42:41.327445 32066 log.cc:525] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e: Log is configured to *not* fsync() on all Append() calls I0703 01:42:41.328982 32067 tablet_bootstrap.cc:578] T 454bab377db147a1a2f8241ecc2241f3 P 56eaab17f812433a8f7880f0744e1744: No blocks or log segments found. Creating new log. I0703 01:42:41.329833 32067 log.cc:525] T 454bab377db147a1a2f8241ecc2241f3 P 56eaab17f812433a8f7880f0744e1744: Log is configured to *not* fsync() on all Append() calls I0703 01:42:41.330296 32066 tablet_bootstrap.cc:427] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e: No bootstrap required, opened a new log I0703 01:42:41.330472 32066 ts_tablet_manager.cc:1045] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.001s I0703 01:42:41.331398 32067 tablet_bootstrap.cc:427] T 454bab377db147a1a2f8241ecc2241f3 P 56eaab17f812433a8f7880f0744e1744: No bootstrap required, opened a new log I0703 01:42:41.331652 32067 ts_tablet_manager.cc:1045] T 454bab377db147a1a2f8241ecc2241f3 P 56eaab17f812433a8f7880f0744e1744: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s I0703 01:42:41.339133 32066 raft_consensus.cc:325] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } I0703 01:42:41.339532 32066 raft_consensus.cc:351] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:42:41.339651 32066 raft_consensus.cc:628] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f3fdb7c089640c0b936a574bc64707e, State: Initialized, Role: FOLLOWER I0703 01:42:41.340219 32066 consensus_queue.cc:227] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } I0703 01:42:41.340323 32067 raft_consensus.cc:325] T 454bab377db147a1a2f8241ecc2241f3 P 56eaab17f812433a8f7880f0744e1744 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } I0703 01:42:41.340778 32067 raft_consensus.cc:351] T 454bab377db147a1a2f8241ecc2241f3 P 56eaab17f812433a8f7880f0744e1744 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:42:41.340862 32067 raft_consensus.cc:628] T 454bab377db147a1a2f8241ecc2241f3 P 56eaab17f812433a8f7880f0744e1744 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56eaab17f812433a8f7880f0744e1744, State: Initialized, Role: FOLLOWER I0703 01:42:41.341717 32067 consensus_queue.cc:227] T 454bab377db147a1a2f8241ecc2241f3 P 56eaab17f812433a8f7880f0744e1744 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } I0703 01:42:41.342115 32066 ts_tablet_manager.cc:1073] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s I0703 01:42:41.343070 32048 heartbeater.cc:467] Master 127.5.240.254:36104 was elected leader, sending a full tablet report... I0703 01:42:41.344036 32067 ts_tablet_manager.cc:1073] T 454bab377db147a1a2f8241ecc2241f3 P 56eaab17f812433a8f7880f0744e1744: Time spent starting tablet: real 0.012s user 0.008s sys 0.003s I0703 01:42:41.466768 32070 raft_consensus.cc:457] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0703 01:42:41.466861 32070 raft_consensus.cc:2730] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e: Snoozing failure detection for 2.126s (starting election) I0703 01:42:41.466949 32070 raft_consensus.cc:479] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } I0703 01:42:41.467973 32070 leader_election.cc:243] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 56eaab17f812433a8f7880f0744e1744 (127.5.240.199:57714) I0703 01:42:41.474054 32005 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "454bab377db147a1a2f8241ecc2241f3" candidate_uuid: "2f3fdb7c089640c0b936a574bc64707e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56eaab17f812433a8f7880f0744e1744" is_pre_election: true I0703 01:42:41.474701 32005 raft_consensus.cc:2730] T 454bab377db147a1a2f8241ecc2241f3 P 56eaab17f812433a8f7880f0744e1744: Snoozing failure detection for 1.881s (vote granted) I0703 01:42:41.474792 32005 raft_consensus.cc:2243] T 454bab377db147a1a2f8241ecc2241f3 P 56eaab17f812433a8f7880f0744e1744 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f3fdb7c089640c0b936a574bc64707e in term 0. I0703 01:42:41.475344 31328 leader_election.cc:379] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e [CANDIDATE]: Term 1 pre-election: Vote granted by peer 56eaab17f812433a8f7880f0744e1744 (127.5.240.199:57714) I0703 01:42:41.475409 31328 leader_election.cc:270] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0703 01:42:41.475666 32070 raft_consensus.cc:2730] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e: Snoozing failure detection for 2.166s (election complete) I0703 01:42:41.475754 32070 raft_consensus.cc:2537] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Leader pre-election won for term 1 I0703 01:42:41.475819 32070 raft_consensus.cc:457] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0703 01:42:41.475872 32070 raft_consensus.cc:2730] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e: Snoozing failure detection for 2.239s (starting election) I0703 01:42:41.475934 32070 raft_consensus.cc:2782] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Advancing to term 1 I0703 01:42:41.477720 32070 raft_consensus.cc:479] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } I0703 01:42:41.477989 32070 leader_election.cc:243] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e [CANDIDATE]: Term 1 election: Requesting vote from peer 56eaab17f812433a8f7880f0744e1744 (127.5.240.199:57714) I0703 01:42:41.478570 32005 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "454bab377db147a1a2f8241ecc2241f3" candidate_uuid: "2f3fdb7c089640c0b936a574bc64707e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56eaab17f812433a8f7880f0744e1744" I0703 01:42:41.478682 32005 raft_consensus.cc:2782] T 454bab377db147a1a2f8241ecc2241f3 P 56eaab17f812433a8f7880f0744e1744 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:42:41.478771 32005 raft_consensus.cc:2730] T 454bab377db147a1a2f8241ecc2241f3 P 56eaab17f812433a8f7880f0744e1744: Snoozing failure detection for 1.660s (vote granted) I0703 01:42:41.480126 32005 raft_consensus.cc:2243] T 454bab377db147a1a2f8241ecc2241f3 P 56eaab17f812433a8f7880f0744e1744 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f3fdb7c089640c0b936a574bc64707e in term 1. I0703 01:42:41.480465 31328 leader_election.cc:379] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e [CANDIDATE]: Term 1 election: Vote granted by peer 56eaab17f812433a8f7880f0744e1744 (127.5.240.199:57714) I0703 01:42:41.480515 31328 leader_election.cc:270] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:42:41.480666 32070 raft_consensus.cc:2730] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e: Snoozing failure detection for 2.092s (election complete) I0703 01:42:41.480737 32070 raft_consensus.cc:2537] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:42:41.481560 32070 raft_consensus.cc:593] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e [term 1 LEADER]: Becoming Leader. State: Replica: 2f3fdb7c089640c0b936a574bc64707e, State: Running, Role: LEADER I0703 01:42:41.481905 32070 consensus_queue.cc:204] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } I0703 01:42:41.485199 31152 catalog_manager.cc:3798] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e reported cstate change: term changed from 0 to 1, leader changed from to 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194). New cstate: current_term: 1 leader_uuid: "2f3fdb7c089640c0b936a574bc64707e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } } W0703 01:42:41.514904 32049 tablet.cc:1678] T 454bab377db147a1a2f8241ecc2241f3 P 56eaab17f812433a8f7880f0744e1744: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:42:41.534858 6083 external_mini_cluster.cc:490] 7 TS(s) registered with all masters I0703 01:42:41.535883 6083 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 1288ca0989104bbea36e22f38c374218 to finish bootstrapping I0703 01:42:41.542258 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 2f3fdb7c089640c0b936a574bc64707e to finish bootstrapping I0703 01:42:41.546591 6083 ts_itest-base.cc:233] Waiting for 0 tablets on tserver de93b58c2d67488dae12285e1806ebac to finish bootstrapping I0703 01:42:41.552652 6083 ts_itest-base.cc:233] Waiting for 0 tablets on tserver d9dc68ace903455d809cb1f715fe36be to finish bootstrapping I0703 01:42:41.558548 6083 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 3c1770123dc14bba80e00db5f182b585 to finish bootstrapping I0703 01:42:41.564785 6083 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 768d8de4b2ba40f180b18194c28be589 to finish bootstrapping I0703 01:42:41.570574 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 56eaab17f812433a8f7880f0744e1744 to finish bootstrapping I0703 01:42:41.575733 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 31563 I0703 01:42:41.576355 31679 generic_service.cc:169] Checking for leaks (request via RPC) W0703 01:42:41.588672 31439 tablet.cc:1678] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:42:41.612782 31804 heartbeater.cc:467] Master 127.5.240.254:36104 was elected leader, sending a full tablet report... I0703 01:42:41.614603 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 31563 W0703 01:42:41.622452 31133 connection.cc:512] server connection from 127.5.240.196:46359 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.196:46359): Connection reset by peer (error 104) I0703 01:42:41.622522 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 31685 I0703 01:42:41.623421 31801 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:42:41.661412 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 31685 I0703 01:42:41.669342 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 31807 W0703 01:42:41.669373 31133 connection.cc:512] server connection from 127.5.240.197:52557 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.197:52557): Connection reset by peer (error 104) I0703 01:42:41.670171 31923 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:42:41.708746 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 31807 W0703 01:42:41.716048 31133 connection.cc:512] server connection from 127.5.240.198:48241 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.198:48241): Connection reset by peer (error 104) I0703 01:42:41.716372 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 31929 I0703 01:42:41.717083 32045 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:42:41.759697 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 31929 W0703 01:42:41.767664 31328 connection.cc:512] client connection to 127.5.240.199:57714 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.199:57714): Connection reset by peer (error 104) W0703 01:42:41.767747 31135 connection.cc:512] server connection from 127.5.240.199:35813 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.199:35813): Connection reset by peer (error 104) W0703 01:42:41.926949 31328 consensus_peers.cc:433] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 56eaab17f812433a8f7880f0744e1744 (127.5.240.199:57714): Couldn't send request to peer 56eaab17f812433a8f7880f0744e1744 for tablet 454bab377db147a1a2f8241ecc2241f3. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.199:57714: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. W0703 01:42:42.335829 31328 consensus_peers.cc:433] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 56eaab17f812433a8f7880f0744e1744 (127.5.240.199:57714): Couldn't send request to peer 56eaab17f812433a8f7880f0744e1744 for tablet 454bab377db147a1a2f8241ecc2241f3. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.199:57714: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. W0703 01:42:42.952215 31328 consensus_peers.cc:433] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 56eaab17f812433a8f7880f0744e1744 (127.5.240.199:57714): Couldn't send request to peer 56eaab17f812433a8f7880f0744e1744 for tablet 454bab377db147a1a2f8241ecc2241f3. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.199:57714: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times. W0703 01:42:43.510578 31328 consensus_peers.cc:433] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 56eaab17f812433a8f7880f0744e1744 (127.5.240.199:57714): Couldn't send request to peer 56eaab17f812433a8f7880f0744e1744 for tablet 454bab377db147a1a2f8241ecc2241f3. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.199:57714: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 4 times. W0703 01:42:44.046825 31328 consensus_peers.cc:433] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 56eaab17f812433a8f7880f0744e1744 (127.5.240.199:57714): Couldn't send request to peer 56eaab17f812433a8f7880f0744e1744 for tablet 454bab377db147a1a2f8241ecc2241f3. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.199:57714: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 5 times. W0703 01:42:44.656246 31328 consensus_peers.cc:433] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 56eaab17f812433a8f7880f0744e1744 (127.5.240.199:57714): Couldn't send request to peer 56eaab17f812433a8f7880f0744e1744 for tablet 454bab377db147a1a2f8241ecc2241f3. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.199:57714: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 6 times. W0703 01:42:45.126314 31328 consensus_peers.cc:433] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 56eaab17f812433a8f7880f0744e1744 (127.5.240.199:57714): Couldn't send request to peer 56eaab17f812433a8f7880f0744e1744 for tablet 454bab377db147a1a2f8241ecc2241f3. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.199:57714: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 7 times. W0703 01:42:45.514909 31328 consensus_peers.cc:433] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 56eaab17f812433a8f7880f0744e1744 (127.5.240.199:57714): Couldn't send request to peer 56eaab17f812433a8f7880f0744e1744 for tablet 454bab377db147a1a2f8241ecc2241f3. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.199:57714: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 8 times. I0703 01:42:45.518712 31152 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.5.240.254:51772: name: "rebalance_test_table_0" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 2 split_rows_range_bounds { } partition_schema { hash_bucket_schemas { columns { name: "key" } num_buckets: 3 seed: 0 } } I0703 01:42:45.527297 31375 tablet_service.cc:759] Processing CreateTablet for tablet 44f06af6e8ef46d78419aea845bb38a7 (table=rebalance_test_table_0 [id=9d68c0e8280a4d18af7d08099d85f9a3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I0703 01:42:45.527644 31375 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 44f06af6e8ef46d78419aea845bb38a7. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:45.531872 31375 ts_tablet_manager.cc:1173] T 44f06af6e8ef46d78419aea845bb38a7 P 2f3fdb7c089640c0b936a574bc64707e: Registered tablet (data state: TABLET_DATA_READY) I0703 01:42:45.534243 32100 ts_tablet_manager.cc:1028] T 44f06af6e8ef46d78419aea845bb38a7 P 2f3fdb7c089640c0b936a574bc64707e: Bootstrapping tablet I0703 01:42:45.534472 32100 tablet_bootstrap.cc:427] T 44f06af6e8ef46d78419aea845bb38a7 P 2f3fdb7c089640c0b936a574bc64707e: Bootstrap starting. I0703 01:42:45.535434 32100 tablet_bootstrap.cc:578] T 44f06af6e8ef46d78419aea845bb38a7 P 2f3fdb7c089640c0b936a574bc64707e: No blocks or log segments found. Creating new log. I0703 01:42:45.536947 32100 tablet_bootstrap.cc:427] T 44f06af6e8ef46d78419aea845bb38a7 P 2f3fdb7c089640c0b936a574bc64707e: No bootstrap required, opened a new log I0703 01:42:45.537078 32100 ts_tablet_manager.cc:1045] T 44f06af6e8ef46d78419aea845bb38a7 P 2f3fdb7c089640c0b936a574bc64707e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s I0703 01:42:45.537825 32100 raft_consensus.cc:325] T 44f06af6e8ef46d78419aea845bb38a7 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } I0703 01:42:45.538038 32100 raft_consensus.cc:351] T 44f06af6e8ef46d78419aea845bb38a7 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:42:45.538116 32100 raft_consensus.cc:628] T 44f06af6e8ef46d78419aea845bb38a7 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f3fdb7c089640c0b936a574bc64707e, State: Initialized, Role: FOLLOWER I0703 01:42:45.538314 32100 consensus_queue.cc:227] T 44f06af6e8ef46d78419aea845bb38a7 P 2f3fdb7c089640c0b936a574bc64707e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } I0703 01:42:45.538874 32100 ts_tablet_manager.cc:1073] T 44f06af6e8ef46d78419aea845bb38a7 P 2f3fdb7c089640c0b936a574bc64707e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s I0703 01:42:45.545367 31252 tablet_service.cc:759] Processing CreateTablet for tablet b2c8b58f935a4d00b5cf3e2cedec4d81 (table=rebalance_test_table_0 [id=9d68c0e8280a4d18af7d08099d85f9a3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I0703 01:42:45.545476 31251 tablet_service.cc:759] Processing CreateTablet for tablet 19c33e54ee9e476796de7de42ba495b6 (table=rebalance_test_table_0 [id=9d68c0e8280a4d18af7d08099d85f9a3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I0703 01:42:45.545513 31496 tablet_service.cc:759] Processing CreateTablet for tablet 19c33e54ee9e476796de7de42ba495b6 (table=rebalance_test_table_0 [id=9d68c0e8280a4d18af7d08099d85f9a3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I0703 01:42:45.545939 31497 tablet_service.cc:759] Processing CreateTablet for tablet b2c8b58f935a4d00b5cf3e2cedec4d81 (table=rebalance_test_table_0 [id=9d68c0e8280a4d18af7d08099d85f9a3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I0703 01:42:45.546180 31251 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 19c33e54ee9e476796de7de42ba495b6. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:45.546311 31496 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 19c33e54ee9e476796de7de42ba495b6. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:45.546538 31252 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet b2c8b58f935a4d00b5cf3e2cedec4d81. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:45.545313 31250 tablet_service.cc:759] Processing CreateTablet for tablet 44f06af6e8ef46d78419aea845bb38a7 (table=rebalance_test_table_0 [id=9d68c0e8280a4d18af7d08099d85f9a3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I0703 01:42:45.547605 31250 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 44f06af6e8ef46d78419aea845bb38a7. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:45.547816 31497 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet b2c8b58f935a4d00b5cf3e2cedec4d81. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:45.562074 31496 ts_tablet_manager.cc:1173] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac: Registered tablet (data state: TABLET_DATA_READY) I0703 01:42:45.562242 31497 ts_tablet_manager.cc:1173] T b2c8b58f935a4d00b5cf3e2cedec4d81 P de93b58c2d67488dae12285e1806ebac: Registered tablet (data state: TABLET_DATA_READY) I0703 01:42:45.563727 31250 ts_tablet_manager.cc:1173] T 44f06af6e8ef46d78419aea845bb38a7 P 1288ca0989104bbea36e22f38c374218: Registered tablet (data state: TABLET_DATA_READY) I0703 01:42:45.564152 32102 ts_tablet_manager.cc:1028] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac: Bootstrapping tablet I0703 01:42:45.564455 32102 tablet_bootstrap.cc:427] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac: Bootstrap starting. I0703 01:42:45.564707 32103 ts_tablet_manager.cc:1028] T 44f06af6e8ef46d78419aea845bb38a7 P 1288ca0989104bbea36e22f38c374218: Bootstrapping tablet I0703 01:42:45.565012 31251 ts_tablet_manager.cc:1173] T 19c33e54ee9e476796de7de42ba495b6 P 1288ca0989104bbea36e22f38c374218: Registered tablet (data state: TABLET_DATA_READY) I0703 01:42:45.565132 32103 tablet_bootstrap.cc:427] T 44f06af6e8ef46d78419aea845bb38a7 P 1288ca0989104bbea36e22f38c374218: Bootstrap starting. I0703 01:42:45.565763 31252 ts_tablet_manager.cc:1173] T b2c8b58f935a4d00b5cf3e2cedec4d81 P 1288ca0989104bbea36e22f38c374218: Registered tablet (data state: TABLET_DATA_READY) I0703 01:42:45.568186 32102 tablet_bootstrap.cc:578] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac: No blocks or log segments found. Creating new log. I0703 01:42:45.568243 32103 tablet_bootstrap.cc:578] T 44f06af6e8ef46d78419aea845bb38a7 P 1288ca0989104bbea36e22f38c374218: No blocks or log segments found. Creating new log. I0703 01:42:45.569259 32102 log.cc:525] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac: Log is configured to *not* fsync() on all Append() calls I0703 01:42:45.569226 32103 log.cc:525] T 44f06af6e8ef46d78419aea845bb38a7 P 1288ca0989104bbea36e22f38c374218: Log is configured to *not* fsync() on all Append() calls I0703 01:42:45.572201 32102 tablet_bootstrap.cc:427] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac: No bootstrap required, opened a new log I0703 01:42:45.572397 32103 tablet_bootstrap.cc:427] T 44f06af6e8ef46d78419aea845bb38a7 P 1288ca0989104bbea36e22f38c374218: No bootstrap required, opened a new log I0703 01:42:45.572418 32102 ts_tablet_manager.cc:1045] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.002s I0703 01:42:45.572584 32103 ts_tablet_manager.cc:1045] T 44f06af6e8ef46d78419aea845bb38a7 P 1288ca0989104bbea36e22f38c374218: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s I0703 01:42:45.581321 32102 raft_consensus.cc:325] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } I0703 01:42:45.582211 32102 raft_consensus.cc:351] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:42:45.581748 32103 raft_consensus.cc:325] T 44f06af6e8ef46d78419aea845bb38a7 P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } I0703 01:42:45.582304 32103 raft_consensus.cc:351] T 44f06af6e8ef46d78419aea845bb38a7 P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:42:45.582286 32102 raft_consensus.cc:628] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de93b58c2d67488dae12285e1806ebac, State: Initialized, Role: FOLLOWER I0703 01:42:45.582372 32103 raft_consensus.cc:628] T 44f06af6e8ef46d78419aea845bb38a7 P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1288ca0989104bbea36e22f38c374218, State: Initialized, Role: FOLLOWER I0703 01:42:45.582826 32103 consensus_queue.cc:227] T 44f06af6e8ef46d78419aea845bb38a7 P 1288ca0989104bbea36e22f38c374218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } I0703 01:42:45.582826 32102 consensus_queue.cc:227] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } I0703 01:42:45.585592 32102 ts_tablet_manager.cc:1073] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s I0703 01:42:45.585685 32103 ts_tablet_manager.cc:1073] T 44f06af6e8ef46d78419aea845bb38a7 P 1288ca0989104bbea36e22f38c374218: Time spent starting tablet: real 0.013s user 0.009s sys 0.003s I0703 01:42:45.585867 32102 ts_tablet_manager.cc:1028] T b2c8b58f935a4d00b5cf3e2cedec4d81 P de93b58c2d67488dae12285e1806ebac: Bootstrapping tablet I0703 01:42:45.585991 32102 tablet_bootstrap.cc:427] T b2c8b58f935a4d00b5cf3e2cedec4d81 P de93b58c2d67488dae12285e1806ebac: Bootstrap starting. I0703 01:42:45.587019 32103 ts_tablet_manager.cc:1028] T 19c33e54ee9e476796de7de42ba495b6 P 1288ca0989104bbea36e22f38c374218: Bootstrapping tablet I0703 01:42:45.587307 32103 tablet_bootstrap.cc:427] T 19c33e54ee9e476796de7de42ba495b6 P 1288ca0989104bbea36e22f38c374218: Bootstrap starting. I0703 01:42:45.588691 32102 tablet_bootstrap.cc:578] T b2c8b58f935a4d00b5cf3e2cedec4d81 P de93b58c2d67488dae12285e1806ebac: No blocks or log segments found. Creating new log. I0703 01:42:45.588692 32103 tablet_bootstrap.cc:578] T 19c33e54ee9e476796de7de42ba495b6 P 1288ca0989104bbea36e22f38c374218: No blocks or log segments found. Creating new log. I0703 01:42:45.589244 32090 raft_consensus.cc:457] T 44f06af6e8ef46d78419aea845bb38a7 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0703 01:42:45.589339 32090 raft_consensus.cc:2730] T 44f06af6e8ef46d78419aea845bb38a7 P 2f3fdb7c089640c0b936a574bc64707e: Snoozing failure detection for 1.958s (starting election) I0703 01:42:45.589428 32090 raft_consensus.cc:479] T 44f06af6e8ef46d78419aea845bb38a7 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } I0703 01:42:45.589772 32090 leader_election.cc:243] T 44f06af6e8ef46d78419aea845bb38a7 P 2f3fdb7c089640c0b936a574bc64707e [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521) I0703 01:42:45.595372 32102 tablet_bootstrap.cc:427] T b2c8b58f935a4d00b5cf3e2cedec4d81 P de93b58c2d67488dae12285e1806ebac: No bootstrap required, opened a new log I0703 01:42:45.595480 32102 ts_tablet_manager.cc:1045] T b2c8b58f935a4d00b5cf3e2cedec4d81 P de93b58c2d67488dae12285e1806ebac: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.006s I0703 01:42:45.595358 31272 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "44f06af6e8ef46d78419aea845bb38a7" candidate_uuid: "2f3fdb7c089640c0b936a574bc64707e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1288ca0989104bbea36e22f38c374218" is_pre_election: true I0703 01:42:45.595769 31272 raft_consensus.cc:2730] T 44f06af6e8ef46d78419aea845bb38a7 P 1288ca0989104bbea36e22f38c374218: Snoozing failure detection for 1.593s (vote granted) I0703 01:42:45.595907 32103 tablet_bootstrap.cc:427] T 19c33e54ee9e476796de7de42ba495b6 P 1288ca0989104bbea36e22f38c374218: No bootstrap required, opened a new log I0703 01:42:45.596035 32103 ts_tablet_manager.cc:1045] T 19c33e54ee9e476796de7de42ba495b6 P 1288ca0989104bbea36e22f38c374218: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.006s I0703 01:42:45.595991 31272 raft_consensus.cc:2243] T 44f06af6e8ef46d78419aea845bb38a7 P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f3fdb7c089640c0b936a574bc64707e in term 0. I0703 01:42:45.596160 32102 raft_consensus.cc:325] T b2c8b58f935a4d00b5cf3e2cedec4d81 P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } I0703 01:42:45.596287 32102 raft_consensus.cc:351] T b2c8b58f935a4d00b5cf3e2cedec4d81 P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:42:45.596340 32102 raft_consensus.cc:628] T b2c8b58f935a4d00b5cf3e2cedec4d81 P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de93b58c2d67488dae12285e1806ebac, State: Initialized, Role: FOLLOWER I0703 01:42:45.596513 32102 consensus_queue.cc:227] T b2c8b58f935a4d00b5cf3e2cedec4d81 P de93b58c2d67488dae12285e1806ebac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } I0703 01:42:45.596834 31327 leader_election.cc:379] T 44f06af6e8ef46d78419aea845bb38a7 P 2f3fdb7c089640c0b936a574bc64707e [CANDIDATE]: Term 1 pre-election: Vote granted by peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521) I0703 01:42:45.596897 31327 leader_election.cc:270] T 44f06af6e8ef46d78419aea845bb38a7 P 2f3fdb7c089640c0b936a574bc64707e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0703 01:42:45.597086 32090 raft_consensus.cc:2730] T 44f06af6e8ef46d78419aea845bb38a7 P 2f3fdb7c089640c0b936a574bc64707e: Snoozing failure detection for 1.610s (election complete) I0703 01:42:45.597152 32090 raft_consensus.cc:2537] T 44f06af6e8ef46d78419aea845bb38a7 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Leader pre-election won for term 1 I0703 01:42:45.597208 32090 raft_consensus.cc:457] T 44f06af6e8ef46d78419aea845bb38a7 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0703 01:42:45.597262 32090 raft_consensus.cc:2730] T 44f06af6e8ef46d78419aea845bb38a7 P 2f3fdb7c089640c0b936a574bc64707e: Snoozing failure detection for 1.875s (starting election) I0703 01:42:45.597319 32090 raft_consensus.cc:2782] T 44f06af6e8ef46d78419aea845bb38a7 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Advancing to term 1 I0703 01:42:45.597429 32102 ts_tablet_manager.cc:1073] T b2c8b58f935a4d00b5cf3e2cedec4d81 P de93b58c2d67488dae12285e1806ebac: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s I0703 01:42:45.597436 32103 raft_consensus.cc:325] T 19c33e54ee9e476796de7de42ba495b6 P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } I0703 01:42:45.597620 32103 raft_consensus.cc:351] T 19c33e54ee9e476796de7de42ba495b6 P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:42:45.597673 32103 raft_consensus.cc:628] T 19c33e54ee9e476796de7de42ba495b6 P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1288ca0989104bbea36e22f38c374218, State: Initialized, Role: FOLLOWER I0703 01:42:45.597853 32103 consensus_queue.cc:227] T 19c33e54ee9e476796de7de42ba495b6 P 1288ca0989104bbea36e22f38c374218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } I0703 01:42:45.598276 32103 ts_tablet_manager.cc:1073] T 19c33e54ee9e476796de7de42ba495b6 P 1288ca0989104bbea36e22f38c374218: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s I0703 01:42:45.598387 32103 ts_tablet_manager.cc:1028] T b2c8b58f935a4d00b5cf3e2cedec4d81 P 1288ca0989104bbea36e22f38c374218: Bootstrapping tablet I0703 01:42:45.598502 32103 tablet_bootstrap.cc:427] T b2c8b58f935a4d00b5cf3e2cedec4d81 P 1288ca0989104bbea36e22f38c374218: Bootstrap starting. I0703 01:42:45.599274 32090 raft_consensus.cc:479] T 44f06af6e8ef46d78419aea845bb38a7 P 2f3fdb7c089640c0b936a574bc64707e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } I0703 01:42:45.599578 32090 leader_election.cc:243] T 44f06af6e8ef46d78419aea845bb38a7 P 2f3fdb7c089640c0b936a574bc64707e [CANDIDATE]: Term 1 election: Requesting vote from peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521) I0703 01:42:45.599951 32103 tablet_bootstrap.cc:578] T b2c8b58f935a4d00b5cf3e2cedec4d81 P 1288ca0989104bbea36e22f38c374218: No blocks or log segments found. Creating new log. I0703 01:42:45.600349 31272 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "44f06af6e8ef46d78419aea845bb38a7" candidate_uuid: "2f3fdb7c089640c0b936a574bc64707e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1288ca0989104bbea36e22f38c374218" I0703 01:42:45.600605 31272 raft_consensus.cc:2782] T 44f06af6e8ef46d78419aea845bb38a7 P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:42:45.600778 31272 raft_consensus.cc:2730] T 44f06af6e8ef46d78419aea845bb38a7 P 1288ca0989104bbea36e22f38c374218: Snoozing failure detection for 2.138s (vote granted) I0703 01:42:45.604136 31272 raft_consensus.cc:2243] T 44f06af6e8ef46d78419aea845bb38a7 P 1288ca0989104bbea36e22f38c374218 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f3fdb7c089640c0b936a574bc64707e in term 1. I0703 01:42:45.604562 31327 leader_election.cc:379] T 44f06af6e8ef46d78419aea845bb38a7 P 2f3fdb7c089640c0b936a574bc64707e [CANDIDATE]: Term 1 election: Vote granted by peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521) I0703 01:42:45.604617 31327 leader_election.cc:270] T 44f06af6e8ef46d78419aea845bb38a7 P 2f3fdb7c089640c0b936a574bc64707e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:42:45.604789 32103 tablet_bootstrap.cc:427] T b2c8b58f935a4d00b5cf3e2cedec4d81 P 1288ca0989104bbea36e22f38c374218: No bootstrap required, opened a new log I0703 01:42:45.604787 32090 raft_consensus.cc:2730] T 44f06af6e8ef46d78419aea845bb38a7 P 2f3fdb7c089640c0b936a574bc64707e: Snoozing failure detection for 1.884s (election complete) I0703 01:42:45.604856 32090 raft_consensus.cc:2537] T 44f06af6e8ef46d78419aea845bb38a7 P 2f3fdb7c089640c0b936a574bc64707e [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:42:45.604889 32103 ts_tablet_manager.cc:1045] T b2c8b58f935a4d00b5cf3e2cedec4d81 P 1288ca0989104bbea36e22f38c374218: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s I0703 01:42:45.605250 32090 raft_consensus.cc:593] T 44f06af6e8ef46d78419aea845bb38a7 P 2f3fdb7c089640c0b936a574bc64707e [term 1 LEADER]: Becoming Leader. State: Replica: 2f3fdb7c089640c0b936a574bc64707e, State: Running, Role: LEADER I0703 01:42:45.605484 32103 raft_consensus.cc:325] T b2c8b58f935a4d00b5cf3e2cedec4d81 P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } I0703 01:42:45.605659 32103 raft_consensus.cc:351] T b2c8b58f935a4d00b5cf3e2cedec4d81 P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:42:45.605571 32090 consensus_queue.cc:204] T 44f06af6e8ef46d78419aea845bb38a7 P 2f3fdb7c089640c0b936a574bc64707e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } I0703 01:42:45.605717 32103 raft_consensus.cc:628] T b2c8b58f935a4d00b5cf3e2cedec4d81 P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1288ca0989104bbea36e22f38c374218, State: Initialized, Role: FOLLOWER I0703 01:42:45.605890 32103 consensus_queue.cc:227] T b2c8b58f935a4d00b5cf3e2cedec4d81 P 1288ca0989104bbea36e22f38c374218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } I0703 01:42:45.606515 32103 ts_tablet_manager.cc:1073] T b2c8b58f935a4d00b5cf3e2cedec4d81 P 1288ca0989104bbea36e22f38c374218: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s I0703 01:42:45.607491 31151 catalog_manager.cc:3798] T 44f06af6e8ef46d78419aea845bb38a7 P 2f3fdb7c089640c0b936a574bc64707e reported cstate change: term changed from 0 to 1, leader changed from to 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194). New cstate: current_term: 1 leader_uuid: "2f3fdb7c089640c0b936a574bc64707e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } } I0703 01:42:45.658157 32106 raft_consensus.cc:457] T b2c8b58f935a4d00b5cf3e2cedec4d81 P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0703 01:42:45.658259 32106 raft_consensus.cc:2730] T b2c8b58f935a4d00b5cf3e2cedec4d81 P de93b58c2d67488dae12285e1806ebac: Snoozing failure detection for 2.169s (starting election) I0703 01:42:45.658360 32106 raft_consensus.cc:479] T b2c8b58f935a4d00b5cf3e2cedec4d81 P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } I0703 01:42:45.658993 32106 leader_election.cc:243] T b2c8b58f935a4d00b5cf3e2cedec4d81 P de93b58c2d67488dae12285e1806ebac [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521) I0703 01:42:45.665119 31272 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "b2c8b58f935a4d00b5cf3e2cedec4d81" candidate_uuid: "de93b58c2d67488dae12285e1806ebac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1288ca0989104bbea36e22f38c374218" is_pre_election: true I0703 01:42:45.665236 31272 raft_consensus.cc:2730] T b2c8b58f935a4d00b5cf3e2cedec4d81 P 1288ca0989104bbea36e22f38c374218: Snoozing failure detection for 1.533s (vote granted) I0703 01:42:45.665295 31272 raft_consensus.cc:2243] T b2c8b58f935a4d00b5cf3e2cedec4d81 P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de93b58c2d67488dae12285e1806ebac in term 0. I0703 01:42:45.665740 31449 leader_election.cc:379] T b2c8b58f935a4d00b5cf3e2cedec4d81 P de93b58c2d67488dae12285e1806ebac [CANDIDATE]: Term 1 pre-election: Vote granted by peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521) I0703 01:42:45.665799 31449 leader_election.cc:270] T b2c8b58f935a4d00b5cf3e2cedec4d81 P de93b58c2d67488dae12285e1806ebac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0703 01:42:45.666041 32106 raft_consensus.cc:2730] T b2c8b58f935a4d00b5cf3e2cedec4d81 P de93b58c2d67488dae12285e1806ebac: Snoozing failure detection for 1.531s (election complete) I0703 01:42:45.666138 32106 raft_consensus.cc:2537] T b2c8b58f935a4d00b5cf3e2cedec4d81 P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Leader pre-election won for term 1 I0703 01:42:45.666203 32106 raft_consensus.cc:457] T b2c8b58f935a4d00b5cf3e2cedec4d81 P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0703 01:42:45.666258 32106 raft_consensus.cc:2730] T b2c8b58f935a4d00b5cf3e2cedec4d81 P de93b58c2d67488dae12285e1806ebac: Snoozing failure detection for 1.962s (starting election) I0703 01:42:45.666326 32106 raft_consensus.cc:2782] T b2c8b58f935a4d00b5cf3e2cedec4d81 P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Advancing to term 1 I0703 01:42:45.667804 32106 raft_consensus.cc:479] T b2c8b58f935a4d00b5cf3e2cedec4d81 P de93b58c2d67488dae12285e1806ebac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } I0703 01:42:45.668123 32106 leader_election.cc:243] T b2c8b58f935a4d00b5cf3e2cedec4d81 P de93b58c2d67488dae12285e1806ebac [CANDIDATE]: Term 1 election: Requesting vote from peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521) I0703 01:42:45.668746 31272 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "b2c8b58f935a4d00b5cf3e2cedec4d81" candidate_uuid: "de93b58c2d67488dae12285e1806ebac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1288ca0989104bbea36e22f38c374218" I0703 01:42:45.668853 31272 raft_consensus.cc:2782] T b2c8b58f935a4d00b5cf3e2cedec4d81 P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:42:45.668922 31272 raft_consensus.cc:2730] T b2c8b58f935a4d00b5cf3e2cedec4d81 P 1288ca0989104bbea36e22f38c374218: Snoozing failure detection for 1.617s (vote granted) I0703 01:42:45.670446 31272 raft_consensus.cc:2243] T b2c8b58f935a4d00b5cf3e2cedec4d81 P 1288ca0989104bbea36e22f38c374218 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de93b58c2d67488dae12285e1806ebac in term 1. I0703 01:42:45.670821 31449 leader_election.cc:379] T b2c8b58f935a4d00b5cf3e2cedec4d81 P de93b58c2d67488dae12285e1806ebac [CANDIDATE]: Term 1 election: Vote granted by peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521) I0703 01:42:45.670871 31449 leader_election.cc:270] T b2c8b58f935a4d00b5cf3e2cedec4d81 P de93b58c2d67488dae12285e1806ebac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:42:45.671017 32106 raft_consensus.cc:2730] T b2c8b58f935a4d00b5cf3e2cedec4d81 P de93b58c2d67488dae12285e1806ebac: Snoozing failure detection for 2.099s (election complete) I0703 01:42:45.671106 32106 raft_consensus.cc:2537] T b2c8b58f935a4d00b5cf3e2cedec4d81 P de93b58c2d67488dae12285e1806ebac [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:42:45.671937 32106 raft_consensus.cc:593] T b2c8b58f935a4d00b5cf3e2cedec4d81 P de93b58c2d67488dae12285e1806ebac [term 1 LEADER]: Becoming Leader. State: Replica: de93b58c2d67488dae12285e1806ebac, State: Running, Role: LEADER I0703 01:42:45.672328 32106 consensus_queue.cc:204] T b2c8b58f935a4d00b5cf3e2cedec4d81 P de93b58c2d67488dae12285e1806ebac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } W0703 01:42:45.672655 31561 tablet.cc:1678] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:42:45.675107 31151 catalog_manager.cc:3798] T b2c8b58f935a4d00b5cf3e2cedec4d81 P de93b58c2d67488dae12285e1806ebac reported cstate change: term changed from 0 to 1, leader changed from to de93b58c2d67488dae12285e1806ebac (127.5.240.195). New cstate: current_term: 1 leader_uuid: "de93b58c2d67488dae12285e1806ebac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } } W0703 01:42:45.765530 31316 tablet.cc:1678] T 19c33e54ee9e476796de7de42ba495b6 P 1288ca0989104bbea36e22f38c374218: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:42:45.917989 32106 raft_consensus.cc:457] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0703 01:42:45.918061 32106 raft_consensus.cc:2730] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac: Snoozing failure detection for 1.818s (starting election) I0703 01:42:45.918130 32106 raft_consensus.cc:479] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } I0703 01:42:45.918452 32106 leader_election.cc:243] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521) I0703 01:42:45.919196 31272 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "19c33e54ee9e476796de7de42ba495b6" candidate_uuid: "de93b58c2d67488dae12285e1806ebac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1288ca0989104bbea36e22f38c374218" is_pre_election: true I0703 01:42:45.919324 31272 raft_consensus.cc:2730] T 19c33e54ee9e476796de7de42ba495b6 P 1288ca0989104bbea36e22f38c374218: Snoozing failure detection for 2.102s (vote granted) I0703 01:42:45.919384 31272 raft_consensus.cc:2243] T 19c33e54ee9e476796de7de42ba495b6 P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de93b58c2d67488dae12285e1806ebac in term 0. I0703 01:42:45.919814 31449 leader_election.cc:379] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac [CANDIDATE]: Term 1 pre-election: Vote granted by peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521) I0703 01:42:45.919875 31449 leader_election.cc:270] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0703 01:42:45.920047 32106 raft_consensus.cc:2730] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac: Snoozing failure detection for 1.559s (election complete) I0703 01:42:45.920130 32106 raft_consensus.cc:2537] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Leader pre-election won for term 1 I0703 01:42:45.920169 32106 raft_consensus.cc:457] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0703 01:42:45.920218 32106 raft_consensus.cc:2730] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac: Snoozing failure detection for 2.174s (starting election) I0703 01:42:45.920258 32106 raft_consensus.cc:2782] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Advancing to term 1 I0703 01:42:45.921599 32106 raft_consensus.cc:479] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } I0703 01:42:45.921835 32106 leader_election.cc:243] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac [CANDIDATE]: Term 1 election: Requesting vote from peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521) I0703 01:42:45.922397 31272 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "19c33e54ee9e476796de7de42ba495b6" candidate_uuid: "de93b58c2d67488dae12285e1806ebac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1288ca0989104bbea36e22f38c374218" I0703 01:42:45.922494 31272 raft_consensus.cc:2782] T 19c33e54ee9e476796de7de42ba495b6 P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:42:45.922565 31272 raft_consensus.cc:2730] T 19c33e54ee9e476796de7de42ba495b6 P 1288ca0989104bbea36e22f38c374218: Snoozing failure detection for 2.098s (vote granted) I0703 01:42:45.924204 31272 raft_consensus.cc:2243] T 19c33e54ee9e476796de7de42ba495b6 P 1288ca0989104bbea36e22f38c374218 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de93b58c2d67488dae12285e1806ebac in term 1. I0703 01:42:45.924587 31449 leader_election.cc:379] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac [CANDIDATE]: Term 1 election: Vote granted by peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521) I0703 01:42:45.924657 31449 leader_election.cc:270] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:42:45.924835 32106 raft_consensus.cc:2730] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac: Snoozing failure detection for 1.589s (election complete) I0703 01:42:45.924901 32106 raft_consensus.cc:2537] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:42:45.925015 32106 raft_consensus.cc:593] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac [term 1 LEADER]: Becoming Leader. State: Replica: de93b58c2d67488dae12285e1806ebac, State: Running, Role: LEADER I0703 01:42:45.925179 32106 consensus_queue.cc:204] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } I0703 01:42:45.926997 31151 catalog_manager.cc:3798] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac reported cstate change: term changed from 0 to 1, leader changed from to de93b58c2d67488dae12285e1806ebac (127.5.240.195). New cstate: current_term: 1 leader_uuid: "de93b58c2d67488dae12285e1806ebac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } } W0703 01:42:46.038460 31328 consensus_peers.cc:433] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 56eaab17f812433a8f7880f0744e1744 (127.5.240.199:57714): Couldn't send request to peer 56eaab17f812433a8f7880f0744e1744 for tablet 454bab377db147a1a2f8241ecc2241f3. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.199:57714: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 9 times. I0703 01:42:46.093628 31272 raft_consensus.cc:1081] T 44f06af6e8ef46d78419aea845bb38a7 P 1288ca0989104bbea36e22f38c374218 [term 1 FOLLOWER]: Refusing update from remote peer 2f3fdb7c089640c0b936a574bc64707e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0703 01:42:46.094521 32090 consensus_queue.cc:962] T 44f06af6e8ef46d78419aea845bb38a7 P 2f3fdb7c089640c0b936a574bc64707e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s W0703 01:42:46.199318 32117 flags.cc:406] Enabled unsafe flag: --never_fsync=true Using table 'rebalance_test_table_0' I0703 01:42:46.297926 31272 raft_consensus.cc:1081] T b2c8b58f935a4d00b5cf3e2cedec4d81 P 1288ca0989104bbea36e22f38c374218 [term 1 FOLLOWER]: Refusing update from remote peer de93b58c2d67488dae12285e1806ebac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0703 01:42:46.298993 32106 consensus_queue.cc:962] T b2c8b58f935a4d00b5cf3e2cedec4d81 P de93b58c2d67488dae12285e1806ebac [LEADER]: Connected to new peer: Peer: permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0703 01:42:46.343549 31271 raft_consensus.cc:1081] T 19c33e54ee9e476796de7de42ba495b6 P 1288ca0989104bbea36e22f38c374218 [term 1 FOLLOWER]: Refusing update from remote peer de93b58c2d67488dae12285e1806ebac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0703 01:42:46.345103 32114 consensus_queue.cc:962] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac [LEADER]: Connected to new peer: Peer: permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s Generator report time total : 171.147 ms time per row: 0.0855737 ms I0703 01:42:46.467388 31150 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.5.240.254:51772: name: "rebalance_test_table_1" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 2 split_rows_range_bounds { } partition_schema { hash_bucket_schemas { columns { name: "key" } num_buckets: 3 seed: 0 } } I0703 01:42:46.474874 31374 tablet_service.cc:759] Processing CreateTablet for tablet 5eaa9c06d73c4e23b51fae2688eaf26c (table=rebalance_test_table_1 [id=7d7c285baa3a4e26a62b27d088cedcc9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I0703 01:42:46.474928 31497 tablet_service.cc:759] Processing CreateTablet for tablet 5eaa9c06d73c4e23b51fae2688eaf26c (table=rebalance_test_table_1 [id=7d7c285baa3a4e26a62b27d088cedcc9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I0703 01:42:46.475108 31375 tablet_service.cc:759] Processing CreateTablet for tablet 1f01bfdde47b4d37a0f77d199bc7b4d2 (table=rebalance_test_table_1 [id=7d7c285baa3a4e26a62b27d088cedcc9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I0703 01:42:46.475114 31252 tablet_service.cc:759] Processing CreateTablet for tablet ee15a8cd712a45d6ac7468bc550d7635 (table=rebalance_test_table_1 [id=7d7c285baa3a4e26a62b27d088cedcc9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I0703 01:42:46.475276 31497 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 5eaa9c06d73c4e23b51fae2688eaf26c. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:46.475411 31375 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 1f01bfdde47b4d37a0f77d199bc7b4d2. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:46.475468 31252 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet ee15a8cd712a45d6ac7468bc550d7635. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:46.475728 31374 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 5eaa9c06d73c4e23b51fae2688eaf26c. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:46.476034 31496 tablet_service.cc:759] Processing CreateTablet for tablet 1f01bfdde47b4d37a0f77d199bc7b4d2 (table=rebalance_test_table_1 [id=7d7c285baa3a4e26a62b27d088cedcc9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I0703 01:42:46.476398 31496 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 1f01bfdde47b4d37a0f77d199bc7b4d2. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:46.479089 31373 tablet_service.cc:759] Processing CreateTablet for tablet ee15a8cd712a45d6ac7468bc550d7635 (table=rebalance_test_table_1 [id=7d7c285baa3a4e26a62b27d088cedcc9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I0703 01:42:46.480029 31252 ts_tablet_manager.cc:1173] T ee15a8cd712a45d6ac7468bc550d7635 P 1288ca0989104bbea36e22f38c374218: Registered tablet (data state: TABLET_DATA_READY) I0703 01:42:46.480727 31373 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet ee15a8cd712a45d6ac7468bc550d7635. 1 dirs total, 0 dirs full, 0 dirs failed W0703 01:42:46.482738 31328 consensus_peers.cc:433] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 56eaab17f812433a8f7880f0744e1744 (127.5.240.199:57714): Couldn't send request to peer 56eaab17f812433a8f7880f0744e1744 for tablet 454bab377db147a1a2f8241ecc2241f3. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.199:57714: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 10 times. I0703 01:42:46.483112 31496 ts_tablet_manager.cc:1173] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P de93b58c2d67488dae12285e1806ebac: Registered tablet (data state: TABLET_DATA_READY) I0703 01:42:46.483527 32154 ts_tablet_manager.cc:1028] T ee15a8cd712a45d6ac7468bc550d7635 P 1288ca0989104bbea36e22f38c374218: Bootstrapping tablet I0703 01:42:46.483770 32154 tablet_bootstrap.cc:427] T ee15a8cd712a45d6ac7468bc550d7635 P 1288ca0989104bbea36e22f38c374218: Bootstrap starting. I0703 01:42:46.483938 31374 ts_tablet_manager.cc:1173] T 5eaa9c06d73c4e23b51fae2688eaf26c P 2f3fdb7c089640c0b936a574bc64707e: Registered tablet (data state: TABLET_DATA_READY) I0703 01:42:46.484352 32155 ts_tablet_manager.cc:1028] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P de93b58c2d67488dae12285e1806ebac: Bootstrapping tablet I0703 01:42:46.484586 32155 tablet_bootstrap.cc:427] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P de93b58c2d67488dae12285e1806ebac: Bootstrap starting. I0703 01:42:46.485244 31497 ts_tablet_manager.cc:1173] T 5eaa9c06d73c4e23b51fae2688eaf26c P de93b58c2d67488dae12285e1806ebac: Registered tablet (data state: TABLET_DATA_READY) I0703 01:42:46.486395 32155 tablet_bootstrap.cc:578] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P de93b58c2d67488dae12285e1806ebac: No blocks or log segments found. Creating new log. I0703 01:42:46.487398 32154 tablet_bootstrap.cc:578] T ee15a8cd712a45d6ac7468bc550d7635 P 1288ca0989104bbea36e22f38c374218: No blocks or log segments found. Creating new log. I0703 01:42:46.488418 31375 ts_tablet_manager.cc:1173] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e: Registered tablet (data state: TABLET_DATA_READY) I0703 01:42:46.488978 32156 ts_tablet_manager.cc:1028] T 5eaa9c06d73c4e23b51fae2688eaf26c P 2f3fdb7c089640c0b936a574bc64707e: Bootstrapping tablet I0703 01:42:46.489192 32156 tablet_bootstrap.cc:427] T 5eaa9c06d73c4e23b51fae2688eaf26c P 2f3fdb7c089640c0b936a574bc64707e: Bootstrap starting. I0703 01:42:46.489387 32155 tablet_bootstrap.cc:427] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P de93b58c2d67488dae12285e1806ebac: No bootstrap required, opened a new log I0703 01:42:46.489485 32155 ts_tablet_manager.cc:1045] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P de93b58c2d67488dae12285e1806ebac: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s I0703 01:42:46.490037 32155 raft_consensus.cc:325] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } I0703 01:42:46.490193 32155 raft_consensus.cc:351] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:42:46.490240 32155 raft_consensus.cc:628] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de93b58c2d67488dae12285e1806ebac, State: Initialized, Role: FOLLOWER I0703 01:42:46.490437 32155 consensus_queue.cc:227] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P de93b58c2d67488dae12285e1806ebac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } I0703 01:42:46.490782 31373 ts_tablet_manager.cc:1173] T ee15a8cd712a45d6ac7468bc550d7635 P 2f3fdb7c089640c0b936a574bc64707e: Registered tablet (data state: TABLET_DATA_READY) I0703 01:42:46.490957 32156 tablet_bootstrap.cc:578] T 5eaa9c06d73c4e23b51fae2688eaf26c P 2f3fdb7c089640c0b936a574bc64707e: No blocks or log segments found. Creating new log. I0703 01:42:46.491276 32155 ts_tablet_manager.cc:1073] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P de93b58c2d67488dae12285e1806ebac: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s I0703 01:42:46.491395 32155 ts_tablet_manager.cc:1028] T 5eaa9c06d73c4e23b51fae2688eaf26c P de93b58c2d67488dae12285e1806ebac: Bootstrapping tablet I0703 01:42:46.491493 32155 tablet_bootstrap.cc:427] T 5eaa9c06d73c4e23b51fae2688eaf26c P de93b58c2d67488dae12285e1806ebac: Bootstrap starting. I0703 01:42:46.492748 32154 tablet_bootstrap.cc:427] T ee15a8cd712a45d6ac7468bc550d7635 P 1288ca0989104bbea36e22f38c374218: No bootstrap required, opened a new log I0703 01:42:46.492754 32155 tablet_bootstrap.cc:578] T 5eaa9c06d73c4e23b51fae2688eaf26c P de93b58c2d67488dae12285e1806ebac: No blocks or log segments found. Creating new log. I0703 01:42:46.492887 32154 ts_tablet_manager.cc:1045] T ee15a8cd712a45d6ac7468bc550d7635 P 1288ca0989104bbea36e22f38c374218: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s I0703 01:42:46.494359 32154 raft_consensus.cc:325] T ee15a8cd712a45d6ac7468bc550d7635 P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } I0703 01:42:46.494876 32156 tablet_bootstrap.cc:427] T 5eaa9c06d73c4e23b51fae2688eaf26c P 2f3fdb7c089640c0b936a574bc64707e: No bootstrap required, opened a new log I0703 01:42:46.494976 32156 ts_tablet_manager.cc:1045] T 5eaa9c06d73c4e23b51fae2688eaf26c P 2f3fdb7c089640c0b936a574bc64707e: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.002s I0703 01:42:46.495164 32155 tablet_bootstrap.cc:427] T 5eaa9c06d73c4e23b51fae2688eaf26c P de93b58c2d67488dae12285e1806ebac: No bootstrap required, opened a new log I0703 01:42:46.495254 32155 ts_tablet_manager.cc:1045] T 5eaa9c06d73c4e23b51fae2688eaf26c P de93b58c2d67488dae12285e1806ebac: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s I0703 01:42:46.495482 32156 raft_consensus.cc:325] T 5eaa9c06d73c4e23b51fae2688eaf26c P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } I0703 01:42:46.495652 32156 raft_consensus.cc:351] T 5eaa9c06d73c4e23b51fae2688eaf26c P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:42:46.495702 32156 raft_consensus.cc:628] T 5eaa9c06d73c4e23b51fae2688eaf26c P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f3fdb7c089640c0b936a574bc64707e, State: Initialized, Role: FOLLOWER I0703 01:42:46.495721 32154 raft_consensus.cc:351] T ee15a8cd712a45d6ac7468bc550d7635 P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:42:46.495765 32154 raft_consensus.cc:628] T ee15a8cd712a45d6ac7468bc550d7635 P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1288ca0989104bbea36e22f38c374218, State: Initialized, Role: FOLLOWER I0703 01:42:46.495744 32155 raft_consensus.cc:325] T 5eaa9c06d73c4e23b51fae2688eaf26c P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } I0703 01:42:46.495900 32155 raft_consensus.cc:351] T 5eaa9c06d73c4e23b51fae2688eaf26c P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:42:46.495940 32155 raft_consensus.cc:628] T 5eaa9c06d73c4e23b51fae2688eaf26c P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de93b58c2d67488dae12285e1806ebac, State: Initialized, Role: FOLLOWER I0703 01:42:46.495949 32156 consensus_queue.cc:227] T 5eaa9c06d73c4e23b51fae2688eaf26c P 2f3fdb7c089640c0b936a574bc64707e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } I0703 01:42:46.495945 32154 consensus_queue.cc:227] T ee15a8cd712a45d6ac7468bc550d7635 P 1288ca0989104bbea36e22f38c374218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } I0703 01:42:46.496109 32155 consensus_queue.cc:227] T 5eaa9c06d73c4e23b51fae2688eaf26c P de93b58c2d67488dae12285e1806ebac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } I0703 01:42:46.496568 32154 ts_tablet_manager.cc:1073] T ee15a8cd712a45d6ac7468bc550d7635 P 1288ca0989104bbea36e22f38c374218: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s I0703 01:42:46.496629 32156 ts_tablet_manager.cc:1073] T 5eaa9c06d73c4e23b51fae2688eaf26c P 2f3fdb7c089640c0b936a574bc64707e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s I0703 01:42:46.496806 32156 ts_tablet_manager.cc:1028] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e: Bootstrapping tablet I0703 01:42:46.496949 32156 tablet_bootstrap.cc:427] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e: Bootstrap starting. I0703 01:42:46.497112 32155 ts_tablet_manager.cc:1073] T 5eaa9c06d73c4e23b51fae2688eaf26c P de93b58c2d67488dae12285e1806ebac: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s I0703 01:42:46.498554 32156 tablet_bootstrap.cc:578] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e: No blocks or log segments found. Creating new log. I0703 01:42:46.501950 32156 tablet_bootstrap.cc:427] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e: No bootstrap required, opened a new log I0703 01:42:46.502097 32156 ts_tablet_manager.cc:1045] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s I0703 01:42:46.502840 32156 raft_consensus.cc:325] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } I0703 01:42:46.503079 32156 raft_consensus.cc:351] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:42:46.503177 32156 raft_consensus.cc:628] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f3fdb7c089640c0b936a574bc64707e, State: Initialized, Role: FOLLOWER I0703 01:42:46.503423 32156 consensus_queue.cc:227] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } I0703 01:42:46.504042 32156 ts_tablet_manager.cc:1073] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s I0703 01:42:46.504277 32156 ts_tablet_manager.cc:1028] T ee15a8cd712a45d6ac7468bc550d7635 P 2f3fdb7c089640c0b936a574bc64707e: Bootstrapping tablet I0703 01:42:46.504446 32156 tablet_bootstrap.cc:427] T ee15a8cd712a45d6ac7468bc550d7635 P 2f3fdb7c089640c0b936a574bc64707e: Bootstrap starting. I0703 01:42:46.506019 32156 tablet_bootstrap.cc:578] T ee15a8cd712a45d6ac7468bc550d7635 P 2f3fdb7c089640c0b936a574bc64707e: No blocks or log segments found. Creating new log. I0703 01:42:46.508981 32156 tablet_bootstrap.cc:427] T ee15a8cd712a45d6ac7468bc550d7635 P 2f3fdb7c089640c0b936a574bc64707e: No bootstrap required, opened a new log I0703 01:42:46.509131 32156 ts_tablet_manager.cc:1045] T ee15a8cd712a45d6ac7468bc550d7635 P 2f3fdb7c089640c0b936a574bc64707e: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s I0703 01:42:46.509881 32156 raft_consensus.cc:325] T ee15a8cd712a45d6ac7468bc550d7635 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } I0703 01:42:46.510146 32156 raft_consensus.cc:351] T ee15a8cd712a45d6ac7468bc550d7635 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:42:46.510242 32156 raft_consensus.cc:628] T ee15a8cd712a45d6ac7468bc550d7635 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f3fdb7c089640c0b936a574bc64707e, State: Initialized, Role: FOLLOWER I0703 01:42:46.510507 32156 consensus_queue.cc:227] T ee15a8cd712a45d6ac7468bc550d7635 P 2f3fdb7c089640c0b936a574bc64707e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } I0703 01:42:46.511118 32156 ts_tablet_manager.cc:1073] T ee15a8cd712a45d6ac7468bc550d7635 P 2f3fdb7c089640c0b936a574bc64707e: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s I0703 01:42:46.542582 32090 raft_consensus.cc:457] T ee15a8cd712a45d6ac7468bc550d7635 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0703 01:42:46.542636 32090 raft_consensus.cc:2730] T ee15a8cd712a45d6ac7468bc550d7635 P 2f3fdb7c089640c0b936a574bc64707e: Snoozing failure detection for 1.516s (starting election) I0703 01:42:46.542697 32090 raft_consensus.cc:479] T ee15a8cd712a45d6ac7468bc550d7635 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } I0703 01:42:46.542978 32090 leader_election.cc:243] T ee15a8cd712a45d6ac7468bc550d7635 P 2f3fdb7c089640c0b936a574bc64707e [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521) I0703 01:42:46.543630 31272 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "ee15a8cd712a45d6ac7468bc550d7635" candidate_uuid: "2f3fdb7c089640c0b936a574bc64707e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1288ca0989104bbea36e22f38c374218" is_pre_election: true I0703 01:42:46.543774 31272 raft_consensus.cc:2730] T ee15a8cd712a45d6ac7468bc550d7635 P 1288ca0989104bbea36e22f38c374218: Snoozing failure detection for 1.635s (vote granted) I0703 01:42:46.543840 31272 raft_consensus.cc:2243] T ee15a8cd712a45d6ac7468bc550d7635 P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f3fdb7c089640c0b936a574bc64707e in term 0. I0703 01:42:46.544279 31327 leader_election.cc:379] T ee15a8cd712a45d6ac7468bc550d7635 P 2f3fdb7c089640c0b936a574bc64707e [CANDIDATE]: Term 1 pre-election: Vote granted by peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521) I0703 01:42:46.544337 31327 leader_election.cc:270] T ee15a8cd712a45d6ac7468bc550d7635 P 2f3fdb7c089640c0b936a574bc64707e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0703 01:42:46.544498 32090 raft_consensus.cc:2730] T ee15a8cd712a45d6ac7468bc550d7635 P 2f3fdb7c089640c0b936a574bc64707e: Snoozing failure detection for 1.990s (election complete) I0703 01:42:46.544562 32090 raft_consensus.cc:2537] T ee15a8cd712a45d6ac7468bc550d7635 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Leader pre-election won for term 1 I0703 01:42:46.544607 32090 raft_consensus.cc:457] T ee15a8cd712a45d6ac7468bc550d7635 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0703 01:42:46.544643 32090 raft_consensus.cc:2730] T ee15a8cd712a45d6ac7468bc550d7635 P 2f3fdb7c089640c0b936a574bc64707e: Snoozing failure detection for 1.861s (starting election) I0703 01:42:46.544682 32090 raft_consensus.cc:2782] T ee15a8cd712a45d6ac7468bc550d7635 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Advancing to term 1 I0703 01:42:46.545888 32090 raft_consensus.cc:479] T ee15a8cd712a45d6ac7468bc550d7635 P 2f3fdb7c089640c0b936a574bc64707e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } I0703 01:42:46.546116 32090 leader_election.cc:243] T ee15a8cd712a45d6ac7468bc550d7635 P 2f3fdb7c089640c0b936a574bc64707e [CANDIDATE]: Term 1 election: Requesting vote from peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521) I0703 01:42:46.546677 31272 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "ee15a8cd712a45d6ac7468bc550d7635" candidate_uuid: "2f3fdb7c089640c0b936a574bc64707e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1288ca0989104bbea36e22f38c374218" I0703 01:42:46.546785 31272 raft_consensus.cc:2782] T ee15a8cd712a45d6ac7468bc550d7635 P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:42:46.546860 31272 raft_consensus.cc:2730] T ee15a8cd712a45d6ac7468bc550d7635 P 1288ca0989104bbea36e22f38c374218: Snoozing failure detection for 2.203s (vote granted) I0703 01:42:46.548120 31272 raft_consensus.cc:2243] T ee15a8cd712a45d6ac7468bc550d7635 P 1288ca0989104bbea36e22f38c374218 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f3fdb7c089640c0b936a574bc64707e in term 1. I0703 01:42:46.548579 31327 leader_election.cc:379] T ee15a8cd712a45d6ac7468bc550d7635 P 2f3fdb7c089640c0b936a574bc64707e [CANDIDATE]: Term 1 election: Vote granted by peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521) I0703 01:42:46.548633 31327 leader_election.cc:270] T ee15a8cd712a45d6ac7468bc550d7635 P 2f3fdb7c089640c0b936a574bc64707e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:42:46.548804 32090 raft_consensus.cc:2730] T ee15a8cd712a45d6ac7468bc550d7635 P 2f3fdb7c089640c0b936a574bc64707e: Snoozing failure detection for 1.849s (election complete) I0703 01:42:46.548866 32090 raft_consensus.cc:2537] T ee15a8cd712a45d6ac7468bc550d7635 P 2f3fdb7c089640c0b936a574bc64707e [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:42:46.548957 32090 raft_consensus.cc:593] T ee15a8cd712a45d6ac7468bc550d7635 P 2f3fdb7c089640c0b936a574bc64707e [term 1 LEADER]: Becoming Leader. State: Replica: 2f3fdb7c089640c0b936a574bc64707e, State: Running, Role: LEADER I0703 01:42:46.549155 32090 consensus_queue.cc:204] T ee15a8cd712a45d6ac7468bc550d7635 P 2f3fdb7c089640c0b936a574bc64707e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } I0703 01:42:46.551252 31147 catalog_manager.cc:3798] T ee15a8cd712a45d6ac7468bc550d7635 P 2f3fdb7c089640c0b936a574bc64707e reported cstate change: term changed from 0 to 1, leader changed from to 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194). New cstate: current_term: 1 leader_uuid: "2f3fdb7c089640c0b936a574bc64707e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } } I0703 01:42:46.615010 32144 raft_consensus.cc:457] T 5eaa9c06d73c4e23b51fae2688eaf26c P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0703 01:42:46.615094 32144 raft_consensus.cc:2730] T 5eaa9c06d73c4e23b51fae2688eaf26c P de93b58c2d67488dae12285e1806ebac: Snoozing failure detection for 1.721s (starting election) I0703 01:42:46.615185 32144 raft_consensus.cc:479] T 5eaa9c06d73c4e23b51fae2688eaf26c P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } I0703 01:42:46.615532 32144 leader_election.cc:243] T 5eaa9c06d73c4e23b51fae2688eaf26c P de93b58c2d67488dae12285e1806ebac [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194:39827) I0703 01:42:46.620753 31395 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "5eaa9c06d73c4e23b51fae2688eaf26c" candidate_uuid: "de93b58c2d67488dae12285e1806ebac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f3fdb7c089640c0b936a574bc64707e" is_pre_election: true I0703 01:42:46.620967 31395 raft_consensus.cc:2730] T 5eaa9c06d73c4e23b51fae2688eaf26c P 2f3fdb7c089640c0b936a574bc64707e: Snoozing failure detection for 1.592s (vote granted) I0703 01:42:46.621038 31395 raft_consensus.cc:2243] T 5eaa9c06d73c4e23b51fae2688eaf26c P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de93b58c2d67488dae12285e1806ebac in term 0. I0703 01:42:46.621527 31448 leader_election.cc:379] T 5eaa9c06d73c4e23b51fae2688eaf26c P de93b58c2d67488dae12285e1806ebac [CANDIDATE]: Term 1 pre-election: Vote granted by peer 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194:39827) I0703 01:42:46.621580 31448 leader_election.cc:270] T 5eaa9c06d73c4e23b51fae2688eaf26c P de93b58c2d67488dae12285e1806ebac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0703 01:42:46.621793 32144 raft_consensus.cc:2730] T 5eaa9c06d73c4e23b51fae2688eaf26c P de93b58c2d67488dae12285e1806ebac: Snoozing failure detection for 1.933s (election complete) I0703 01:42:46.621848 32144 raft_consensus.cc:2537] T 5eaa9c06d73c4e23b51fae2688eaf26c P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Leader pre-election won for term 1 I0703 01:42:46.621891 32144 raft_consensus.cc:457] T 5eaa9c06d73c4e23b51fae2688eaf26c P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0703 01:42:46.621927 32144 raft_consensus.cc:2730] T 5eaa9c06d73c4e23b51fae2688eaf26c P de93b58c2d67488dae12285e1806ebac: Snoozing failure detection for 2.239s (starting election) I0703 01:42:46.621969 32144 raft_consensus.cc:2782] T 5eaa9c06d73c4e23b51fae2688eaf26c P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Advancing to term 1 I0703 01:42:46.623320 32144 raft_consensus.cc:479] T 5eaa9c06d73c4e23b51fae2688eaf26c P de93b58c2d67488dae12285e1806ebac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } I0703 01:42:46.623581 32144 leader_election.cc:243] T 5eaa9c06d73c4e23b51fae2688eaf26c P de93b58c2d67488dae12285e1806ebac [CANDIDATE]: Term 1 election: Requesting vote from peer 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194:39827) I0703 01:42:46.624181 31395 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "5eaa9c06d73c4e23b51fae2688eaf26c" candidate_uuid: "de93b58c2d67488dae12285e1806ebac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f3fdb7c089640c0b936a574bc64707e" I0703 01:42:46.624284 31395 raft_consensus.cc:2782] T 5eaa9c06d73c4e23b51fae2688eaf26c P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Advancing to term 1 I0703 01:42:46.624341 31395 raft_consensus.cc:2730] T 5eaa9c06d73c4e23b51fae2688eaf26c P 2f3fdb7c089640c0b936a574bc64707e: Snoozing failure detection for 1.965s (vote granted) I0703 01:42:46.626000 31395 raft_consensus.cc:2243] T 5eaa9c06d73c4e23b51fae2688eaf26c P 2f3fdb7c089640c0b936a574bc64707e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de93b58c2d67488dae12285e1806ebac in term 1. I0703 01:42:46.626369 31448 leader_election.cc:379] T 5eaa9c06d73c4e23b51fae2688eaf26c P de93b58c2d67488dae12285e1806ebac [CANDIDATE]: Term 1 election: Vote granted by peer 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194:39827) I0703 01:42:46.626420 31448 leader_election.cc:270] T 5eaa9c06d73c4e23b51fae2688eaf26c P de93b58c2d67488dae12285e1806ebac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:42:46.626611 32144 raft_consensus.cc:2730] T 5eaa9c06d73c4e23b51fae2688eaf26c P de93b58c2d67488dae12285e1806ebac: Snoozing failure detection for 1.943s (election complete) I0703 01:42:46.626672 32144 raft_consensus.cc:2537] T 5eaa9c06d73c4e23b51fae2688eaf26c P de93b58c2d67488dae12285e1806ebac [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:42:46.626778 32144 raft_consensus.cc:593] T 5eaa9c06d73c4e23b51fae2688eaf26c P de93b58c2d67488dae12285e1806ebac [term 1 LEADER]: Becoming Leader. State: Replica: de93b58c2d67488dae12285e1806ebac, State: Running, Role: LEADER I0703 01:42:46.627048 32144 consensus_queue.cc:204] T 5eaa9c06d73c4e23b51fae2688eaf26c P de93b58c2d67488dae12285e1806ebac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } I0703 01:42:46.629252 31148 catalog_manager.cc:3798] T 5eaa9c06d73c4e23b51fae2688eaf26c P de93b58c2d67488dae12285e1806ebac reported cstate change: term changed from 0 to 1, leader changed from to de93b58c2d67488dae12285e1806ebac (127.5.240.195). New cstate: current_term: 1 leader_uuid: "de93b58c2d67488dae12285e1806ebac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } } I0703 01:42:46.820175 32090 raft_consensus.cc:457] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0703 01:42:46.820245 32090 raft_consensus.cc:2730] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e: Snoozing failure detection for 1.666s (starting election) I0703 01:42:46.820305 32090 raft_consensus.cc:479] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } I0703 01:42:46.820650 32090 leader_election.cc:243] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321) I0703 01:42:46.825208 31517 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "1f01bfdde47b4d37a0f77d199bc7b4d2" candidate_uuid: "2f3fdb7c089640c0b936a574bc64707e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de93b58c2d67488dae12285e1806ebac" is_pre_election: true I0703 01:42:46.825429 31517 raft_consensus.cc:2730] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P de93b58c2d67488dae12285e1806ebac: Snoozing failure detection for 2.036s (vote granted) I0703 01:42:46.825507 31517 raft_consensus.cc:2243] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f3fdb7c089640c0b936a574bc64707e in term 0. I0703 01:42:46.826030 31329 leader_election.cc:379] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e [CANDIDATE]: Term 1 pre-election: Vote granted by peer de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321) I0703 01:42:46.826081 31329 leader_election.cc:270] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0703 01:42:46.826251 32090 raft_consensus.cc:2730] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e: Snoozing failure detection for 1.598s (election complete) I0703 01:42:46.826340 32090 raft_consensus.cc:2537] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Leader pre-election won for term 1 I0703 01:42:46.826378 32090 raft_consensus.cc:457] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0703 01:42:46.826419 32090 raft_consensus.cc:2730] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e: Snoozing failure detection for 1.657s (starting election) I0703 01:42:46.826459 32090 raft_consensus.cc:2782] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Advancing to term 1 I0703 01:42:46.827740 32090 raft_consensus.cc:479] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } I0703 01:42:46.827963 32090 leader_election.cc:243] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e [CANDIDATE]: Term 1 election: Requesting vote from peer de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321) I0703 01:42:46.828742 31517 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "1f01bfdde47b4d37a0f77d199bc7b4d2" candidate_uuid: "2f3fdb7c089640c0b936a574bc64707e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de93b58c2d67488dae12285e1806ebac" I0703 01:42:46.828867 31517 raft_consensus.cc:2782] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Advancing to term 1 I0703 01:42:46.829005 31517 raft_consensus.cc:2730] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P de93b58c2d67488dae12285e1806ebac: Snoozing failure detection for 2.133s (vote granted) I0703 01:42:46.830873 31517 raft_consensus.cc:2243] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P de93b58c2d67488dae12285e1806ebac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f3fdb7c089640c0b936a574bc64707e in term 1. I0703 01:42:46.831365 31329 leader_election.cc:379] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e [CANDIDATE]: Term 1 election: Vote granted by peer de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321) I0703 01:42:46.831427 31329 leader_election.cc:270] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:42:46.831600 32090 raft_consensus.cc:2730] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e: Snoozing failure detection for 1.898s (election complete) I0703 01:42:46.831692 32090 raft_consensus.cc:2537] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:42:46.831816 32090 raft_consensus.cc:593] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e [term 1 LEADER]: Becoming Leader. State: Replica: 2f3fdb7c089640c0b936a574bc64707e, State: Running, Role: LEADER I0703 01:42:46.832131 32090 consensus_queue.cc:204] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } I0703 01:42:46.834939 31148 catalog_manager.cc:3798] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e reported cstate change: term changed from 0 to 1, leader changed from to 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194). New cstate: current_term: 1 leader_uuid: "2f3fdb7c089640c0b936a574bc64707e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } } W0703 01:42:47.075664 31328 consensus_peers.cc:433] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 56eaab17f812433a8f7880f0744e1744 (127.5.240.199:57714): Couldn't send request to peer 56eaab17f812433a8f7880f0744e1744 for tablet 454bab377db147a1a2f8241ecc2241f3. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.199:57714: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 11 times. I0703 01:42:47.085178 31272 raft_consensus.cc:1081] T ee15a8cd712a45d6ac7468bc550d7635 P 1288ca0989104bbea36e22f38c374218 [term 1 FOLLOWER]: Refusing update from remote peer 2f3fdb7c089640c0b936a574bc64707e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0703 01:42:47.085817 32090 consensus_queue.cc:962] T ee15a8cd712a45d6ac7468bc550d7635 P 2f3fdb7c089640c0b936a574bc64707e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s W0703 01:42:47.142325 32168 flags.cc:406] Enabled unsafe flag: --never_fsync=true Using table 'rebalance_test_table_1' I0703 01:42:47.198752 31395 raft_consensus.cc:1081] T 5eaa9c06d73c4e23b51fae2688eaf26c P 2f3fdb7c089640c0b936a574bc64707e [term 1 FOLLOWER]: Refusing update from remote peer de93b58c2d67488dae12285e1806ebac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0703 01:42:47.199594 32144 consensus_queue.cc:962] T 5eaa9c06d73c4e23b51fae2688eaf26c P de93b58c2d67488dae12285e1806ebac [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0703 01:42:47.269002 31517 raft_consensus.cc:1081] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P de93b58c2d67488dae12285e1806ebac [term 1 FOLLOWER]: Refusing update from remote peer 2f3fdb7c089640c0b936a574bc64707e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0703 01:42:47.269762 32111 consensus_queue.cc:962] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e [LEADER]: Connected to new peer: Peer: permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s Generator report time total : 219.423 ms time per row: 0.109712 ms I0703 01:42:47.457311 31146 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.5.240.254:51772: name: "rebalance_test_table_2" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 2 split_rows_range_bounds { } partition_schema { hash_bucket_schemas { columns { name: "key" } num_buckets: 3 seed: 0 } } I0703 01:42:47.463934 31252 tablet_service.cc:759] Processing CreateTablet for tablet 517dc72fa73f42f0aae102212fbc2d36 (table=rebalance_test_table_2 [id=b1de6184513b4ad5b539fd284d2c95d9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I0703 01:42:47.464301 31252 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 517dc72fa73f42f0aae102212fbc2d36. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:47.464604 31496 tablet_service.cc:759] Processing CreateTablet for tablet 517dc72fa73f42f0aae102212fbc2d36 (table=rebalance_test_table_2 [id=b1de6184513b4ad5b539fd284d2c95d9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I0703 01:42:47.464954 31496 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 517dc72fa73f42f0aae102212fbc2d36. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:47.465147 31373 tablet_service.cc:759] Processing CreateTablet for tablet 479104ce31404ee4ad2488d9f46509e4 (table=rebalance_test_table_2 [id=b1de6184513b4ad5b539fd284d2c95d9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I0703 01:42:47.465474 31373 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 479104ce31404ee4ad2488d9f46509e4. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:47.465746 31250 tablet_service.cc:759] Processing CreateTablet for tablet 479104ce31404ee4ad2488d9f46509e4 (table=rebalance_test_table_2 [id=b1de6184513b4ad5b539fd284d2c95d9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I0703 01:42:47.466043 31250 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 479104ce31404ee4ad2488d9f46509e4. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:47.469285 31497 tablet_service.cc:759] Processing CreateTablet for tablet a200f8a763f14d54a1e84082dbafce8f (table=rebalance_test_table_2 [id=b1de6184513b4ad5b539fd284d2c95d9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I0703 01:42:47.469426 31496 ts_tablet_manager.cc:1173] T 517dc72fa73f42f0aae102212fbc2d36 P de93b58c2d67488dae12285e1806ebac: Registered tablet (data state: TABLET_DATA_READY) I0703 01:42:47.469629 31497 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet a200f8a763f14d54a1e84082dbafce8f. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:47.469697 31251 tablet_service.cc:759] Processing CreateTablet for tablet a200f8a763f14d54a1e84082dbafce8f (table=rebalance_test_table_2 [id=b1de6184513b4ad5b539fd284d2c95d9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I0703 01:42:47.470052 31251 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet a200f8a763f14d54a1e84082dbafce8f. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:47.470866 31373 ts_tablet_manager.cc:1173] T 479104ce31404ee4ad2488d9f46509e4 P 2f3fdb7c089640c0b936a574bc64707e: Registered tablet (data state: TABLET_DATA_READY) I0703 01:42:47.471904 31252 ts_tablet_manager.cc:1173] T 517dc72fa73f42f0aae102212fbc2d36 P 1288ca0989104bbea36e22f38c374218: Registered tablet (data state: TABLET_DATA_READY) I0703 01:42:47.472901 31250 ts_tablet_manager.cc:1173] T 479104ce31404ee4ad2488d9f46509e4 P 1288ca0989104bbea36e22f38c374218: Registered tablet (data state: TABLET_DATA_READY) I0703 01:42:47.473593 31497 ts_tablet_manager.cc:1173] T a200f8a763f14d54a1e84082dbafce8f P de93b58c2d67488dae12285e1806ebac: Registered tablet (data state: TABLET_DATA_READY) I0703 01:42:47.475126 31251 ts_tablet_manager.cc:1173] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218: Registered tablet (data state: TABLET_DATA_READY) I0703 01:42:47.475698 32204 ts_tablet_manager.cc:1028] T 479104ce31404ee4ad2488d9f46509e4 P 2f3fdb7c089640c0b936a574bc64707e: Bootstrapping tablet I0703 01:42:47.475889 32204 tablet_bootstrap.cc:427] T 479104ce31404ee4ad2488d9f46509e4 P 2f3fdb7c089640c0b936a574bc64707e: Bootstrap starting. I0703 01:42:47.476176 32206 ts_tablet_manager.cc:1028] T a200f8a763f14d54a1e84082dbafce8f P de93b58c2d67488dae12285e1806ebac: Bootstrapping tablet I0703 01:42:47.476292 32206 tablet_bootstrap.cc:427] T a200f8a763f14d54a1e84082dbafce8f P de93b58c2d67488dae12285e1806ebac: Bootstrap starting. I0703 01:42:47.477181 32206 tablet_bootstrap.cc:578] T a200f8a763f14d54a1e84082dbafce8f P de93b58c2d67488dae12285e1806ebac: No blocks or log segments found. Creating new log. I0703 01:42:47.477314 32204 tablet_bootstrap.cc:578] T 479104ce31404ee4ad2488d9f46509e4 P 2f3fdb7c089640c0b936a574bc64707e: No blocks or log segments found. Creating new log. I0703 01:42:47.478462 32205 ts_tablet_manager.cc:1028] T 517dc72fa73f42f0aae102212fbc2d36 P 1288ca0989104bbea36e22f38c374218: Bootstrapping tablet I0703 01:42:47.478658 32205 tablet_bootstrap.cc:427] T 517dc72fa73f42f0aae102212fbc2d36 P 1288ca0989104bbea36e22f38c374218: Bootstrap starting. I0703 01:42:47.478777 32206 tablet_bootstrap.cc:427] T a200f8a763f14d54a1e84082dbafce8f P de93b58c2d67488dae12285e1806ebac: No bootstrap required, opened a new log I0703 01:42:47.478873 32206 ts_tablet_manager.cc:1045] T a200f8a763f14d54a1e84082dbafce8f P de93b58c2d67488dae12285e1806ebac: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s I0703 01:42:47.479053 32204 tablet_bootstrap.cc:427] T 479104ce31404ee4ad2488d9f46509e4 P 2f3fdb7c089640c0b936a574bc64707e: No bootstrap required, opened a new log I0703 01:42:47.479162 32204 ts_tablet_manager.cc:1045] T 479104ce31404ee4ad2488d9f46509e4 P 2f3fdb7c089640c0b936a574bc64707e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s I0703 01:42:47.479425 32206 raft_consensus.cc:325] T a200f8a763f14d54a1e84082dbafce8f P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } I0703 01:42:47.479576 32206 raft_consensus.cc:351] T a200f8a763f14d54a1e84082dbafce8f P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:42:47.479622 32206 raft_consensus.cc:628] T a200f8a763f14d54a1e84082dbafce8f P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de93b58c2d67488dae12285e1806ebac, State: Initialized, Role: FOLLOWER I0703 01:42:47.479801 32206 consensus_queue.cc:227] T a200f8a763f14d54a1e84082dbafce8f P de93b58c2d67488dae12285e1806ebac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } I0703 01:42:47.479884 32205 tablet_bootstrap.cc:578] T 517dc72fa73f42f0aae102212fbc2d36 P 1288ca0989104bbea36e22f38c374218: No blocks or log segments found. Creating new log. I0703 01:42:47.479825 32204 raft_consensus.cc:325] T 479104ce31404ee4ad2488d9f46509e4 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } I0703 01:42:47.480032 32204 raft_consensus.cc:351] T 479104ce31404ee4ad2488d9f46509e4 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:42:47.480077 32204 raft_consensus.cc:628] T 479104ce31404ee4ad2488d9f46509e4 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f3fdb7c089640c0b936a574bc64707e, State: Initialized, Role: FOLLOWER I0703 01:42:47.480422 32206 ts_tablet_manager.cc:1073] T a200f8a763f14d54a1e84082dbafce8f P de93b58c2d67488dae12285e1806ebac: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s I0703 01:42:47.480651 32206 ts_tablet_manager.cc:1028] T 517dc72fa73f42f0aae102212fbc2d36 P de93b58c2d67488dae12285e1806ebac: Bootstrapping tablet I0703 01:42:47.480777 32206 tablet_bootstrap.cc:427] T 517dc72fa73f42f0aae102212fbc2d36 P de93b58c2d67488dae12285e1806ebac: Bootstrap starting. I0703 01:42:47.480309 32204 consensus_queue.cc:227] T 479104ce31404ee4ad2488d9f46509e4 P 2f3fdb7c089640c0b936a574bc64707e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } I0703 01:42:47.481787 32206 tablet_bootstrap.cc:578] T 517dc72fa73f42f0aae102212fbc2d36 P de93b58c2d67488dae12285e1806ebac: No blocks or log segments found. Creating new log. I0703 01:42:47.482576 32205 tablet_bootstrap.cc:427] T 517dc72fa73f42f0aae102212fbc2d36 P 1288ca0989104bbea36e22f38c374218: No bootstrap required, opened a new log I0703 01:42:47.482739 32205 ts_tablet_manager.cc:1045] T 517dc72fa73f42f0aae102212fbc2d36 P 1288ca0989104bbea36e22f38c374218: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s I0703 01:42:47.483469 32204 ts_tablet_manager.cc:1073] T 479104ce31404ee4ad2488d9f46509e4 P 2f3fdb7c089640c0b936a574bc64707e: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s I0703 01:42:47.483836 32206 tablet_bootstrap.cc:427] T 517dc72fa73f42f0aae102212fbc2d36 P de93b58c2d67488dae12285e1806ebac: No bootstrap required, opened a new log I0703 01:42:47.483952 32206 ts_tablet_manager.cc:1045] T 517dc72fa73f42f0aae102212fbc2d36 P de93b58c2d67488dae12285e1806ebac: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s I0703 01:42:47.484350 32205 raft_consensus.cc:325] T 517dc72fa73f42f0aae102212fbc2d36 P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } I0703 01:42:47.484377 32206 raft_consensus.cc:325] T 517dc72fa73f42f0aae102212fbc2d36 P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } I0703 01:42:47.484478 32206 raft_consensus.cc:351] T 517dc72fa73f42f0aae102212fbc2d36 P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:42:47.484484 32205 raft_consensus.cc:351] T 517dc72fa73f42f0aae102212fbc2d36 P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:42:47.484519 32206 raft_consensus.cc:628] T 517dc72fa73f42f0aae102212fbc2d36 P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de93b58c2d67488dae12285e1806ebac, State: Initialized, Role: FOLLOWER I0703 01:42:47.484553 32205 raft_consensus.cc:628] T 517dc72fa73f42f0aae102212fbc2d36 P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1288ca0989104bbea36e22f38c374218, State: Initialized, Role: FOLLOWER I0703 01:42:47.484692 32206 consensus_queue.cc:227] T 517dc72fa73f42f0aae102212fbc2d36 P de93b58c2d67488dae12285e1806ebac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } I0703 01:42:47.484742 32205 consensus_queue.cc:227] T 517dc72fa73f42f0aae102212fbc2d36 P 1288ca0989104bbea36e22f38c374218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } I0703 01:42:47.485215 32205 ts_tablet_manager.cc:1073] T 517dc72fa73f42f0aae102212fbc2d36 P 1288ca0989104bbea36e22f38c374218: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s I0703 01:42:47.485478 32206 ts_tablet_manager.cc:1073] T 517dc72fa73f42f0aae102212fbc2d36 P de93b58c2d67488dae12285e1806ebac: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s I0703 01:42:47.485841 32205 ts_tablet_manager.cc:1028] T 479104ce31404ee4ad2488d9f46509e4 P 1288ca0989104bbea36e22f38c374218: Bootstrapping tablet I0703 01:42:47.485957 32205 tablet_bootstrap.cc:427] T 479104ce31404ee4ad2488d9f46509e4 P 1288ca0989104bbea36e22f38c374218: Bootstrap starting. I0703 01:42:47.487372 32205 tablet_bootstrap.cc:578] T 479104ce31404ee4ad2488d9f46509e4 P 1288ca0989104bbea36e22f38c374218: No blocks or log segments found. Creating new log. I0703 01:42:47.488931 32205 tablet_bootstrap.cc:427] T 479104ce31404ee4ad2488d9f46509e4 P 1288ca0989104bbea36e22f38c374218: No bootstrap required, opened a new log I0703 01:42:47.489019 32205 ts_tablet_manager.cc:1045] T 479104ce31404ee4ad2488d9f46509e4 P 1288ca0989104bbea36e22f38c374218: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s I0703 01:42:47.489434 32205 raft_consensus.cc:325] T 479104ce31404ee4ad2488d9f46509e4 P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } I0703 01:42:47.489569 32205 raft_consensus.cc:351] T 479104ce31404ee4ad2488d9f46509e4 P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:42:47.489616 32205 raft_consensus.cc:628] T 479104ce31404ee4ad2488d9f46509e4 P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1288ca0989104bbea36e22f38c374218, State: Initialized, Role: FOLLOWER I0703 01:42:47.489765 32205 consensus_queue.cc:227] T 479104ce31404ee4ad2488d9f46509e4 P 1288ca0989104bbea36e22f38c374218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } I0703 01:42:47.490240 32205 ts_tablet_manager.cc:1073] T 479104ce31404ee4ad2488d9f46509e4 P 1288ca0989104bbea36e22f38c374218: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s I0703 01:42:47.490391 32205 ts_tablet_manager.cc:1028] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218: Bootstrapping tablet I0703 01:42:47.490483 32205 tablet_bootstrap.cc:427] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218: Bootstrap starting. I0703 01:42:47.491586 32205 tablet_bootstrap.cc:578] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218: No blocks or log segments found. Creating new log. I0703 01:42:47.493010 32205 tablet_bootstrap.cc:427] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218: No bootstrap required, opened a new log I0703 01:42:47.493098 32205 ts_tablet_manager.cc:1045] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.002s I0703 01:42:47.493507 32205 raft_consensus.cc:325] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } I0703 01:42:47.493659 32205 raft_consensus.cc:351] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:42:47.493707 32205 raft_consensus.cc:628] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1288ca0989104bbea36e22f38c374218, State: Initialized, Role: FOLLOWER I0703 01:42:47.493856 32205 consensus_queue.cc:227] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } I0703 01:42:47.494299 32205 ts_tablet_manager.cc:1073] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s I0703 01:42:47.518606 32195 raft_consensus.cc:457] T 479104ce31404ee4ad2488d9f46509e4 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0703 01:42:47.518682 32195 raft_consensus.cc:2730] T 479104ce31404ee4ad2488d9f46509e4 P 2f3fdb7c089640c0b936a574bc64707e: Snoozing failure detection for 2.136s (starting election) I0703 01:42:47.518765 32195 raft_consensus.cc:479] T 479104ce31404ee4ad2488d9f46509e4 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } I0703 01:42:47.519107 32195 leader_election.cc:243] T 479104ce31404ee4ad2488d9f46509e4 P 2f3fdb7c089640c0b936a574bc64707e [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521) I0703 01:42:47.519801 31272 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "479104ce31404ee4ad2488d9f46509e4" candidate_uuid: "2f3fdb7c089640c0b936a574bc64707e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1288ca0989104bbea36e22f38c374218" is_pre_election: true I0703 01:42:47.519963 31272 raft_consensus.cc:2730] T 479104ce31404ee4ad2488d9f46509e4 P 1288ca0989104bbea36e22f38c374218: Snoozing failure detection for 2.249s (vote granted) I0703 01:42:47.520023 31272 raft_consensus.cc:2243] T 479104ce31404ee4ad2488d9f46509e4 P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f3fdb7c089640c0b936a574bc64707e in term 0. I0703 01:42:47.520475 31327 leader_election.cc:379] T 479104ce31404ee4ad2488d9f46509e4 P 2f3fdb7c089640c0b936a574bc64707e [CANDIDATE]: Term 1 pre-election: Vote granted by peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521) I0703 01:42:47.520532 31327 leader_election.cc:270] T 479104ce31404ee4ad2488d9f46509e4 P 2f3fdb7c089640c0b936a574bc64707e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0703 01:42:47.520742 32195 raft_consensus.cc:2730] T 479104ce31404ee4ad2488d9f46509e4 P 2f3fdb7c089640c0b936a574bc64707e: Snoozing failure detection for 1.595s (election complete) I0703 01:42:47.520807 32195 raft_consensus.cc:2537] T 479104ce31404ee4ad2488d9f46509e4 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Leader pre-election won for term 1 I0703 01:42:47.520864 32195 raft_consensus.cc:457] T 479104ce31404ee4ad2488d9f46509e4 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0703 01:42:47.520915 32195 raft_consensus.cc:2730] T 479104ce31404ee4ad2488d9f46509e4 P 2f3fdb7c089640c0b936a574bc64707e: Snoozing failure detection for 1.884s (starting election) I0703 01:42:47.520978 32195 raft_consensus.cc:2782] T 479104ce31404ee4ad2488d9f46509e4 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Advancing to term 1 I0703 01:42:47.522297 32195 raft_consensus.cc:479] T 479104ce31404ee4ad2488d9f46509e4 P 2f3fdb7c089640c0b936a574bc64707e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } I0703 01:42:47.522555 32195 leader_election.cc:243] T 479104ce31404ee4ad2488d9f46509e4 P 2f3fdb7c089640c0b936a574bc64707e [CANDIDATE]: Term 1 election: Requesting vote from peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521) I0703 01:42:47.523172 31272 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "479104ce31404ee4ad2488d9f46509e4" candidate_uuid: "2f3fdb7c089640c0b936a574bc64707e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1288ca0989104bbea36e22f38c374218" I0703 01:42:47.523270 31272 raft_consensus.cc:2782] T 479104ce31404ee4ad2488d9f46509e4 P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:42:47.523337 31272 raft_consensus.cc:2730] T 479104ce31404ee4ad2488d9f46509e4 P 1288ca0989104bbea36e22f38c374218: Snoozing failure detection for 1.633s (vote granted) I0703 01:42:47.524618 31272 raft_consensus.cc:2243] T 479104ce31404ee4ad2488d9f46509e4 P 1288ca0989104bbea36e22f38c374218 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f3fdb7c089640c0b936a574bc64707e in term 1. I0703 01:42:47.525048 31327 leader_election.cc:379] T 479104ce31404ee4ad2488d9f46509e4 P 2f3fdb7c089640c0b936a574bc64707e [CANDIDATE]: Term 1 election: Vote granted by peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521) I0703 01:42:47.525104 31327 leader_election.cc:270] T 479104ce31404ee4ad2488d9f46509e4 P 2f3fdb7c089640c0b936a574bc64707e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:42:47.525274 32195 raft_consensus.cc:2730] T 479104ce31404ee4ad2488d9f46509e4 P 2f3fdb7c089640c0b936a574bc64707e: Snoozing failure detection for 2.083s (election complete) I0703 01:42:47.525341 32195 raft_consensus.cc:2537] T 479104ce31404ee4ad2488d9f46509e4 P 2f3fdb7c089640c0b936a574bc64707e [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:42:47.525458 32195 raft_consensus.cc:593] T 479104ce31404ee4ad2488d9f46509e4 P 2f3fdb7c089640c0b936a574bc64707e [term 1 LEADER]: Becoming Leader. State: Replica: 2f3fdb7c089640c0b936a574bc64707e, State: Running, Role: LEADER I0703 01:42:47.525837 32195 consensus_queue.cc:204] T 479104ce31404ee4ad2488d9f46509e4 P 2f3fdb7c089640c0b936a574bc64707e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } I0703 01:42:47.527516 31144 catalog_manager.cc:3798] T 479104ce31404ee4ad2488d9f46509e4 P 2f3fdb7c089640c0b936a574bc64707e reported cstate change: term changed from 0 to 1, leader changed from to 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194). New cstate: current_term: 1 leader_uuid: "2f3fdb7c089640c0b936a574bc64707e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } } W0703 01:42:47.627995 31328 consensus_peers.cc:433] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 56eaab17f812433a8f7880f0744e1744 (127.5.240.199:57714): Couldn't send request to peer 56eaab17f812433a8f7880f0744e1744 for tablet 454bab377db147a1a2f8241ecc2241f3. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.199:57714: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 12 times. I0703 01:42:47.696784 32169 raft_consensus.cc:457] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0703 01:42:47.696873 32169 raft_consensus.cc:2730] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218: Snoozing failure detection for 2.183s (starting election) I0703 01:42:47.696981 32169 raft_consensus.cc:479] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } I0703 01:42:47.697659 32169 leader_election.cc:243] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321) I0703 01:42:47.703202 31517 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "a200f8a763f14d54a1e84082dbafce8f" candidate_uuid: "1288ca0989104bbea36e22f38c374218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de93b58c2d67488dae12285e1806ebac" is_pre_election: true I0703 01:42:47.703351 31517 raft_consensus.cc:2730] T a200f8a763f14d54a1e84082dbafce8f P de93b58c2d67488dae12285e1806ebac: Snoozing failure detection for 1.870s (vote granted) I0703 01:42:47.703420 31517 raft_consensus.cc:2243] T a200f8a763f14d54a1e84082dbafce8f P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1288ca0989104bbea36e22f38c374218 in term 0. I0703 01:42:47.703869 31206 leader_election.cc:379] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218 [CANDIDATE]: Term 1 pre-election: Vote granted by peer de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321) I0703 01:42:47.703938 31206 leader_election.cc:270] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0703 01:42:47.704208 32169 raft_consensus.cc:2730] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218: Snoozing failure detection for 1.644s (election complete) I0703 01:42:47.704306 32169 raft_consensus.cc:2537] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0703 01:42:47.704350 32169 raft_consensus.cc:457] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0703 01:42:47.704382 32169 raft_consensus.cc:2730] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218: Snoozing failure detection for 2.248s (starting election) I0703 01:42:47.704421 32169 raft_consensus.cc:2782] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:42:47.705796 32169 raft_consensus.cc:479] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } I0703 01:42:47.706030 32169 leader_election.cc:243] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218 [CANDIDATE]: Term 1 election: Requesting vote from peer de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321) I0703 01:42:47.706598 31517 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "a200f8a763f14d54a1e84082dbafce8f" candidate_uuid: "1288ca0989104bbea36e22f38c374218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de93b58c2d67488dae12285e1806ebac" I0703 01:42:47.706720 31517 raft_consensus.cc:2782] T a200f8a763f14d54a1e84082dbafce8f P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Advancing to term 1 I0703 01:42:47.706784 31517 raft_consensus.cc:2730] T a200f8a763f14d54a1e84082dbafce8f P de93b58c2d67488dae12285e1806ebac: Snoozing failure detection for 2.165s (vote granted) I0703 01:42:47.708068 31517 raft_consensus.cc:2243] T a200f8a763f14d54a1e84082dbafce8f P de93b58c2d67488dae12285e1806ebac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1288ca0989104bbea36e22f38c374218 in term 1. I0703 01:42:47.708464 31206 leader_election.cc:379] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218 [CANDIDATE]: Term 1 election: Vote granted by peer de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321) I0703 01:42:47.708540 31206 leader_election.cc:270] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:42:47.708727 32169 raft_consensus.cc:2730] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218: Snoozing failure detection for 1.638s (election complete) I0703 01:42:47.708796 32169 raft_consensus.cc:2537] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218 [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:42:47.709686 32169 raft_consensus.cc:593] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218 [term 1 LEADER]: Becoming Leader. State: Replica: 1288ca0989104bbea36e22f38c374218, State: Running, Role: LEADER I0703 01:42:47.710059 32169 consensus_queue.cc:204] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } I0703 01:42:47.712510 31144 catalog_manager.cc:3798] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218 reported cstate change: term changed from 0 to 1, leader changed from to 1288ca0989104bbea36e22f38c374218 (127.5.240.193). New cstate: current_term: 1 leader_uuid: "1288ca0989104bbea36e22f38c374218" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } } I0703 01:42:47.781293 32169 raft_consensus.cc:457] T 517dc72fa73f42f0aae102212fbc2d36 P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0703 01:42:47.781360 32169 raft_consensus.cc:2730] T 517dc72fa73f42f0aae102212fbc2d36 P 1288ca0989104bbea36e22f38c374218: Snoozing failure detection for 1.771s (starting election) I0703 01:42:47.781433 32169 raft_consensus.cc:479] T 517dc72fa73f42f0aae102212fbc2d36 P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } I0703 01:42:47.781719 32169 leader_election.cc:243] T 517dc72fa73f42f0aae102212fbc2d36 P 1288ca0989104bbea36e22f38c374218 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321) I0703 01:42:47.782335 31517 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "517dc72fa73f42f0aae102212fbc2d36" candidate_uuid: "1288ca0989104bbea36e22f38c374218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de93b58c2d67488dae12285e1806ebac" is_pre_election: true I0703 01:42:47.782460 31517 raft_consensus.cc:2730] T 517dc72fa73f42f0aae102212fbc2d36 P de93b58c2d67488dae12285e1806ebac: Snoozing failure detection for 2.038s (vote granted) I0703 01:42:47.782536 31517 raft_consensus.cc:2243] T 517dc72fa73f42f0aae102212fbc2d36 P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1288ca0989104bbea36e22f38c374218 in term 0. I0703 01:42:47.782882 31206 leader_election.cc:379] T 517dc72fa73f42f0aae102212fbc2d36 P 1288ca0989104bbea36e22f38c374218 [CANDIDATE]: Term 1 pre-election: Vote granted by peer de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321) I0703 01:42:47.782937 31206 leader_election.cc:270] T 517dc72fa73f42f0aae102212fbc2d36 P 1288ca0989104bbea36e22f38c374218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0703 01:42:47.783113 32169 raft_consensus.cc:2730] T 517dc72fa73f42f0aae102212fbc2d36 P 1288ca0989104bbea36e22f38c374218: Snoozing failure detection for 1.689s (election complete) I0703 01:42:47.783183 32169 raft_consensus.cc:2537] T 517dc72fa73f42f0aae102212fbc2d36 P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0703 01:42:47.783232 32169 raft_consensus.cc:457] T 517dc72fa73f42f0aae102212fbc2d36 P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0703 01:42:47.783291 32169 raft_consensus.cc:2730] T 517dc72fa73f42f0aae102212fbc2d36 P 1288ca0989104bbea36e22f38c374218: Snoozing failure detection for 1.810s (starting election) I0703 01:42:47.783347 32169 raft_consensus.cc:2782] T 517dc72fa73f42f0aae102212fbc2d36 P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:42:47.784535 32169 raft_consensus.cc:479] T 517dc72fa73f42f0aae102212fbc2d36 P 1288ca0989104bbea36e22f38c374218 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } I0703 01:42:47.784802 32169 leader_election.cc:243] T 517dc72fa73f42f0aae102212fbc2d36 P 1288ca0989104bbea36e22f38c374218 [CANDIDATE]: Term 1 election: Requesting vote from peer de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321) I0703 01:42:47.785383 31517 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "517dc72fa73f42f0aae102212fbc2d36" candidate_uuid: "1288ca0989104bbea36e22f38c374218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de93b58c2d67488dae12285e1806ebac" I0703 01:42:47.785483 31517 raft_consensus.cc:2782] T 517dc72fa73f42f0aae102212fbc2d36 P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Advancing to term 1 I0703 01:42:47.785567 31517 raft_consensus.cc:2730] T 517dc72fa73f42f0aae102212fbc2d36 P de93b58c2d67488dae12285e1806ebac: Snoozing failure detection for 1.925s (vote granted) I0703 01:42:47.786773 31517 raft_consensus.cc:2243] T 517dc72fa73f42f0aae102212fbc2d36 P de93b58c2d67488dae12285e1806ebac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1288ca0989104bbea36e22f38c374218 in term 1. I0703 01:42:47.787159 31206 leader_election.cc:379] T 517dc72fa73f42f0aae102212fbc2d36 P 1288ca0989104bbea36e22f38c374218 [CANDIDATE]: Term 1 election: Vote granted by peer de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321) I0703 01:42:47.787220 31206 leader_election.cc:270] T 517dc72fa73f42f0aae102212fbc2d36 P 1288ca0989104bbea36e22f38c374218 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:42:47.787408 32169 raft_consensus.cc:2730] T 517dc72fa73f42f0aae102212fbc2d36 P 1288ca0989104bbea36e22f38c374218: Snoozing failure detection for 1.502s (election complete) I0703 01:42:47.787468 32169 raft_consensus.cc:2537] T 517dc72fa73f42f0aae102212fbc2d36 P 1288ca0989104bbea36e22f38c374218 [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:42:47.787598 32169 raft_consensus.cc:593] T 517dc72fa73f42f0aae102212fbc2d36 P 1288ca0989104bbea36e22f38c374218 [term 1 LEADER]: Becoming Leader. State: Replica: 1288ca0989104bbea36e22f38c374218, State: Running, Role: LEADER I0703 01:42:47.787761 32169 consensus_queue.cc:204] T 517dc72fa73f42f0aae102212fbc2d36 P 1288ca0989104bbea36e22f38c374218 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } I0703 01:42:47.789902 31144 catalog_manager.cc:3798] T 517dc72fa73f42f0aae102212fbc2d36 P 1288ca0989104bbea36e22f38c374218 reported cstate change: term changed from 0 to 1, leader changed from to 1288ca0989104bbea36e22f38c374218 (127.5.240.193). New cstate: current_term: 1 leader_uuid: "1288ca0989104bbea36e22f38c374218" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } } I0703 01:42:47.974225 31272 raft_consensus.cc:1081] T 479104ce31404ee4ad2488d9f46509e4 P 1288ca0989104bbea36e22f38c374218 [term 1 FOLLOWER]: Refusing update from remote peer 2f3fdb7c089640c0b936a574bc64707e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0703 01:42:47.974812 32195 consensus_queue.cc:962] T 479104ce31404ee4ad2488d9f46509e4 P 2f3fdb7c089640c0b936a574bc64707e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s W0703 01:42:48.047850 32219 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:42:48.083214 31328 consensus_peers.cc:433] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 56eaab17f812433a8f7880f0744e1744 (127.5.240.199:57714): Couldn't send request to peer 56eaab17f812433a8f7880f0744e1744 for tablet 454bab377db147a1a2f8241ecc2241f3. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.199:57714: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 13 times. Using table 'rebalance_test_table_2' I0703 01:42:48.183019 31517 raft_consensus.cc:1081] T a200f8a763f14d54a1e84082dbafce8f P de93b58c2d67488dae12285e1806ebac [term 1 FOLLOWER]: Refusing update from remote peer 1288ca0989104bbea36e22f38c374218: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0703 01:42:48.184288 32169 consensus_queue.cc:962] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0703 01:42:48.184906 31517 raft_consensus.cc:1081] T 517dc72fa73f42f0aae102212fbc2d36 P de93b58c2d67488dae12285e1806ebac [term 1 FOLLOWER]: Refusing update from remote peer 1288ca0989104bbea36e22f38c374218: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0703 01:42:48.187443 32169 consensus_queue.cc:962] T 517dc72fa73f42f0aae102212fbc2d36 P 1288ca0989104bbea36e22f38c374218 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s Generator report time total : 202.974 ms time per row: 0.101487 ms I0703 01:42:48.343984 31146 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.5.240.254:51772: name: "rebalance_test_table_3" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 2 split_rows_range_bounds { } partition_schema { hash_bucket_schemas { columns { name: "key" } num_buckets: 3 seed: 0 } } I0703 01:42:48.350294 31373 tablet_service.cc:759] Processing CreateTablet for tablet 3d7a6e87780a49949cf7e024817cf3a3 (table=rebalance_test_table_3 [id=b06785934be6432b91c9f50ee8c104d2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I0703 01:42:48.350697 31373 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 3d7a6e87780a49949cf7e024817cf3a3. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:48.351294 31375 tablet_service.cc:759] Processing CreateTablet for tablet e0af316d40d143a8b387933151adf0e4 (table=rebalance_test_table_3 [id=b06785934be6432b91c9f50ee8c104d2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I0703 01:42:48.351547 31252 tablet_service.cc:759] Processing CreateTablet for tablet f9f1c40150da4144acb68186f9aad763 (table=rebalance_test_table_3 [id=b06785934be6432b91c9f50ee8c104d2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I0703 01:42:48.351725 31375 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet e0af316d40d143a8b387933151adf0e4. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:48.351897 31252 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet f9f1c40150da4144acb68186f9aad763. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:48.351997 31497 tablet_service.cc:759] Processing CreateTablet for tablet 3d7a6e87780a49949cf7e024817cf3a3 (table=rebalance_test_table_3 [id=b06785934be6432b91c9f50ee8c104d2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I0703 01:42:48.352327 31497 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 3d7a6e87780a49949cf7e024817cf3a3. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:48.353096 31374 tablet_service.cc:759] Processing CreateTablet for tablet f9f1c40150da4144acb68186f9aad763 (table=rebalance_test_table_3 [id=b06785934be6432b91c9f50ee8c104d2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I0703 01:42:48.353435 31374 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet f9f1c40150da4144acb68186f9aad763. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:48.355851 31496 tablet_service.cc:759] Processing CreateTablet for tablet e0af316d40d143a8b387933151adf0e4 (table=rebalance_test_table_3 [id=b06785934be6432b91c9f50ee8c104d2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I0703 01:42:48.356123 31496 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet e0af316d40d143a8b387933151adf0e4. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:48.356212 31252 ts_tablet_manager.cc:1173] T f9f1c40150da4144acb68186f9aad763 P 1288ca0989104bbea36e22f38c374218: Registered tablet (data state: TABLET_DATA_READY) I0703 01:42:48.357017 31373 ts_tablet_manager.cc:1173] T 3d7a6e87780a49949cf7e024817cf3a3 P 2f3fdb7c089640c0b936a574bc64707e: Registered tablet (data state: TABLET_DATA_READY) I0703 01:42:48.359174 31374 ts_tablet_manager.cc:1173] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e: Registered tablet (data state: TABLET_DATA_READY) I0703 01:42:48.360653 32257 ts_tablet_manager.cc:1028] T f9f1c40150da4144acb68186f9aad763 P 1288ca0989104bbea36e22f38c374218: Bootstrapping tablet I0703 01:42:48.361280 31496 ts_tablet_manager.cc:1173] T e0af316d40d143a8b387933151adf0e4 P de93b58c2d67488dae12285e1806ebac: Registered tablet (data state: TABLET_DATA_READY) I0703 01:42:48.361665 31497 ts_tablet_manager.cc:1173] T 3d7a6e87780a49949cf7e024817cf3a3 P de93b58c2d67488dae12285e1806ebac: Registered tablet (data state: TABLET_DATA_READY) I0703 01:42:48.362066 32257 tablet_bootstrap.cc:427] T f9f1c40150da4144acb68186f9aad763 P 1288ca0989104bbea36e22f38c374218: Bootstrap starting. I0703 01:42:48.362077 32258 ts_tablet_manager.cc:1028] T 3d7a6e87780a49949cf7e024817cf3a3 P 2f3fdb7c089640c0b936a574bc64707e: Bootstrapping tablet I0703 01:42:48.362211 32258 tablet_bootstrap.cc:427] T 3d7a6e87780a49949cf7e024817cf3a3 P 2f3fdb7c089640c0b936a574bc64707e: Bootstrap starting. I0703 01:42:48.362411 32259 ts_tablet_manager.cc:1028] T e0af316d40d143a8b387933151adf0e4 P de93b58c2d67488dae12285e1806ebac: Bootstrapping tablet I0703 01:42:48.362629 32259 tablet_bootstrap.cc:427] T e0af316d40d143a8b387933151adf0e4 P de93b58c2d67488dae12285e1806ebac: Bootstrap starting. I0703 01:42:48.363245 32257 tablet_bootstrap.cc:578] T f9f1c40150da4144acb68186f9aad763 P 1288ca0989104bbea36e22f38c374218: No blocks or log segments found. Creating new log. I0703 01:42:48.363842 32258 tablet_bootstrap.cc:578] T 3d7a6e87780a49949cf7e024817cf3a3 P 2f3fdb7c089640c0b936a574bc64707e: No blocks or log segments found. Creating new log. I0703 01:42:48.363919 32259 tablet_bootstrap.cc:578] T e0af316d40d143a8b387933151adf0e4 P de93b58c2d67488dae12285e1806ebac: No blocks or log segments found. Creating new log. I0703 01:42:48.365057 32257 tablet_bootstrap.cc:427] T f9f1c40150da4144acb68186f9aad763 P 1288ca0989104bbea36e22f38c374218: No bootstrap required, opened a new log I0703 01:42:48.365191 32257 ts_tablet_manager.cc:1045] T f9f1c40150da4144acb68186f9aad763 P 1288ca0989104bbea36e22f38c374218: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s I0703 01:42:48.365787 31375 ts_tablet_manager.cc:1173] T e0af316d40d143a8b387933151adf0e4 P 2f3fdb7c089640c0b936a574bc64707e: Registered tablet (data state: TABLET_DATA_READY) I0703 01:42:48.365859 32257 raft_consensus.cc:325] T f9f1c40150da4144acb68186f9aad763 P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } I0703 01:42:48.366168 32257 raft_consensus.cc:351] T f9f1c40150da4144acb68186f9aad763 P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:42:48.366235 32257 raft_consensus.cc:628] T f9f1c40150da4144acb68186f9aad763 P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1288ca0989104bbea36e22f38c374218, State: Initialized, Role: FOLLOWER I0703 01:42:48.366431 32257 consensus_queue.cc:227] T f9f1c40150da4144acb68186f9aad763 P 1288ca0989104bbea36e22f38c374218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } I0703 01:42:48.367038 32258 tablet_bootstrap.cc:427] T 3d7a6e87780a49949cf7e024817cf3a3 P 2f3fdb7c089640c0b936a574bc64707e: No bootstrap required, opened a new log I0703 01:42:48.367082 32257 ts_tablet_manager.cc:1073] T f9f1c40150da4144acb68186f9aad763 P 1288ca0989104bbea36e22f38c374218: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s I0703 01:42:48.367141 32258 ts_tablet_manager.cc:1045] T 3d7a6e87780a49949cf7e024817cf3a3 P 2f3fdb7c089640c0b936a574bc64707e: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s I0703 01:42:48.367768 32258 raft_consensus.cc:325] T 3d7a6e87780a49949cf7e024817cf3a3 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } I0703 01:42:48.367960 32258 raft_consensus.cc:351] T 3d7a6e87780a49949cf7e024817cf3a3 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:42:48.368011 32258 raft_consensus.cc:628] T 3d7a6e87780a49949cf7e024817cf3a3 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f3fdb7c089640c0b936a574bc64707e, State: Initialized, Role: FOLLOWER I0703 01:42:48.368161 32258 consensus_queue.cc:227] T 3d7a6e87780a49949cf7e024817cf3a3 P 2f3fdb7c089640c0b936a574bc64707e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } I0703 01:42:48.368806 32258 ts_tablet_manager.cc:1073] T 3d7a6e87780a49949cf7e024817cf3a3 P 2f3fdb7c089640c0b936a574bc64707e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s I0703 01:42:48.368963 32258 ts_tablet_manager.cc:1028] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e: Bootstrapping tablet I0703 01:42:48.369068 32258 tablet_bootstrap.cc:427] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e: Bootstrap starting. I0703 01:42:48.370007 32258 tablet_bootstrap.cc:578] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e: No blocks or log segments found. Creating new log. I0703 01:42:48.370105 32259 tablet_bootstrap.cc:427] T e0af316d40d143a8b387933151adf0e4 P de93b58c2d67488dae12285e1806ebac: No bootstrap required, opened a new log I0703 01:42:48.370204 32259 ts_tablet_manager.cc:1045] T e0af316d40d143a8b387933151adf0e4 P de93b58c2d67488dae12285e1806ebac: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.002s I0703 01:42:48.370800 32259 raft_consensus.cc:325] T e0af316d40d143a8b387933151adf0e4 P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } I0703 01:42:48.371417 32259 raft_consensus.cc:351] T e0af316d40d143a8b387933151adf0e4 P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:42:48.371496 32259 raft_consensus.cc:628] T e0af316d40d143a8b387933151adf0e4 P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de93b58c2d67488dae12285e1806ebac, State: Initialized, Role: FOLLOWER I0703 01:42:48.371709 32259 consensus_queue.cc:227] T e0af316d40d143a8b387933151adf0e4 P de93b58c2d67488dae12285e1806ebac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } I0703 01:42:48.371956 32258 tablet_bootstrap.cc:427] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e: No bootstrap required, opened a new log I0703 01:42:48.372043 32258 ts_tablet_manager.cc:1045] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s I0703 01:42:48.372458 32259 ts_tablet_manager.cc:1073] T e0af316d40d143a8b387933151adf0e4 P de93b58c2d67488dae12285e1806ebac: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s I0703 01:42:48.372468 32258 raft_consensus.cc:325] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } I0703 01:42:48.372623 32258 raft_consensus.cc:351] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:42:48.372601 32259 ts_tablet_manager.cc:1028] T 3d7a6e87780a49949cf7e024817cf3a3 P de93b58c2d67488dae12285e1806ebac: Bootstrapping tablet I0703 01:42:48.372673 32258 raft_consensus.cc:628] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f3fdb7c089640c0b936a574bc64707e, State: Initialized, Role: FOLLOWER I0703 01:42:48.372834 32258 consensus_queue.cc:227] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } I0703 01:42:48.373230 32259 tablet_bootstrap.cc:427] T 3d7a6e87780a49949cf7e024817cf3a3 P de93b58c2d67488dae12285e1806ebac: Bootstrap starting. I0703 01:42:48.373327 32258 ts_tablet_manager.cc:1073] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s I0703 01:42:48.373507 32258 ts_tablet_manager.cc:1028] T e0af316d40d143a8b387933151adf0e4 P 2f3fdb7c089640c0b936a574bc64707e: Bootstrapping tablet I0703 01:42:48.373630 32258 tablet_bootstrap.cc:427] T e0af316d40d143a8b387933151adf0e4 P 2f3fdb7c089640c0b936a574bc64707e: Bootstrap starting. I0703 01:42:48.374475 32258 tablet_bootstrap.cc:578] T e0af316d40d143a8b387933151adf0e4 P 2f3fdb7c089640c0b936a574bc64707e: No blocks or log segments found. Creating new log. I0703 01:42:48.375738 32259 tablet_bootstrap.cc:578] T 3d7a6e87780a49949cf7e024817cf3a3 P de93b58c2d67488dae12285e1806ebac: No blocks or log segments found. Creating new log. I0703 01:42:48.376235 32258 tablet_bootstrap.cc:427] T e0af316d40d143a8b387933151adf0e4 P 2f3fdb7c089640c0b936a574bc64707e: No bootstrap required, opened a new log I0703 01:42:48.376323 32258 ts_tablet_manager.cc:1045] T e0af316d40d143a8b387933151adf0e4 P 2f3fdb7c089640c0b936a574bc64707e: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s I0703 01:42:48.376947 32258 raft_consensus.cc:325] T e0af316d40d143a8b387933151adf0e4 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } I0703 01:42:48.377064 32258 raft_consensus.cc:351] T e0af316d40d143a8b387933151adf0e4 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:42:48.377112 32258 raft_consensus.cc:628] T e0af316d40d143a8b387933151adf0e4 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f3fdb7c089640c0b936a574bc64707e, State: Initialized, Role: FOLLOWER I0703 01:42:48.377279 32258 consensus_queue.cc:227] T e0af316d40d143a8b387933151adf0e4 P 2f3fdb7c089640c0b936a574bc64707e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } I0703 01:42:48.377985 32258 ts_tablet_manager.cc:1073] T e0af316d40d143a8b387933151adf0e4 P 2f3fdb7c089640c0b936a574bc64707e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s I0703 01:42:48.378226 32259 tablet_bootstrap.cc:427] T 3d7a6e87780a49949cf7e024817cf3a3 P de93b58c2d67488dae12285e1806ebac: No bootstrap required, opened a new log I0703 01:42:48.378334 32259 ts_tablet_manager.cc:1045] T 3d7a6e87780a49949cf7e024817cf3a3 P de93b58c2d67488dae12285e1806ebac: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s I0703 01:42:48.378973 32259 raft_consensus.cc:325] T 3d7a6e87780a49949cf7e024817cf3a3 P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } I0703 01:42:48.379144 32259 raft_consensus.cc:351] T 3d7a6e87780a49949cf7e024817cf3a3 P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:42:48.379199 32259 raft_consensus.cc:628] T 3d7a6e87780a49949cf7e024817cf3a3 P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de93b58c2d67488dae12285e1806ebac, State: Initialized, Role: FOLLOWER I0703 01:42:48.379374 32259 consensus_queue.cc:227] T 3d7a6e87780a49949cf7e024817cf3a3 P de93b58c2d67488dae12285e1806ebac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } I0703 01:42:48.380363 32259 ts_tablet_manager.cc:1073] T 3d7a6e87780a49949cf7e024817cf3a3 P de93b58c2d67488dae12285e1806ebac: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s I0703 01:42:48.389611 32195 raft_consensus.cc:457] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0703 01:42:48.389667 32195 raft_consensus.cc:2730] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e: Snoozing failure detection for 1.975s (starting election) I0703 01:42:48.389745 32195 raft_consensus.cc:479] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } I0703 01:42:48.390034 32195 leader_election.cc:243] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521) I0703 01:42:48.390707 31272 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "f9f1c40150da4144acb68186f9aad763" candidate_uuid: "2f3fdb7c089640c0b936a574bc64707e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1288ca0989104bbea36e22f38c374218" is_pre_election: true I0703 01:42:48.390832 31272 raft_consensus.cc:2730] T f9f1c40150da4144acb68186f9aad763 P 1288ca0989104bbea36e22f38c374218: Snoozing failure detection for 1.921s (vote granted) I0703 01:42:48.390892 31272 raft_consensus.cc:2243] T f9f1c40150da4144acb68186f9aad763 P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f3fdb7c089640c0b936a574bc64707e in term 0. I0703 01:42:48.391324 31327 leader_election.cc:379] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e [CANDIDATE]: Term 1 pre-election: Vote granted by peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521) I0703 01:42:48.391381 31327 leader_election.cc:270] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0703 01:42:48.391587 32195 raft_consensus.cc:2730] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e: Snoozing failure detection for 1.840s (election complete) I0703 01:42:48.391655 32195 raft_consensus.cc:2537] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Leader pre-election won for term 1 I0703 01:42:48.391710 32195 raft_consensus.cc:457] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0703 01:42:48.391763 32195 raft_consensus.cc:2730] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e: Snoozing failure detection for 1.811s (starting election) I0703 01:42:48.391825 32195 raft_consensus.cc:2782] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Advancing to term 1 I0703 01:42:48.392954 32195 raft_consensus.cc:479] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } I0703 01:42:48.393208 32195 leader_election.cc:243] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e [CANDIDATE]: Term 1 election: Requesting vote from peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521) I0703 01:42:48.393827 31272 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "f9f1c40150da4144acb68186f9aad763" candidate_uuid: "2f3fdb7c089640c0b936a574bc64707e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1288ca0989104bbea36e22f38c374218" I0703 01:42:48.393926 31272 raft_consensus.cc:2782] T f9f1c40150da4144acb68186f9aad763 P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:42:48.393992 31272 raft_consensus.cc:2730] T f9f1c40150da4144acb68186f9aad763 P 1288ca0989104bbea36e22f38c374218: Snoozing failure detection for 1.843s (vote granted) I0703 01:42:48.395175 31272 raft_consensus.cc:2243] T f9f1c40150da4144acb68186f9aad763 P 1288ca0989104bbea36e22f38c374218 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f3fdb7c089640c0b936a574bc64707e in term 1. I0703 01:42:48.395663 31327 leader_election.cc:379] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e [CANDIDATE]: Term 1 election: Vote granted by peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521) I0703 01:42:48.395717 31327 leader_election.cc:270] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:42:48.395892 32195 raft_consensus.cc:2730] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e: Snoozing failure detection for 2.103s (election complete) I0703 01:42:48.395962 32195 raft_consensus.cc:2537] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:42:48.396077 32195 raft_consensus.cc:593] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e [term 1 LEADER]: Becoming Leader. State: Replica: 2f3fdb7c089640c0b936a574bc64707e, State: Running, Role: LEADER I0703 01:42:48.396275 32195 consensus_queue.cc:204] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } I0703 01:42:48.397938 31146 catalog_manager.cc:3798] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e reported cstate change: term changed from 0 to 1, leader changed from to 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194). New cstate: current_term: 1 leader_uuid: "2f3fdb7c089640c0b936a574bc64707e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } } W0703 01:42:48.490991 31328 consensus_peers.cc:433] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 56eaab17f812433a8f7880f0744e1744 (127.5.240.199:57714): Couldn't send request to peer 56eaab17f812433a8f7880f0744e1744 for tablet 454bab377db147a1a2f8241ecc2241f3. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.199:57714: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 14 times. I0703 01:42:48.526654 32181 raft_consensus.cc:457] T e0af316d40d143a8b387933151adf0e4 P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0703 01:42:48.526722 32181 raft_consensus.cc:2730] T e0af316d40d143a8b387933151adf0e4 P de93b58c2d67488dae12285e1806ebac: Snoozing failure detection for 1.982s (starting election) I0703 01:42:48.526792 32181 raft_consensus.cc:479] T e0af316d40d143a8b387933151adf0e4 P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } I0703 01:42:48.527155 32181 leader_election.cc:243] T e0af316d40d143a8b387933151adf0e4 P de93b58c2d67488dae12285e1806ebac [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194:39827) I0703 01:42:48.527833 31395 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "e0af316d40d143a8b387933151adf0e4" candidate_uuid: "de93b58c2d67488dae12285e1806ebac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f3fdb7c089640c0b936a574bc64707e" is_pre_election: true I0703 01:42:48.527959 31395 raft_consensus.cc:2730] T e0af316d40d143a8b387933151adf0e4 P 2f3fdb7c089640c0b936a574bc64707e: Snoozing failure detection for 2.219s (vote granted) I0703 01:42:48.528025 31395 raft_consensus.cc:2243] T e0af316d40d143a8b387933151adf0e4 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de93b58c2d67488dae12285e1806ebac in term 0. I0703 01:42:48.528172 32195 raft_consensus.cc:457] T 3d7a6e87780a49949cf7e024817cf3a3 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0703 01:42:48.528287 32195 raft_consensus.cc:2730] T 3d7a6e87780a49949cf7e024817cf3a3 P 2f3fdb7c089640c0b936a574bc64707e: Snoozing failure detection for 2.054s (starting election) I0703 01:42:48.528388 32195 raft_consensus.cc:479] T 3d7a6e87780a49949cf7e024817cf3a3 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } I0703 01:42:48.528563 31448 leader_election.cc:379] T e0af316d40d143a8b387933151adf0e4 P de93b58c2d67488dae12285e1806ebac [CANDIDATE]: Term 1 pre-election: Vote granted by peer 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194:39827) I0703 01:42:48.528650 31448 leader_election.cc:270] T e0af316d40d143a8b387933151adf0e4 P de93b58c2d67488dae12285e1806ebac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0703 01:42:48.528738 32195 leader_election.cc:243] T 3d7a6e87780a49949cf7e024817cf3a3 P 2f3fdb7c089640c0b936a574bc64707e [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321) I0703 01:42:48.528844 32181 raft_consensus.cc:2730] T e0af316d40d143a8b387933151adf0e4 P de93b58c2d67488dae12285e1806ebac: Snoozing failure detection for 1.618s (election complete) I0703 01:42:48.528913 32181 raft_consensus.cc:2537] T e0af316d40d143a8b387933151adf0e4 P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Leader pre-election won for term 1 I0703 01:42:48.528980 32181 raft_consensus.cc:457] T e0af316d40d143a8b387933151adf0e4 P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0703 01:42:48.529036 32181 raft_consensus.cc:2730] T e0af316d40d143a8b387933151adf0e4 P de93b58c2d67488dae12285e1806ebac: Snoozing failure detection for 1.546s (starting election) I0703 01:42:48.529134 32181 raft_consensus.cc:2782] T e0af316d40d143a8b387933151adf0e4 P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Advancing to term 1 I0703 01:42:48.529472 31517 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "3d7a6e87780a49949cf7e024817cf3a3" candidate_uuid: "2f3fdb7c089640c0b936a574bc64707e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de93b58c2d67488dae12285e1806ebac" is_pre_election: true I0703 01:42:48.529660 31517 raft_consensus.cc:2730] T 3d7a6e87780a49949cf7e024817cf3a3 P de93b58c2d67488dae12285e1806ebac: Snoozing failure detection for 1.817s (vote granted) I0703 01:42:48.529757 31517 raft_consensus.cc:2243] T 3d7a6e87780a49949cf7e024817cf3a3 P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f3fdb7c089640c0b936a574bc64707e in term 0. I0703 01:42:48.530302 31329 leader_election.cc:379] T 3d7a6e87780a49949cf7e024817cf3a3 P 2f3fdb7c089640c0b936a574bc64707e [CANDIDATE]: Term 1 pre-election: Vote granted by peer de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321) I0703 01:42:48.530359 31329 leader_election.cc:270] T 3d7a6e87780a49949cf7e024817cf3a3 P 2f3fdb7c089640c0b936a574bc64707e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0703 01:42:48.530550 32195 raft_consensus.cc:2730] T 3d7a6e87780a49949cf7e024817cf3a3 P 2f3fdb7c089640c0b936a574bc64707e: Snoozing failure detection for 1.604s (election complete) I0703 01:42:48.530611 32195 raft_consensus.cc:2537] T 3d7a6e87780a49949cf7e024817cf3a3 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Leader pre-election won for term 1 I0703 01:42:48.530661 32195 raft_consensus.cc:457] T 3d7a6e87780a49949cf7e024817cf3a3 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0703 01:42:48.530722 32195 raft_consensus.cc:2730] T 3d7a6e87780a49949cf7e024817cf3a3 P 2f3fdb7c089640c0b936a574bc64707e: Snoozing failure detection for 1.621s (starting election) I0703 01:42:48.530777 32195 raft_consensus.cc:2782] T 3d7a6e87780a49949cf7e024817cf3a3 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Advancing to term 1 I0703 01:42:48.531168 32181 raft_consensus.cc:479] T e0af316d40d143a8b387933151adf0e4 P de93b58c2d67488dae12285e1806ebac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } I0703 01:42:48.531406 32181 leader_election.cc:243] T e0af316d40d143a8b387933151adf0e4 P de93b58c2d67488dae12285e1806ebac [CANDIDATE]: Term 1 election: Requesting vote from peer 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194:39827) I0703 01:42:48.532125 31395 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "e0af316d40d143a8b387933151adf0e4" candidate_uuid: "de93b58c2d67488dae12285e1806ebac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f3fdb7c089640c0b936a574bc64707e" I0703 01:42:48.532143 32195 raft_consensus.cc:479] T 3d7a6e87780a49949cf7e024817cf3a3 P 2f3fdb7c089640c0b936a574bc64707e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } I0703 01:42:48.532302 31395 raft_consensus.cc:2782] T e0af316d40d143a8b387933151adf0e4 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Advancing to term 1 I0703 01:42:48.532371 31395 raft_consensus.cc:2730] T e0af316d40d143a8b387933151adf0e4 P 2f3fdb7c089640c0b936a574bc64707e: Snoozing failure detection for 2.172s (vote granted) I0703 01:42:48.532515 32195 leader_election.cc:243] T 3d7a6e87780a49949cf7e024817cf3a3 P 2f3fdb7c089640c0b936a574bc64707e [CANDIDATE]: Term 1 election: Requesting vote from peer de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321) I0703 01:42:48.533277 31517 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "3d7a6e87780a49949cf7e024817cf3a3" candidate_uuid: "2f3fdb7c089640c0b936a574bc64707e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de93b58c2d67488dae12285e1806ebac" I0703 01:42:48.533411 31517 raft_consensus.cc:2782] T 3d7a6e87780a49949cf7e024817cf3a3 P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Advancing to term 1 I0703 01:42:48.533481 31517 raft_consensus.cc:2730] T 3d7a6e87780a49949cf7e024817cf3a3 P de93b58c2d67488dae12285e1806ebac: Snoozing failure detection for 2.077s (vote granted) I0703 01:42:48.534324 31395 raft_consensus.cc:2243] T e0af316d40d143a8b387933151adf0e4 P 2f3fdb7c089640c0b936a574bc64707e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de93b58c2d67488dae12285e1806ebac in term 1. I0703 01:42:48.534893 31448 leader_election.cc:379] T e0af316d40d143a8b387933151adf0e4 P de93b58c2d67488dae12285e1806ebac [CANDIDATE]: Term 1 election: Vote granted by peer 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194:39827) I0703 01:42:48.534986 31448 leader_election.cc:270] T e0af316d40d143a8b387933151adf0e4 P de93b58c2d67488dae12285e1806ebac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:42:48.535233 32181 raft_consensus.cc:2730] T e0af316d40d143a8b387933151adf0e4 P de93b58c2d67488dae12285e1806ebac: Snoozing failure detection for 1.951s (election complete) I0703 01:42:48.535387 32181 raft_consensus.cc:2537] T e0af316d40d143a8b387933151adf0e4 P de93b58c2d67488dae12285e1806ebac [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:42:48.535389 31517 raft_consensus.cc:2243] T 3d7a6e87780a49949cf7e024817cf3a3 P de93b58c2d67488dae12285e1806ebac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f3fdb7c089640c0b936a574bc64707e in term 1. I0703 01:42:48.535850 32181 raft_consensus.cc:593] T e0af316d40d143a8b387933151adf0e4 P de93b58c2d67488dae12285e1806ebac [term 1 LEADER]: Becoming Leader. State: Replica: de93b58c2d67488dae12285e1806ebac, State: Running, Role: LEADER I0703 01:42:48.535909 31329 leader_election.cc:379] T 3d7a6e87780a49949cf7e024817cf3a3 P 2f3fdb7c089640c0b936a574bc64707e [CANDIDATE]: Term 1 election: Vote granted by peer de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321) I0703 01:42:48.535964 31329 leader_election.cc:270] T 3d7a6e87780a49949cf7e024817cf3a3 P 2f3fdb7c089640c0b936a574bc64707e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:42:48.536154 32195 raft_consensus.cc:2730] T 3d7a6e87780a49949cf7e024817cf3a3 P 2f3fdb7c089640c0b936a574bc64707e: Snoozing failure detection for 1.814s (election complete) I0703 01:42:48.536139 32181 consensus_queue.cc:204] T e0af316d40d143a8b387933151adf0e4 P de93b58c2d67488dae12285e1806ebac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } I0703 01:42:48.536239 32195 raft_consensus.cc:2537] T 3d7a6e87780a49949cf7e024817cf3a3 P 2f3fdb7c089640c0b936a574bc64707e [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:42:48.536327 32195 raft_consensus.cc:593] T 3d7a6e87780a49949cf7e024817cf3a3 P 2f3fdb7c089640c0b936a574bc64707e [term 1 LEADER]: Becoming Leader. State: Replica: 2f3fdb7c089640c0b936a574bc64707e, State: Running, Role: LEADER I0703 01:42:48.536577 32195 consensus_queue.cc:204] T 3d7a6e87780a49949cf7e024817cf3a3 P 2f3fdb7c089640c0b936a574bc64707e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } I0703 01:42:48.538192 31146 catalog_manager.cc:3798] T e0af316d40d143a8b387933151adf0e4 P de93b58c2d67488dae12285e1806ebac reported cstate change: term changed from 0 to 1, leader changed from to de93b58c2d67488dae12285e1806ebac (127.5.240.195). New cstate: current_term: 1 leader_uuid: "de93b58c2d67488dae12285e1806ebac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } } I0703 01:42:48.538444 31152 catalog_manager.cc:3798] T 3d7a6e87780a49949cf7e024817cf3a3 P 2f3fdb7c089640c0b936a574bc64707e reported cstate change: term changed from 0 to 1, leader changed from to 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194). New cstate: current_term: 1 leader_uuid: "2f3fdb7c089640c0b936a574bc64707e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } } W0703 01:42:48.760910 32270 flags.cc:406] Enabled unsafe flag: --never_fsync=true Using table 'rebalance_test_table_3' I0703 01:42:48.901845 31395 raft_consensus.cc:1081] T e0af316d40d143a8b387933151adf0e4 P 2f3fdb7c089640c0b936a574bc64707e [term 1 FOLLOWER]: Refusing update from remote peer de93b58c2d67488dae12285e1806ebac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0703 01:42:48.902807 32181 consensus_queue.cc:962] T e0af316d40d143a8b387933151adf0e4 P de93b58c2d67488dae12285e1806ebac [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0703 01:42:48.908465 31517 raft_consensus.cc:1081] T 3d7a6e87780a49949cf7e024817cf3a3 P de93b58c2d67488dae12285e1806ebac [term 1 FOLLOWER]: Refusing update from remote peer 2f3fdb7c089640c0b936a574bc64707e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0703 01:42:48.909667 32195 consensus_queue.cc:962] T 3d7a6e87780a49949cf7e024817cf3a3 P 2f3fdb7c089640c0b936a574bc64707e [LEADER]: Connected to new peer: Peer: permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0703 01:42:48.909844 31272 raft_consensus.cc:1081] T f9f1c40150da4144acb68186f9aad763 P 1288ca0989104bbea36e22f38c374218 [term 1 FOLLOWER]: Refusing update from remote peer 2f3fdb7c089640c0b936a574bc64707e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0703 01:42:48.911101 32195 consensus_queue.cc:962] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s W0703 01:42:48.965279 31328 consensus_peers.cc:433] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 56eaab17f812433a8f7880f0744e1744 (127.5.240.199:57714): Couldn't send request to peer 56eaab17f812433a8f7880f0744e1744 for tablet 454bab377db147a1a2f8241ecc2241f3. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.199:57714: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 15 times. Generator report time total : 200.582 ms time per row: 0.100291 ms I0703 01:42:49.054468 31152 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.5.240.254:51772: name: "rebalance_test_table_4" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 2 split_rows_range_bounds { } partition_schema { hash_bucket_schemas { columns { name: "key" } num_buckets: 3 seed: 0 } } I0703 01:42:49.061192 31252 tablet_service.cc:759] Processing CreateTablet for tablet bbe64aa4b0e04f468b035048629b969b (table=rebalance_test_table_4 [id=b0493a1d5ef04606a6d4de013749c53b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I0703 01:42:49.061511 31252 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet bbe64aa4b0e04f468b035048629b969b. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:49.061460 31496 tablet_service.cc:759] Processing CreateTablet for tablet bbe64aa4b0e04f468b035048629b969b (table=rebalance_test_table_4 [id=b0493a1d5ef04606a6d4de013749c53b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I0703 01:42:49.061836 31496 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet bbe64aa4b0e04f468b035048629b969b. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:49.062434 31375 tablet_service.cc:759] Processing CreateTablet for tablet 73518ef45438415da1300ef5004ea7d5 (table=rebalance_test_table_4 [id=b0493a1d5ef04606a6d4de013749c53b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I0703 01:42:49.062773 31375 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 73518ef45438415da1300ef5004ea7d5. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:49.062819 31497 tablet_service.cc:759] Processing CreateTablet for tablet ffe66a5cfb564964951c619e51e7c47c (table=rebalance_test_table_4 [id=b0493a1d5ef04606a6d4de013749c53b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I0703 01:42:49.063113 31497 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet ffe66a5cfb564964951c619e51e7c47c. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:49.066000 31252 ts_tablet_manager.cc:1173] T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218: Registered tablet (data state: TABLET_DATA_READY) I0703 01:42:49.067091 31375 ts_tablet_manager.cc:1173] T 73518ef45438415da1300ef5004ea7d5 P 2f3fdb7c089640c0b936a574bc64707e: Registered tablet (data state: TABLET_DATA_READY) I0703 01:42:49.067977 31496 ts_tablet_manager.cc:1173] T bbe64aa4b0e04f468b035048629b969b P de93b58c2d67488dae12285e1806ebac: Registered tablet (data state: TABLET_DATA_READY) I0703 01:42:49.067952 31251 tablet_service.cc:759] Processing CreateTablet for tablet ffe66a5cfb564964951c619e51e7c47c (table=rebalance_test_table_4 [id=b0493a1d5ef04606a6d4de013749c53b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I0703 01:42:49.068056 31250 tablet_service.cc:759] Processing CreateTablet for tablet 73518ef45438415da1300ef5004ea7d5 (table=rebalance_test_table_4 [id=b0493a1d5ef04606a6d4de013749c53b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I0703 01:42:49.068229 31251 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet ffe66a5cfb564964951c619e51e7c47c. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:49.068804 31250 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 73518ef45438415da1300ef5004ea7d5. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:49.071074 32307 ts_tablet_manager.cc:1028] T 73518ef45438415da1300ef5004ea7d5 P 2f3fdb7c089640c0b936a574bc64707e: Bootstrapping tablet I0703 01:42:49.071210 32307 tablet_bootstrap.cc:427] T 73518ef45438415da1300ef5004ea7d5 P 2f3fdb7c089640c0b936a574bc64707e: Bootstrap starting. I0703 01:42:49.071581 32308 ts_tablet_manager.cc:1028] T bbe64aa4b0e04f468b035048629b969b P de93b58c2d67488dae12285e1806ebac: Bootstrapping tablet I0703 01:42:49.071727 32308 tablet_bootstrap.cc:427] T bbe64aa4b0e04f468b035048629b969b P de93b58c2d67488dae12285e1806ebac: Bootstrap starting. I0703 01:42:49.072635 32307 tablet_bootstrap.cc:578] T 73518ef45438415da1300ef5004ea7d5 P 2f3fdb7c089640c0b936a574bc64707e: No blocks or log segments found. Creating new log. I0703 01:42:49.073096 32308 tablet_bootstrap.cc:578] T bbe64aa4b0e04f468b035048629b969b P de93b58c2d67488dae12285e1806ebac: No blocks or log segments found. Creating new log. I0703 01:42:49.074652 31251 ts_tablet_manager.cc:1173] T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218: Registered tablet (data state: TABLET_DATA_READY) I0703 01:42:49.075244 32306 ts_tablet_manager.cc:1028] T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218: Bootstrapping tablet I0703 01:42:49.075412 32306 tablet_bootstrap.cc:427] T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218: Bootstrap starting. I0703 01:42:49.075601 31250 ts_tablet_manager.cc:1173] T 73518ef45438415da1300ef5004ea7d5 P 1288ca0989104bbea36e22f38c374218: Registered tablet (data state: TABLET_DATA_READY) I0703 01:42:49.075660 31497 ts_tablet_manager.cc:1173] T ffe66a5cfb564964951c619e51e7c47c P de93b58c2d67488dae12285e1806ebac: Registered tablet (data state: TABLET_DATA_READY) I0703 01:42:49.076346 32307 tablet_bootstrap.cc:427] T 73518ef45438415da1300ef5004ea7d5 P 2f3fdb7c089640c0b936a574bc64707e: No bootstrap required, opened a new log I0703 01:42:49.076444 32307 ts_tablet_manager.cc:1045] T 73518ef45438415da1300ef5004ea7d5 P 2f3fdb7c089640c0b936a574bc64707e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s I0703 01:42:49.076741 32308 tablet_bootstrap.cc:427] T bbe64aa4b0e04f468b035048629b969b P de93b58c2d67488dae12285e1806ebac: No bootstrap required, opened a new log I0703 01:42:49.076828 32308 ts_tablet_manager.cc:1045] T bbe64aa4b0e04f468b035048629b969b P de93b58c2d67488dae12285e1806ebac: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s I0703 01:42:49.077013 32307 raft_consensus.cc:325] T 73518ef45438415da1300ef5004ea7d5 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } I0703 01:42:49.077162 32307 raft_consensus.cc:351] T 73518ef45438415da1300ef5004ea7d5 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:42:49.077211 32307 raft_consensus.cc:628] T 73518ef45438415da1300ef5004ea7d5 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f3fdb7c089640c0b936a574bc64707e, State: Initialized, Role: FOLLOWER I0703 01:42:49.077316 32308 raft_consensus.cc:325] T bbe64aa4b0e04f468b035048629b969b P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } I0703 01:42:49.077373 32307 consensus_queue.cc:227] T 73518ef45438415da1300ef5004ea7d5 P 2f3fdb7c089640c0b936a574bc64707e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } I0703 01:42:49.077463 32308 raft_consensus.cc:351] T bbe64aa4b0e04f468b035048629b969b P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:42:49.077495 32308 raft_consensus.cc:628] T bbe64aa4b0e04f468b035048629b969b P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de93b58c2d67488dae12285e1806ebac, State: Initialized, Role: FOLLOWER I0703 01:42:49.077657 32308 consensus_queue.cc:227] T bbe64aa4b0e04f468b035048629b969b P de93b58c2d67488dae12285e1806ebac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } I0703 01:42:49.078254 32307 ts_tablet_manager.cc:1073] T 73518ef45438415da1300ef5004ea7d5 P 2f3fdb7c089640c0b936a574bc64707e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s I0703 01:42:49.078302 32308 ts_tablet_manager.cc:1073] T bbe64aa4b0e04f468b035048629b969b P de93b58c2d67488dae12285e1806ebac: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s I0703 01:42:49.078436 32308 ts_tablet_manager.cc:1028] T ffe66a5cfb564964951c619e51e7c47c P de93b58c2d67488dae12285e1806ebac: Bootstrapping tablet I0703 01:42:49.078583 32308 tablet_bootstrap.cc:427] T ffe66a5cfb564964951c619e51e7c47c P de93b58c2d67488dae12285e1806ebac: Bootstrap starting. I0703 01:42:49.078938 32306 tablet_bootstrap.cc:578] T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218: No blocks or log segments found. Creating new log. I0703 01:42:49.079741 32308 tablet_bootstrap.cc:578] T ffe66a5cfb564964951c619e51e7c47c P de93b58c2d67488dae12285e1806ebac: No blocks or log segments found. Creating new log. I0703 01:42:49.081765 32306 tablet_bootstrap.cc:427] T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218: No bootstrap required, opened a new log I0703 01:42:49.081869 32306 ts_tablet_manager.cc:1045] T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.001s I0703 01:42:49.081883 32308 tablet_bootstrap.cc:427] T ffe66a5cfb564964951c619e51e7c47c P de93b58c2d67488dae12285e1806ebac: No bootstrap required, opened a new log I0703 01:42:49.081984 32308 ts_tablet_manager.cc:1045] T ffe66a5cfb564964951c619e51e7c47c P de93b58c2d67488dae12285e1806ebac: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s I0703 01:42:49.082509 32306 raft_consensus.cc:325] T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } I0703 01:42:49.082710 32306 raft_consensus.cc:351] T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:42:49.082769 32306 raft_consensus.cc:628] T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1288ca0989104bbea36e22f38c374218, State: Initialized, Role: FOLLOWER I0703 01:42:49.082736 32308 raft_consensus.cc:325] T ffe66a5cfb564964951c619e51e7c47c P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } I0703 01:42:49.082888 32308 raft_consensus.cc:351] T ffe66a5cfb564964951c619e51e7c47c P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:42:49.082942 32308 raft_consensus.cc:628] T ffe66a5cfb564964951c619e51e7c47c P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de93b58c2d67488dae12285e1806ebac, State: Initialized, Role: FOLLOWER I0703 01:42:49.082957 32306 consensus_queue.cc:227] T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } I0703 01:42:49.083097 32308 consensus_queue.cc:227] T ffe66a5cfb564964951c619e51e7c47c P de93b58c2d67488dae12285e1806ebac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } I0703 01:42:49.083662 32306 ts_tablet_manager.cc:1073] T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s I0703 01:42:49.083756 32308 ts_tablet_manager.cc:1073] T ffe66a5cfb564964951c619e51e7c47c P de93b58c2d67488dae12285e1806ebac: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s I0703 01:42:49.083842 32306 ts_tablet_manager.cc:1028] T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218: Bootstrapping tablet I0703 01:42:49.083969 32306 tablet_bootstrap.cc:427] T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218: Bootstrap starting. I0703 01:42:49.084990 32306 tablet_bootstrap.cc:578] T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218: No blocks or log segments found. Creating new log. I0703 01:42:49.086709 32306 tablet_bootstrap.cc:427] T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218: No bootstrap required, opened a new log I0703 01:42:49.086828 32306 ts_tablet_manager.cc:1045] T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.002s I0703 01:42:49.087265 32306 raft_consensus.cc:325] T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } I0703 01:42:49.087378 32306 raft_consensus.cc:351] T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:42:49.087430 32306 raft_consensus.cc:628] T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1288ca0989104bbea36e22f38c374218, State: Initialized, Role: FOLLOWER I0703 01:42:49.087589 32306 consensus_queue.cc:227] T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } I0703 01:42:49.087985 32306 ts_tablet_manager.cc:1073] T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s I0703 01:42:49.088158 32306 ts_tablet_manager.cc:1028] T 73518ef45438415da1300ef5004ea7d5 P 1288ca0989104bbea36e22f38c374218: Bootstrapping tablet I0703 01:42:49.088253 32306 tablet_bootstrap.cc:427] T 73518ef45438415da1300ef5004ea7d5 P 1288ca0989104bbea36e22f38c374218: Bootstrap starting. I0703 01:42:49.089026 32306 tablet_bootstrap.cc:578] T 73518ef45438415da1300ef5004ea7d5 P 1288ca0989104bbea36e22f38c374218: No blocks or log segments found. Creating new log. I0703 01:42:49.090819 32306 tablet_bootstrap.cc:427] T 73518ef45438415da1300ef5004ea7d5 P 1288ca0989104bbea36e22f38c374218: No bootstrap required, opened a new log I0703 01:42:49.090907 32306 ts_tablet_manager.cc:1045] T 73518ef45438415da1300ef5004ea7d5 P 1288ca0989104bbea36e22f38c374218: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s I0703 01:42:49.091320 32306 raft_consensus.cc:325] T 73518ef45438415da1300ef5004ea7d5 P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } I0703 01:42:49.091439 32306 raft_consensus.cc:351] T 73518ef45438415da1300ef5004ea7d5 P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:42:49.091490 32306 raft_consensus.cc:628] T 73518ef45438415da1300ef5004ea7d5 P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1288ca0989104bbea36e22f38c374218, State: Initialized, Role: FOLLOWER I0703 01:42:49.091696 32306 consensus_queue.cc:227] T 73518ef45438415da1300ef5004ea7d5 P 1288ca0989104bbea36e22f38c374218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } I0703 01:42:49.092103 32306 ts_tablet_manager.cc:1073] T 73518ef45438415da1300ef5004ea7d5 P 1288ca0989104bbea36e22f38c374218: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s I0703 01:42:49.101999 32169 raft_consensus.cc:457] T 73518ef45438415da1300ef5004ea7d5 P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0703 01:42:49.102052 32169 raft_consensus.cc:2730] T 73518ef45438415da1300ef5004ea7d5 P 1288ca0989104bbea36e22f38c374218: Snoozing failure detection for 2.025s (starting election) I0703 01:42:49.102114 32169 raft_consensus.cc:479] T 73518ef45438415da1300ef5004ea7d5 P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } I0703 01:42:49.102389 32169 leader_election.cc:243] T 73518ef45438415da1300ef5004ea7d5 P 1288ca0989104bbea36e22f38c374218 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194:39827) I0703 01:42:49.106703 31395 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "73518ef45438415da1300ef5004ea7d5" candidate_uuid: "1288ca0989104bbea36e22f38c374218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f3fdb7c089640c0b936a574bc64707e" is_pre_election: true I0703 01:42:49.106812 31395 raft_consensus.cc:2730] T 73518ef45438415da1300ef5004ea7d5 P 2f3fdb7c089640c0b936a574bc64707e: Snoozing failure detection for 1.935s (vote granted) I0703 01:42:49.106868 31395 raft_consensus.cc:2243] T 73518ef45438415da1300ef5004ea7d5 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1288ca0989104bbea36e22f38c374218 in term 0. I0703 01:42:49.107218 31203 leader_election.cc:379] T 73518ef45438415da1300ef5004ea7d5 P 1288ca0989104bbea36e22f38c374218 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194:39827) I0703 01:42:49.107270 31203 leader_election.cc:270] T 73518ef45438415da1300ef5004ea7d5 P 1288ca0989104bbea36e22f38c374218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0703 01:42:49.107446 32169 raft_consensus.cc:2730] T 73518ef45438415da1300ef5004ea7d5 P 1288ca0989104bbea36e22f38c374218: Snoozing failure detection for 2.172s (election complete) I0703 01:42:49.107503 32169 raft_consensus.cc:2537] T 73518ef45438415da1300ef5004ea7d5 P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0703 01:42:49.107545 32169 raft_consensus.cc:457] T 73518ef45438415da1300ef5004ea7d5 P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0703 01:42:49.107576 32169 raft_consensus.cc:2730] T 73518ef45438415da1300ef5004ea7d5 P 1288ca0989104bbea36e22f38c374218: Snoozing failure detection for 1.687s (starting election) I0703 01:42:49.107614 32169 raft_consensus.cc:2782] T 73518ef45438415da1300ef5004ea7d5 P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:42:49.108736 32169 raft_consensus.cc:479] T 73518ef45438415da1300ef5004ea7d5 P 1288ca0989104bbea36e22f38c374218 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } I0703 01:42:49.108984 32169 leader_election.cc:243] T 73518ef45438415da1300ef5004ea7d5 P 1288ca0989104bbea36e22f38c374218 [CANDIDATE]: Term 1 election: Requesting vote from peer 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194:39827) I0703 01:42:49.109576 31395 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "73518ef45438415da1300ef5004ea7d5" candidate_uuid: "1288ca0989104bbea36e22f38c374218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f3fdb7c089640c0b936a574bc64707e" I0703 01:42:49.109704 31395 raft_consensus.cc:2782] T 73518ef45438415da1300ef5004ea7d5 P 2f3fdb7c089640c0b936a574bc64707e [term 0 FOLLOWER]: Advancing to term 1 I0703 01:42:49.109772 31395 raft_consensus.cc:2730] T 73518ef45438415da1300ef5004ea7d5 P 2f3fdb7c089640c0b936a574bc64707e: Snoozing failure detection for 1.763s (vote granted) I0703 01:42:49.110966 31395 raft_consensus.cc:2243] T 73518ef45438415da1300ef5004ea7d5 P 2f3fdb7c089640c0b936a574bc64707e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1288ca0989104bbea36e22f38c374218 in term 1. I0703 01:42:49.111359 31203 leader_election.cc:379] T 73518ef45438415da1300ef5004ea7d5 P 1288ca0989104bbea36e22f38c374218 [CANDIDATE]: Term 1 election: Vote granted by peer 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194:39827) I0703 01:42:49.111413 31203 leader_election.cc:270] T 73518ef45438415da1300ef5004ea7d5 P 1288ca0989104bbea36e22f38c374218 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:42:49.111599 32169 raft_consensus.cc:2730] T 73518ef45438415da1300ef5004ea7d5 P 1288ca0989104bbea36e22f38c374218: Snoozing failure detection for 2.194s (election complete) I0703 01:42:49.111688 32169 raft_consensus.cc:2537] T 73518ef45438415da1300ef5004ea7d5 P 1288ca0989104bbea36e22f38c374218 [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:42:49.111955 32169 raft_consensus.cc:593] T 73518ef45438415da1300ef5004ea7d5 P 1288ca0989104bbea36e22f38c374218 [term 1 LEADER]: Becoming Leader. State: Replica: 1288ca0989104bbea36e22f38c374218, State: Running, Role: LEADER I0703 01:42:49.112123 32169 consensus_queue.cc:204] T 73518ef45438415da1300ef5004ea7d5 P 1288ca0989104bbea36e22f38c374218 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } I0703 01:42:49.113610 31147 catalog_manager.cc:3798] T 73518ef45438415da1300ef5004ea7d5 P 1288ca0989104bbea36e22f38c374218 reported cstate change: term changed from 0 to 1, leader changed from to 1288ca0989104bbea36e22f38c374218 (127.5.240.193). New cstate: current_term: 1 leader_uuid: "1288ca0989104bbea36e22f38c374218" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } } I0703 01:42:49.194979 32169 raft_consensus.cc:457] T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0703 01:42:49.195046 32169 raft_consensus.cc:2730] T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218: Snoozing failure detection for 2.087s (starting election) I0703 01:42:49.195107 32169 raft_consensus.cc:479] T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } I0703 01:42:49.195395 32169 leader_election.cc:243] T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321) I0703 01:42:49.196059 31517 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "bbe64aa4b0e04f468b035048629b969b" candidate_uuid: "1288ca0989104bbea36e22f38c374218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de93b58c2d67488dae12285e1806ebac" is_pre_election: true I0703 01:42:49.196182 31517 raft_consensus.cc:2730] T bbe64aa4b0e04f468b035048629b969b P de93b58c2d67488dae12285e1806ebac: Snoozing failure detection for 2.171s (vote granted) I0703 01:42:49.196239 31517 raft_consensus.cc:2243] T bbe64aa4b0e04f468b035048629b969b P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1288ca0989104bbea36e22f38c374218 in term 0. I0703 01:42:49.196643 31206 leader_election.cc:379] T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218 [CANDIDATE]: Term 1 pre-election: Vote granted by peer de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321) I0703 01:42:49.196703 31206 leader_election.cc:270] T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0703 01:42:49.196884 32169 raft_consensus.cc:2730] T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218: Snoozing failure detection for 2.017s (election complete) I0703 01:42:49.196943 32169 raft_consensus.cc:2537] T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0703 01:42:49.196981 32169 raft_consensus.cc:457] T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0703 01:42:49.197016 32169 raft_consensus.cc:2730] T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218: Snoozing failure detection for 2.038s (starting election) I0703 01:42:49.197067 32169 raft_consensus.cc:2782] T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:42:49.198336 32169 raft_consensus.cc:479] T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } I0703 01:42:49.198583 32169 leader_election.cc:243] T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218 [CANDIDATE]: Term 1 election: Requesting vote from peer de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321) I0703 01:42:49.199165 31517 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "bbe64aa4b0e04f468b035048629b969b" candidate_uuid: "1288ca0989104bbea36e22f38c374218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de93b58c2d67488dae12285e1806ebac" I0703 01:42:49.199261 31517 raft_consensus.cc:2782] T bbe64aa4b0e04f468b035048629b969b P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Advancing to term 1 I0703 01:42:49.199316 31517 raft_consensus.cc:2730] T bbe64aa4b0e04f468b035048629b969b P de93b58c2d67488dae12285e1806ebac: Snoozing failure detection for 1.959s (vote granted) I0703 01:42:49.200650 31517 raft_consensus.cc:2243] T bbe64aa4b0e04f468b035048629b969b P de93b58c2d67488dae12285e1806ebac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1288ca0989104bbea36e22f38c374218 in term 1. I0703 01:42:49.201071 31206 leader_election.cc:379] T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218 [CANDIDATE]: Term 1 election: Vote granted by peer de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321) I0703 01:42:49.201128 31206 leader_election.cc:270] T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:42:49.201301 32169 raft_consensus.cc:2730] T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218: Snoozing failure detection for 1.627s (election complete) I0703 01:42:49.201355 32169 raft_consensus.cc:2537] T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218 [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:42:49.201447 32169 raft_consensus.cc:593] T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218 [term 1 LEADER]: Becoming Leader. State: Replica: 1288ca0989104bbea36e22f38c374218, State: Running, Role: LEADER I0703 01:42:49.201617 32169 consensus_queue.cc:204] T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } I0703 01:42:49.203255 31147 catalog_manager.cc:3798] T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218 reported cstate change: term changed from 0 to 1, leader changed from to 1288ca0989104bbea36e22f38c374218 (127.5.240.193). New cstate: current_term: 1 leader_uuid: "1288ca0989104bbea36e22f38c374218" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } } I0703 01:42:49.324908 32169 raft_consensus.cc:457] T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0703 01:42:49.324959 32169 raft_consensus.cc:2730] T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218: Snoozing failure detection for 2.081s (starting election) I0703 01:42:49.325013 32169 raft_consensus.cc:479] T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } I0703 01:42:49.325225 32169 leader_election.cc:243] T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321) I0703 01:42:49.325790 31517 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "ffe66a5cfb564964951c619e51e7c47c" candidate_uuid: "1288ca0989104bbea36e22f38c374218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de93b58c2d67488dae12285e1806ebac" is_pre_election: true I0703 01:42:49.325899 31517 raft_consensus.cc:2730] T ffe66a5cfb564964951c619e51e7c47c P de93b58c2d67488dae12285e1806ebac: Snoozing failure detection for 1.638s (vote granted) I0703 01:42:49.325948 31517 raft_consensus.cc:2243] T ffe66a5cfb564964951c619e51e7c47c P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1288ca0989104bbea36e22f38c374218 in term 0. I0703 01:42:49.326339 31206 leader_election.cc:379] T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218 [CANDIDATE]: Term 1 pre-election: Vote granted by peer de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321) I0703 01:42:49.326395 31206 leader_election.cc:270] T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0703 01:42:49.326571 32169 raft_consensus.cc:2730] T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218: Snoozing failure detection for 1.973s (election complete) I0703 01:42:49.326624 32169 raft_consensus.cc:2537] T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0703 01:42:49.326666 32169 raft_consensus.cc:457] T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0703 01:42:49.326701 32169 raft_consensus.cc:2730] T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218: Snoozing failure detection for 2.201s (starting election) I0703 01:42:49.326740 32169 raft_consensus.cc:2782] T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:42:49.327821 32169 raft_consensus.cc:479] T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } I0703 01:42:49.328037 32169 leader_election.cc:243] T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218 [CANDIDATE]: Term 1 election: Requesting vote from peer de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321) I0703 01:42:49.328622 31517 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "ffe66a5cfb564964951c619e51e7c47c" candidate_uuid: "1288ca0989104bbea36e22f38c374218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de93b58c2d67488dae12285e1806ebac" I0703 01:42:49.328742 31517 raft_consensus.cc:2782] T ffe66a5cfb564964951c619e51e7c47c P de93b58c2d67488dae12285e1806ebac [term 0 FOLLOWER]: Advancing to term 1 I0703 01:42:49.328800 31517 raft_consensus.cc:2730] T ffe66a5cfb564964951c619e51e7c47c P de93b58c2d67488dae12285e1806ebac: Snoozing failure detection for 1.620s (vote granted) I0703 01:42:49.329941 31517 raft_consensus.cc:2243] T ffe66a5cfb564964951c619e51e7c47c P de93b58c2d67488dae12285e1806ebac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1288ca0989104bbea36e22f38c374218 in term 1. I0703 01:42:49.330323 31206 leader_election.cc:379] T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218 [CANDIDATE]: Term 1 election: Vote granted by peer de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321) I0703 01:42:49.330379 31206 leader_election.cc:270] T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:42:49.330554 32169 raft_consensus.cc:2730] T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218: Snoozing failure detection for 1.791s (election complete) I0703 01:42:49.330616 32169 raft_consensus.cc:2537] T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218 [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:42:49.330704 32169 raft_consensus.cc:593] T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218 [term 1 LEADER]: Becoming Leader. State: Replica: 1288ca0989104bbea36e22f38c374218, State: Running, Role: LEADER I0703 01:42:49.330838 32169 consensus_queue.cc:204] T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } I0703 01:42:49.332222 31147 catalog_manager.cc:3798] T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218 reported cstate change: term changed from 0 to 1, leader changed from to 1288ca0989104bbea36e22f38c374218 (127.5.240.193). New cstate: current_term: 1 leader_uuid: "1288ca0989104bbea36e22f38c374218" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } } W0703 01:42:49.547600 31328 consensus_peers.cc:433] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 56eaab17f812433a8f7880f0744e1744 (127.5.240.199:57714): Couldn't send request to peer 56eaab17f812433a8f7880f0744e1744 for tablet 454bab377db147a1a2f8241ecc2241f3. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.199:57714: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 16 times. I0703 01:42:49.569249 31395 raft_consensus.cc:1081] T 73518ef45438415da1300ef5004ea7d5 P 2f3fdb7c089640c0b936a574bc64707e [term 1 FOLLOWER]: Refusing update from remote peer 1288ca0989104bbea36e22f38c374218: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0703 01:42:49.570016 32169 consensus_queue.cc:962] T 73518ef45438415da1300ef5004ea7d5 P 1288ca0989104bbea36e22f38c374218 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s W0703 01:42:49.579303 32320 flags.cc:406] Enabled unsafe flag: --never_fsync=true Using table 'rebalance_test_table_4' I0703 01:42:49.716156 31517 raft_consensus.cc:1081] T ffe66a5cfb564964951c619e51e7c47c P de93b58c2d67488dae12285e1806ebac [term 1 FOLLOWER]: Refusing update from remote peer 1288ca0989104bbea36e22f38c374218: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0703 01:42:49.716900 32316 consensus_queue.cc:962] T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s I0703 01:42:49.721043 31516 raft_consensus.cc:1081] T bbe64aa4b0e04f468b035048629b969b P de93b58c2d67488dae12285e1806ebac [term 1 FOLLOWER]: Refusing update from remote peer 1288ca0989104bbea36e22f38c374218: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0703 01:42:49.722895 32316 consensus_queue.cc:962] T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s Generator report time total : 192.317 ms time per row: 0.0961586 ms I0703 01:42:49.863487 6083 test_util.cc:208] Using random seed: 1284653361 I0703 01:42:49.874130 6083 test_workload.cc:306] TestWorkload: Skipping table creation because table rebalance_test_table_0 already exists I0703 01:42:49.874555 6083 test_util.cc:208] Using random seed: 1284664308 I0703 01:42:49.889245 6083 test_workload.cc:306] TestWorkload: Skipping table creation because table rebalance_test_table_1 already exists I0703 01:42:49.889618 6083 test_util.cc:208] Using random seed: 1284679370 I0703 01:42:49.906146 6083 test_workload.cc:306] TestWorkload: Skipping table creation because table rebalance_test_table_2 already exists I0703 01:42:49.907357 6083 test_util.cc:208] Using random seed: 1284696998 W0703 01:42:49.936503 31328 consensus_peers.cc:433] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 56eaab17f812433a8f7880f0744e1744 (127.5.240.199:57714): Couldn't send request to peer 56eaab17f812433a8f7880f0744e1744 for tablet 454bab377db147a1a2f8241ecc2241f3. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.199:57714: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 17 times. I0703 01:42:49.944255 6083 test_workload.cc:306] TestWorkload: Skipping table creation because table rebalance_test_table_3 already exists I0703 01:42:49.945812 6083 test_util.cc:208] Using random seed: 1284735596 I0703 01:42:50.021128 6083 test_workload.cc:306] TestWorkload: Skipping table creation because table rebalance_test_table_4 already exists I0703 01:42:50.026024 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.5.240.196:56543 --local_ip_for_outbound_sockets=127.5.240.196 --webserver_port=39924 --webserver_interface=127.5.240.196 --tserver_master_addrs=127.5.240.254:36104 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:42:50.450696 32417 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:42:50.451962 32417 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:42:50.454587 32417 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0703 01:42:50.454674 32417 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:42:50.454813 32417 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.196 W0703 01:42:50.536151 31328 consensus_peers.cc:433] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 56eaab17f812433a8f7880f0744e1744 (127.5.240.199:57714): Couldn't send request to peer 56eaab17f812433a8f7880f0744e1744 for tablet 454bab377db147a1a2f8241ecc2241f3. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.199:57714: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 18 times. I0703 01:42:50.539559 32417 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.196:56543 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb --webserver_interface=127.5.240.196 --webserver_port=39924 --tserver_master_addrs=127.5.240.254:36104 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.196 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:42:50.553421 32417 tablet_server_main.cc:85] Initializing tablet server... I0703 01:42:50.557224 32417 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 238708us I0703 01:42:50.558369 32417 fs_manager.cc:261] Metadata directory not provided I0703 01:42:50.558498 32417 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:42:50.588088 32417 fs_manager.cc:397] Time spent opening directory manager: real 0.026s user 0.003s sys 0.009s I0703 01:42:50.589679 32417 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:42:50.590570 32417 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:42:50.596074 32417 fs_manager.cc:417] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s I0703 01:42:50.596196 32417 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal uuid: "d9dc68ace903455d809cb1f715fe36be" format_stamp: "Formatted at 2018-07-03 08:42:40 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:42:50.598665 32417 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:42:50.693208 32417 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:42:50.700574 32417 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:42:50.705755 32417 tablet_server_main.cc:90] Starting tablet server... W0703 01:42:50.953119 31328 consensus_peers.cc:433] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 56eaab17f812433a8f7880f0744e1744 (127.5.240.199:57714): Couldn't send request to peer 56eaab17f812433a8f7880f0744e1744 for tablet 454bab377db147a1a2f8241ecc2241f3. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.199:57714: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 19 times. I0703 01:42:51.369024 32417 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.196:56543 I0703 01:42:51.369685 32417 webserver.cc:175] Starting webserver on 127.5.240.196:39924 I0703 01:42:51.369757 32417 webserver.cc:186] Document root disabled I0703 01:42:51.377099 32417 webserver.cc:313] Webserver started. Bound to: http://127.5.240.196:39924/ I0703 01:42:51.378023 32417 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb I0703 01:42:51.387038 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 32417 I0703 01:42:51.387578 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data --block_manager=log --rpc_bind_addresses=127.5.240.197:44326 --local_ip_for_outbound_sockets=127.5.240.197 --webserver_port=41260 --webserver_interface=127.5.240.197 --tserver_master_addrs=127.5.240.254:36104 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0703 01:42:51.394575 32417 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:42:51.428012 32549 heartbeater.cc:346] Connected to a master server at 127.5.240.254:36104 I0703 01:42:51.428220 32549 heartbeater.cc:426] Registering TS with master... I0703 01:42:51.429361 32549 heartbeater.cc:475] Master 127.5.240.254:36104 requested a full tablet report, sending... I0703 01:42:51.431121 31150 ts_manager.cc:104] Re-registered known tserver with Master: d9dc68ace903455d809cb1f715fe36be (127.5.240.196:56543) I0703 01:42:51.432375 31150 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.196:46282 W0703 01:42:51.518313 31328 consensus_peers.cc:433] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 56eaab17f812433a8f7880f0744e1744 (127.5.240.199:57714): Couldn't send request to peer 56eaab17f812433a8f7880f0744e1744 for tablet 454bab377db147a1a2f8241ecc2241f3. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.199:57714: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 20 times. WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:42:51.760089 32553 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:42:51.761649 32553 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:42:51.765326 32553 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0703 01:42:51.765439 32553 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:42:51.765650 32553 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.197 I0703 01:42:51.784847 32553 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.197:44326 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb --webserver_interface=127.5.240.197 --webserver_port=41260 --tserver_master_addrs=127.5.240.254:36104 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.197 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:42:51.796947 32553 tablet_server_main.cc:85] Initializing tablet server... I0703 01:42:51.799165 32553 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 239208us I0703 01:42:51.800140 32553 fs_manager.cc:261] Metadata directory not provided I0703 01:42:51.800256 32553 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:42:51.823966 32553 fs_manager.cc:397] Time spent opening directory manager: real 0.022s user 0.002s sys 0.007s I0703 01:42:51.826963 32553 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:42:51.828419 32553 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:42:51.841862 32553 fs_manager.cc:417] Time spent opening block manager: real 0.010s user 0.002s sys 0.004s I0703 01:42:51.841990 32553 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal uuid: "3c1770123dc14bba80e00db5f182b585" format_stamp: "Formatted at 2018-07-03 08:42:40 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:42:51.846658 32553 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:42:52.000259 32553 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:42:52.002321 32553 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:42:52.003250 32553 tablet_server_main.cc:90] Starting tablet server... W0703 01:42:52.043362 31328 consensus_peers.cc:433] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 56eaab17f812433a8f7880f0744e1744 (127.5.240.199:57714): Couldn't send request to peer 56eaab17f812433a8f7880f0744e1744 for tablet 454bab377db147a1a2f8241ecc2241f3. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.199:57714: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 21 times. I0703 01:42:52.435482 32549 heartbeater.cc:467] Master 127.5.240.254:36104 was elected leader, sending a full tablet report... W0703 01:42:52.645220 31328 consensus_peers.cc:433] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 56eaab17f812433a8f7880f0744e1744 (127.5.240.199:57714): Couldn't send request to peer 56eaab17f812433a8f7880f0744e1744 for tablet 454bab377db147a1a2f8241ecc2241f3. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.199:57714: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 22 times. I0703 01:42:52.700701 32553 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.197:44326 I0703 01:42:52.701347 32553 webserver.cc:175] Starting webserver on 127.5.240.197:41260 I0703 01:42:52.701419 32553 webserver.cc:186] Document root disabled I0703 01:42:52.707473 32553 webserver.cc:313] Webserver started. Bound to: http://127.5.240.197:41260/ I0703 01:42:52.708372 32553 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb I0703 01:42:52.718724 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 32553 I0703 01:42:52.719331 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data --block_manager=log --rpc_bind_addresses=127.5.240.198:42654 --local_ip_for_outbound_sockets=127.5.240.198 --webserver_port=35397 --webserver_interface=127.5.240.198 --tserver_master_addrs=127.5.240.254:36104 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0703 01:42:52.737695 32553 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:42:52.780901 32682 heartbeater.cc:346] Connected to a master server at 127.5.240.254:36104 I0703 01:42:52.781100 32682 heartbeater.cc:426] Registering TS with master... I0703 01:42:52.782377 32682 heartbeater.cc:475] Master 127.5.240.254:36104 requested a full tablet report, sending... I0703 01:42:52.784176 31150 ts_manager.cc:104] Re-registered known tserver with Master: 3c1770123dc14bba80e00db5f182b585 (127.5.240.197:44326) I0703 01:42:52.785503 31150 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.197:59728 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:42:53.112251 32686 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:42:53.113507 32686 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:42:53.117676 32686 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0703 01:42:53.117789 32686 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:42:53.118140 32686 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.198 I0703 01:42:53.143812 32686 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.198:42654 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data/info.pb --webserver_interface=127.5.240.198 --webserver_port=35397 --tserver_master_addrs=127.5.240.254:36104 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.198 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:42:53.155169 32686 tablet_server_main.cc:85] Initializing tablet server... I0703 01:42:53.158884 32686 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 240208us I0703 01:42:53.159754 32686 fs_manager.cc:261] Metadata directory not provided I0703 01:42:53.159873 32686 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:42:53.177265 32686 fs_manager.cc:397] Time spent opening directory manager: real 0.015s user 0.003s sys 0.007s I0703 01:42:53.179882 32686 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:42:53.180495 32686 file_cache.cc:470] Constructed file cache lbm with capacity 13107 W0703 01:42:53.190035 31328 consensus_peers.cc:433] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 56eaab17f812433a8f7880f0744e1744 (127.5.240.199:57714): Couldn't send request to peer 56eaab17f812433a8f7880f0744e1744 for tablet 454bab377db147a1a2f8241ecc2241f3. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.199:57714: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 23 times. I0703 01:42:53.209594 32686 fs_manager.cc:417] Time spent opening block manager: real 0.026s user 0.001s sys 0.011s I0703 01:42:53.210939 32686 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/wal uuid: "768d8de4b2ba40f180b18194c28be589" format_stamp: "Formatted at 2018-07-03 08:42:40 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:42:53.218046 32686 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:42:53.493152 32686 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:42:53.501310 32686 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:42:53.506331 32686 tablet_server_main.cc:90] Starting tablet server... W0703 01:42:53.558461 31328 consensus_peers.cc:433] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 56eaab17f812433a8f7880f0744e1744 (127.5.240.199:57714): Couldn't send request to peer 56eaab17f812433a8f7880f0744e1744 for tablet 454bab377db147a1a2f8241ecc2241f3. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.199:57714: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 24 times. I0703 01:42:53.790683 32682 heartbeater.cc:467] Master 127.5.240.254:36104 was elected leader, sending a full tablet report... W0703 01:42:53.989356 31328 consensus_peers.cc:433] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 56eaab17f812433a8f7880f0744e1744 (127.5.240.199:57714): Couldn't send request to peer 56eaab17f812433a8f7880f0744e1744 for tablet 454bab377db147a1a2f8241ecc2241f3. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.199:57714: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 25 times. I0703 01:42:54.094957 32686 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.198:42654 I0703 01:42:54.095592 32686 webserver.cc:175] Starting webserver on 127.5.240.198:35397 I0703 01:42:54.095688 32686 webserver.cc:186] Document root disabled I0703 01:42:54.102798 32686 webserver.cc:313] Webserver started. Bound to: http://127.5.240.198:35397/ I0703 01:42:54.103724 32686 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data/info.pb I0703 01:42:54.109859 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 32686 I0703 01:42:54.110371 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data --block_manager=log --rpc_bind_addresses=127.5.240.199:57714 --local_ip_for_outbound_sockets=127.5.240.199 --webserver_port=58133 --webserver_interface=127.5.240.199 --tserver_master_addrs=127.5.240.254:36104 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=false with env {} I0703 01:42:54.110754 32686 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:42:54.163506 344 heartbeater.cc:346] Connected to a master server at 127.5.240.254:36104 I0703 01:42:54.163820 344 heartbeater.cc:426] Registering TS with master... I0703 01:42:54.165381 344 heartbeater.cc:475] Master 127.5.240.254:36104 requested a full tablet report, sending... I0703 01:42:54.167392 31150 ts_manager.cc:104] Re-registered known tserver with Master: 768d8de4b2ba40f180b18194c28be589 (127.5.240.198:42654) I0703 01:42:54.168728 31150 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.198:42191 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:42:54.482158 348 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:42:54.483359 348 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:42:54.487177 348 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false W0703 01:42:54.487293 348 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:42:54.487478 348 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.199 I0703 01:42:54.504186 348 tablet_server_main.cc:78] Tablet server non-default flags: --raft_prepare_replacement_before_eviction=false --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.199:57714 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data/info.pb --webserver_interface=127.5.240.199 --webserver_port=58133 --tserver_master_addrs=127.5.240.254:36104 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.199 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:42:54.513497 348 tablet_server_main.cc:85] Initializing tablet server... I0703 01:42:54.516716 348 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 240708us I0703 01:42:54.518420 348 fs_manager.cc:261] Metadata directory not provided I0703 01:42:54.518541 348 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:42:54.537624 348 fs_manager.cc:397] Time spent opening directory manager: real 0.016s user 0.002s sys 0.010s I0703 01:42:54.539505 348 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:42:54.540117 348 file_cache.cc:470] Constructed file cache lbm with capacity 13107 W0703 01:42:54.550577 31328 consensus_peers.cc:433] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 56eaab17f812433a8f7880f0744e1744 (127.5.240.199:57714): Couldn't send request to peer 56eaab17f812433a8f7880f0744e1744 for tablet 454bab377db147a1a2f8241ecc2241f3. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.199:57714: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 26 times. I0703 01:42:54.585528 348 fs_manager.cc:417] Time spent opening block manager: real 0.043s user 0.004s sys 0.026s I0703 01:42:54.585690 348 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/wal uuid: "56eaab17f812433a8f7880f0744e1744" format_stamp: "Formatted at 2018-07-03 08:42:41 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:42:54.588289 348 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:42:54.660639 348 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:42:54.664134 348 ts_tablet_manager.cc:344] Loading tablet metadata (0/1 complete) I0703 01:42:54.664187 348 ts_tablet_manager.cc:1005] T 454bab377db147a1a2f8241ecc2241f3 P 56eaab17f812433a8f7880f0744e1744: Loading tablet metadata I0703 01:42:54.673653 348 ts_tablet_manager.cc:356] Loaded tablet metadata (1 live tablets) I0703 01:42:54.678478 348 ts_tablet_manager.cc:1173] T 454bab377db147a1a2f8241ecc2241f3 P 56eaab17f812433a8f7880f0744e1744: Registered tablet (data state: TABLET_DATA_READY) I0703 01:42:54.688810 366 ts_tablet_manager.cc:1028] T 454bab377db147a1a2f8241ecc2241f3 P 56eaab17f812433a8f7880f0744e1744: Bootstrapping tablet I0703 01:42:54.697734 366 tablet_bootstrap.cc:427] T 454bab377db147a1a2f8241ecc2241f3 P 56eaab17f812433a8f7880f0744e1744: Bootstrap starting. I0703 01:42:54.710166 348 tablet_server_main.cc:90] Starting tablet server... I0703 01:42:54.737031 366 log.cc:525] T 454bab377db147a1a2f8241ecc2241f3 P 56eaab17f812433a8f7880f0744e1744: Log is configured to *not* fsync() on all Append() calls I0703 01:42:54.747126 366 tablet_bootstrap.cc:427] T 454bab377db147a1a2f8241ecc2241f3 P 56eaab17f812433a8f7880f0744e1744: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:42:54.747761 366 tablet_bootstrap.cc:427] T 454bab377db147a1a2f8241ecc2241f3 P 56eaab17f812433a8f7880f0744e1744: Bootstrap complete. I0703 01:42:54.749425 366 ts_tablet_manager.cc:1045] T 454bab377db147a1a2f8241ecc2241f3 P 56eaab17f812433a8f7880f0744e1744: Time spent bootstrapping tablet: real 0.060s user 0.012s sys 0.011s I0703 01:42:54.775763 366 raft_consensus.cc:325] T 454bab377db147a1a2f8241ecc2241f3 P 56eaab17f812433a8f7880f0744e1744 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } I0703 01:42:54.780175 366 raft_consensus.cc:628] T 454bab377db147a1a2f8241ecc2241f3 P 56eaab17f812433a8f7880f0744e1744 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56eaab17f812433a8f7880f0744e1744, State: Initialized, Role: FOLLOWER I0703 01:42:54.780920 366 consensus_queue.cc:227] T 454bab377db147a1a2f8241ecc2241f3 P 56eaab17f812433a8f7880f0744e1744 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } I0703 01:42:54.786243 366 ts_tablet_manager.cc:1073] T 454bab377db147a1a2f8241ecc2241f3 P 56eaab17f812433a8f7880f0744e1744: Time spent starting tablet: real 0.037s user 0.012s sys 0.014s I0703 01:42:55.175607 344 heartbeater.cc:467] Master 127.5.240.254:36104 was elected leader, sending a full tablet report... W0703 01:42:55.190948 31328 consensus_peers.cc:433] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 56eaab17f812433a8f7880f0744e1744 (127.5.240.199:57714): Couldn't send request to peer 56eaab17f812433a8f7880f0744e1744 for tablet 454bab377db147a1a2f8241ecc2241f3. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.199:57714: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 27 times. I0703 01:42:55.506235 348 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.199:57714 I0703 01:42:55.506860 348 webserver.cc:175] Starting webserver on 127.5.240.199:58133 I0703 01:42:55.506922 348 webserver.cc:186] Document root disabled I0703 01:42:55.513478 348 webserver.cc:313] Webserver started. Bound to: http://127.5.240.199:58133/ I0703 01:42:55.514418 348 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data/info.pb I0703 01:42:55.521091 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 348 W0703 01:42:55.543879 479 tablet.cc:1678] T 454bab377db147a1a2f8241ecc2241f3 P 56eaab17f812433a8f7880f0744e1744: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:42:55.544103 348 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:42:55.593698 478 heartbeater.cc:346] Connected to a master server at 127.5.240.254:36104 I0703 01:42:55.593832 478 heartbeater.cc:426] Registering TS with master... I0703 01:42:55.594269 478 heartbeater.cc:475] Master 127.5.240.254:36104 requested a full tablet report, sending... I0703 01:42:55.597786 31150 ts_manager.cc:104] Re-registered known tserver with Master: 56eaab17f812433a8f7880f0744e1744 (127.5.240.199:57714) I0703 01:42:55.599278 31150 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.199:38158 I0703 01:42:55.720866 478 heartbeater.cc:467] Master 127.5.240.254:36104 was elected leader, sending a full tablet report... I0703 01:42:56.969072 31394 consensus_peers.cc:455] T 479104ce31404ee4ad2488d9f46509e4 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521): Closing peer: 1288ca0989104bbea36e22f38c374218 I0703 01:42:56.969377 31394 consensus_queue.cc:204] T 479104ce31404ee4ad2488d9f46509e4 P 2f3fdb7c089640c0b936a574bc64707e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 136, Committed index: 136, Last appended: 1.137, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 138 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "3c1770123dc14bba80e00db5f182b585" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 44326 } } I0703 01:42:56.978204 31271 raft_consensus.cc:1081] T 479104ce31404ee4ad2488d9f46509e4 P 1288ca0989104bbea36e22f38c374218 [term 1 FOLLOWER]: Refusing update from remote peer 2f3fdb7c089640c0b936a574bc64707e: Log matching property violated. Preceding OpId in replica: term: 1 index: 137. Preceding OpId from leader: term: 1 index: 138. (index mismatch) I0703 01:42:56.981822 349 consensus_queue.cc:962] T 479104ce31404ee4ad2488d9f46509e4 P 2f3fdb7c089640c0b936a574bc64707e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 138, Last known committed idx: 136, Time since last communication: 0.000s W0703 01:42:56.990476 31326 consensus_peers.cc:433] T 479104ce31404ee4ad2488d9f46509e4 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 3c1770123dc14bba80e00db5f182b585 (127.5.240.197:44326): Couldn't send request to peer 3c1770123dc14bba80e00db5f182b585 for tablet 479104ce31404ee4ad2488d9f46509e4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 479104ce31404ee4ad2488d9f46509e4. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:42:56.991693 349 raft_consensus.cc:2684] T 479104ce31404ee4ad2488d9f46509e4 P 2f3fdb7c089640c0b936a574bc64707e [term 1 LEADER]: Committing config change with OpId 1.138: config changed from index -1 to 138, VOTER 3c1770123dc14bba80e00db5f182b585 (127.5.240.197) added. New config: { opid_index: 138 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "3c1770123dc14bba80e00db5f182b585" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 44326 } } } I0703 01:42:57.001211 31149 catalog_manager.cc:3798] T 479104ce31404ee4ad2488d9f46509e4 P 2f3fdb7c089640c0b936a574bc64707e reported cstate change: config changed from index -1 to 138, VOTER 3c1770123dc14bba80e00db5f182b585 (127.5.240.197) added. New cstate: current_term: 1 leader_uuid: "2f3fdb7c089640c0b936a574bc64707e" committed_config { opid_index: 138 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "3c1770123dc14bba80e00db5f182b585" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 44326 } } } I0703 01:42:57.001376 31271 raft_consensus.cc:2684] T 479104ce31404ee4ad2488d9f46509e4 P 1288ca0989104bbea36e22f38c374218 [term 1 FOLLOWER]: Committing config change with OpId 1.138: config changed from index -1 to 138, VOTER 3c1770123dc14bba80e00db5f182b585 (127.5.240.197) added. New config: { opid_index: 138 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "3c1770123dc14bba80e00db5f182b585" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 44326 } } } I0703 01:42:57.153761 31271 consensus_peers.cc:455] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218 -> Peer de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321): Closing peer: de93b58c2d67488dae12285e1806ebac I0703 01:42:57.154742 31271 consensus_queue.cc:204] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 1.130, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 131 OBSOLETE_local: false peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } I0703 01:42:57.172710 31514 raft_consensus.cc:1081] T a200f8a763f14d54a1e84082dbafce8f P de93b58c2d67488dae12285e1806ebac [term 1 FOLLOWER]: Refusing update from remote peer 1288ca0989104bbea36e22f38c374218: Log matching property violated. Preceding OpId in replica: term: 1 index: 129. Preceding OpId from leader: term: 1 index: 131. (index mismatch) I0703 01:42:57.185014 32346 consensus_queue.cc:962] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 131, Last known committed idx: 129, Time since last communication: 0.000s W0703 01:42:57.191859 31203 consensus_peers.cc:433] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218 -> Peer 768d8de4b2ba40f180b18194c28be589 (127.5.240.198:42654): Couldn't send request to peer 768d8de4b2ba40f180b18194c28be589 for tablet a200f8a763f14d54a1e84082dbafce8f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a200f8a763f14d54a1e84082dbafce8f. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:42:57.201694 32346 raft_consensus.cc:2684] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218 [term 1 LEADER]: Committing config change with OpId 1.131: config changed from index -1 to 131, VOTER 768d8de4b2ba40f180b18194c28be589 (127.5.240.198) added. New config: { opid_index: 131 OBSOLETE_local: false peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } } I0703 01:42:57.208187 31150 catalog_manager.cc:3798] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218 reported cstate change: config changed from index -1 to 131, VOTER 768d8de4b2ba40f180b18194c28be589 (127.5.240.198) added. New cstate: current_term: 1 leader_uuid: "1288ca0989104bbea36e22f38c374218" committed_config { opid_index: 131 OBSOLETE_local: false peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } } I0703 01:42:57.210500 31514 raft_consensus.cc:2684] T a200f8a763f14d54a1e84082dbafce8f P de93b58c2d67488dae12285e1806ebac [term 1 FOLLOWER]: Committing config change with OpId 1.131: config changed from index -1 to 131, VOTER 768d8de4b2ba40f180b18194c28be589 (127.5.240.198) added. New config: { opid_index: 131 OBSOLETE_local: false peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } } I0703 01:42:57.362026 31514 consensus_peers.cc:455] T 5eaa9c06d73c4e23b51fae2688eaf26c P de93b58c2d67488dae12285e1806ebac -> Peer 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194:39827): Closing peer: 2f3fdb7c089640c0b936a574bc64707e I0703 01:42:57.362278 31514 consensus_queue.cc:204] T 5eaa9c06d73c4e23b51fae2688eaf26c P de93b58c2d67488dae12285e1806ebac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 215, Committed index: 215, Last appended: 1.215, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } I0703 01:42:57.365149 31394 raft_consensus.cc:1081] T 5eaa9c06d73c4e23b51fae2688eaf26c P 2f3fdb7c089640c0b936a574bc64707e [term 1 FOLLOWER]: Refusing update from remote peer de93b58c2d67488dae12285e1806ebac: Log matching property violated. Preceding OpId in replica: term: 1 index: 215. Preceding OpId from leader: term: 1 index: 216. (index mismatch) I0703 01:42:57.366390 32387 consensus_queue.cc:962] T 5eaa9c06d73c4e23b51fae2688eaf26c P de93b58c2d67488dae12285e1806ebac [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 216, Last known committed idx: 215, Time since last communication: 0.000s I0703 01:42:57.383385 32688 raft_consensus.cc:2684] T 5eaa9c06d73c4e23b51fae2688eaf26c P de93b58c2d67488dae12285e1806ebac [term 1 LEADER]: Committing config change with OpId 1.216: config changed from index -1 to 216, VOTER d9dc68ace903455d809cb1f715fe36be (127.5.240.196) added. New config: { opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } } I0703 01:42:57.384658 31395 raft_consensus.cc:2684] T 5eaa9c06d73c4e23b51fae2688eaf26c P 2f3fdb7c089640c0b936a574bc64707e [term 1 FOLLOWER]: Committing config change with OpId 1.216: config changed from index -1 to 216, VOTER d9dc68ace903455d809cb1f715fe36be (127.5.240.196) added. New config: { opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } } I0703 01:42:57.388790 31150 catalog_manager.cc:3798] T 5eaa9c06d73c4e23b51fae2688eaf26c P 2f3fdb7c089640c0b936a574bc64707e reported cstate change: config changed from index -1 to 216, VOTER d9dc68ace903455d809cb1f715fe36be (127.5.240.196) added. New cstate: current_term: 1 leader_uuid: "de93b58c2d67488dae12285e1806ebac" committed_config { opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } } I0703 01:42:57.407948 618 ts_tablet_manager.cc:669] T 479104ce31404ee4ad2488d9f46509e4 P 3c1770123dc14bba80e00db5f182b585: Initiating tablet copy from peer 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194:39827) W0703 01:42:57.415046 31450 consensus_peers.cc:433] T 5eaa9c06d73c4e23b51fae2688eaf26c P de93b58c2d67488dae12285e1806ebac -> Peer d9dc68ace903455d809cb1f715fe36be (127.5.240.196:56543): Couldn't send request to peer d9dc68ace903455d809cb1f715fe36be for tablet 5eaa9c06d73c4e23b51fae2688eaf26c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5eaa9c06d73c4e23b51fae2688eaf26c. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:42:57.415799 618 tablet_copy_client.cc:239] T 479104ce31404ee4ad2488d9f46509e4 P 3c1770123dc14bba80e00db5f182b585: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.194:39827 I0703 01:42:57.439537 31415 tablet_copy_service.cc:135] P 2f3fdb7c089640c0b936a574bc64707e: Received BeginTabletCopySession request for tablet 479104ce31404ee4ad2488d9f46509e4 from peer 3c1770123dc14bba80e00db5f182b585 ({username='jenkins'} at 127.5.240.197:38739) I0703 01:42:57.439751 31415 tablet_copy_service.cc:156] P 2f3fdb7c089640c0b936a574bc64707e: Beginning new tablet copy session on tablet 479104ce31404ee4ad2488d9f46509e4 from peer 3c1770123dc14bba80e00db5f182b585 at {username='jenkins'} at 127.5.240.197:38739: session id = 3c1770123dc14bba80e00db5f182b585-479104ce31404ee4ad2488d9f46509e4 I0703 01:42:57.443616 31415 tablet_copy_source_session.cc:202] T 479104ce31404ee4ad2488d9f46509e4 P 2f3fdb7c089640c0b936a574bc64707e: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:42:57.448413 618 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 479104ce31404ee4ad2488d9f46509e4. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:57.460100 618 ts_tablet_manager.cc:1173] T 479104ce31404ee4ad2488d9f46509e4 P 3c1770123dc14bba80e00db5f182b585: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:42:57.460382 618 tablet_copy_client.cc:566] T 479104ce31404ee4ad2488d9f46509e4 P 3c1770123dc14bba80e00db5f182b585: tablet copy: Starting download of 0 data blocks... I0703 01:42:57.460606 618 tablet_copy_client.cc:529] T 479104ce31404ee4ad2488d9f46509e4 P 3c1770123dc14bba80e00db5f182b585: tablet copy: Starting download of 1 WAL segments... I0703 01:42:57.476269 618 tablet_copy_client.cc:405] T 479104ce31404ee4ad2488d9f46509e4 P 3c1770123dc14bba80e00db5f182b585: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:42:57.483186 618 ts_tablet_manager.cc:1028] T 479104ce31404ee4ad2488d9f46509e4 P 3c1770123dc14bba80e00db5f182b585: Bootstrapping tablet I0703 01:42:57.483845 618 tablet_bootstrap.cc:427] T 479104ce31404ee4ad2488d9f46509e4 P 3c1770123dc14bba80e00db5f182b585: Bootstrap starting. I0703 01:42:57.542856 618 log.cc:525] T 479104ce31404ee4ad2488d9f46509e4 P 3c1770123dc14bba80e00db5f182b585: Log is configured to *not* fsync() on all Append() calls I0703 01:42:57.576350 31271 consensus_peers.cc:455] T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218 -> Peer de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321): Closing peer: de93b58c2d67488dae12285e1806ebac I0703 01:42:57.576591 31271 consensus_queue.cc:204] T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 155, Committed index: 155, Last appended: 1.155, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } I0703 01:42:57.580754 31515 raft_consensus.cc:1081] T bbe64aa4b0e04f468b035048629b969b P de93b58c2d67488dae12285e1806ebac [term 1 FOLLOWER]: Refusing update from remote peer 1288ca0989104bbea36e22f38c374218: Log matching property violated. Preceding OpId in replica: term: 1 index: 155. Preceding OpId from leader: term: 1 index: 157. (index mismatch) I0703 01:42:57.582645 509 consensus_queue.cc:962] T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 156, Last known committed idx: 155, Time since last communication: 0.000s I0703 01:42:57.587757 635 ts_tablet_manager.cc:669] T a200f8a763f14d54a1e84082dbafce8f P 768d8de4b2ba40f180b18194c28be589: Initiating tablet copy from peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521) I0703 01:42:57.588073 635 tablet_copy_client.cc:239] T a200f8a763f14d54a1e84082dbafce8f P 768d8de4b2ba40f180b18194c28be589: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.193:33521 I0703 01:42:57.593396 32346 raft_consensus.cc:2684] T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218 [term 1 LEADER]: Committing config change with OpId 1.156: config changed from index -1 to 156, VOTER 56eaab17f812433a8f7880f0744e1744 (127.5.240.199) added. New config: { opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } } I0703 01:42:57.598611 31150 catalog_manager.cc:3798] T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218 reported cstate change: config changed from index -1 to 156, VOTER 56eaab17f812433a8f7880f0744e1744 (127.5.240.199) added. New cstate: current_term: 1 leader_uuid: "1288ca0989104bbea36e22f38c374218" committed_config { opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } } I0703 01:42:57.606464 31515 raft_consensus.cc:2684] T bbe64aa4b0e04f468b035048629b969b P de93b58c2d67488dae12285e1806ebac [term 1 FOLLOWER]: Committing config change with OpId 1.156: config changed from index -1 to 156, VOTER 56eaab17f812433a8f7880f0744e1744 (127.5.240.199) added. New config: { opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } } W0703 01:42:57.610713 31205 consensus_peers.cc:433] T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218 -> Peer 56eaab17f812433a8f7880f0744e1744 (127.5.240.199:57714): Couldn't send request to peer 56eaab17f812433a8f7880f0744e1744 for tablet bbe64aa4b0e04f468b035048629b969b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bbe64aa4b0e04f468b035048629b969b. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:42:57.641046 31292 tablet_copy_service.cc:135] P 1288ca0989104bbea36e22f38c374218: Received BeginTabletCopySession request for tablet a200f8a763f14d54a1e84082dbafce8f from peer 768d8de4b2ba40f180b18194c28be589 ({username='jenkins'} at 127.5.240.198:42829) I0703 01:42:57.641234 31292 tablet_copy_service.cc:156] P 1288ca0989104bbea36e22f38c374218: Beginning new tablet copy session on tablet a200f8a763f14d54a1e84082dbafce8f from peer 768d8de4b2ba40f180b18194c28be589 at {username='jenkins'} at 127.5.240.198:42829: session id = 768d8de4b2ba40f180b18194c28be589-a200f8a763f14d54a1e84082dbafce8f I0703 01:42:57.644079 31292 tablet_copy_source_session.cc:202] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:42:57.648865 635 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet a200f8a763f14d54a1e84082dbafce8f. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:57.678927 635 ts_tablet_manager.cc:1173] T a200f8a763f14d54a1e84082dbafce8f P 768d8de4b2ba40f180b18194c28be589: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:42:57.680625 635 tablet_copy_client.cc:566] T a200f8a763f14d54a1e84082dbafce8f P 768d8de4b2ba40f180b18194c28be589: tablet copy: Starting download of 0 data blocks... I0703 01:42:57.681831 635 tablet_copy_client.cc:529] T a200f8a763f14d54a1e84082dbafce8f P 768d8de4b2ba40f180b18194c28be589: tablet copy: Starting download of 1 WAL segments... I0703 01:42:57.687328 635 tablet_copy_client.cc:405] T a200f8a763f14d54a1e84082dbafce8f P 768d8de4b2ba40f180b18194c28be589: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:42:57.690459 635 ts_tablet_manager.cc:1028] T a200f8a763f14d54a1e84082dbafce8f P 768d8de4b2ba40f180b18194c28be589: Bootstrapping tablet I0703 01:42:57.699066 635 tablet_bootstrap.cc:427] T a200f8a763f14d54a1e84082dbafce8f P 768d8de4b2ba40f180b18194c28be589: Bootstrap starting. I0703 01:42:57.756675 635 log.cc:525] T a200f8a763f14d54a1e84082dbafce8f P 768d8de4b2ba40f180b18194c28be589: Log is configured to *not* fsync() on all Append() calls I0703 01:42:57.771410 31515 consensus_peers.cc:455] T e0af316d40d143a8b387933151adf0e4 P de93b58c2d67488dae12285e1806ebac -> Peer 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194:39827): Closing peer: 2f3fdb7c089640c0b936a574bc64707e I0703 01:42:57.771714 31515 consensus_queue.cc:204] T e0af316d40d143a8b387933151adf0e4 P de93b58c2d67488dae12285e1806ebac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 242, Committed index: 242, Last appended: 1.242, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 243 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "3c1770123dc14bba80e00db5f182b585" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 44326 } } I0703 01:42:57.779280 31395 raft_consensus.cc:1081] T e0af316d40d143a8b387933151adf0e4 P 2f3fdb7c089640c0b936a574bc64707e [term 1 FOLLOWER]: Refusing update from remote peer de93b58c2d67488dae12285e1806ebac: Log matching property violated. Preceding OpId in replica: term: 1 index: 242. Preceding OpId from leader: term: 1 index: 243. (index mismatch) I0703 01:42:57.781723 597 consensus_queue.cc:962] T e0af316d40d143a8b387933151adf0e4 P de93b58c2d67488dae12285e1806ebac [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 243, Last known committed idx: 242, Time since last communication: 0.000s I0703 01:42:57.789222 32527 raft_consensus.cc:2684] T e0af316d40d143a8b387933151adf0e4 P de93b58c2d67488dae12285e1806ebac [term 1 LEADER]: Committing config change with OpId 1.243: config changed from index -1 to 243, VOTER 3c1770123dc14bba80e00db5f182b585 (127.5.240.197) added. New config: { opid_index: 243 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "3c1770123dc14bba80e00db5f182b585" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 44326 } } } I0703 01:42:57.795030 31395 raft_consensus.cc:2684] T e0af316d40d143a8b387933151adf0e4 P 2f3fdb7c089640c0b936a574bc64707e [term 1 FOLLOWER]: Committing config change with OpId 1.243: config changed from index -1 to 243, VOTER 3c1770123dc14bba80e00db5f182b585 (127.5.240.197) added. New config: { opid_index: 243 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "3c1770123dc14bba80e00db5f182b585" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 44326 } } } W0703 01:42:57.800016 31448 consensus_peers.cc:433] T e0af316d40d143a8b387933151adf0e4 P de93b58c2d67488dae12285e1806ebac -> Peer 3c1770123dc14bba80e00db5f182b585 (127.5.240.197:44326): Couldn't send request to peer 3c1770123dc14bba80e00db5f182b585 for tablet e0af316d40d143a8b387933151adf0e4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e0af316d40d143a8b387933151adf0e4. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:42:57.799998 31150 catalog_manager.cc:3798] T e0af316d40d143a8b387933151adf0e4 P de93b58c2d67488dae12285e1806ebac reported cstate change: config changed from index -1 to 243, VOTER 3c1770123dc14bba80e00db5f182b585 (127.5.240.197) added. New cstate: current_term: 1 leader_uuid: "de93b58c2d67488dae12285e1806ebac" committed_config { opid_index: 243 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "3c1770123dc14bba80e00db5f182b585" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 44326 } } } W0703 01:42:57.870720 31326 consensus_peers.cc:433] T 479104ce31404ee4ad2488d9f46509e4 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 3c1770123dc14bba80e00db5f182b585 (127.5.240.197:44326): Couldn't send request to peer 3c1770123dc14bba80e00db5f182b585 for tablet 479104ce31404ee4ad2488d9f46509e4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 2 times. I0703 01:42:57.946301 666 ts_tablet_manager.cc:669] T 5eaa9c06d73c4e23b51fae2688eaf26c P d9dc68ace903455d809cb1f715fe36be: Initiating tablet copy from peer de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321) I0703 01:42:57.947270 666 tablet_copy_client.cc:239] T 5eaa9c06d73c4e23b51fae2688eaf26c P d9dc68ace903455d809cb1f715fe36be: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.195:59321 I0703 01:42:57.972873 31537 tablet_copy_service.cc:135] P de93b58c2d67488dae12285e1806ebac: Received BeginTabletCopySession request for tablet 5eaa9c06d73c4e23b51fae2688eaf26c from peer d9dc68ace903455d809cb1f715fe36be ({username='jenkins'} at 127.5.240.196:37222) I0703 01:42:57.973083 31537 tablet_copy_service.cc:156] P de93b58c2d67488dae12285e1806ebac: Beginning new tablet copy session on tablet 5eaa9c06d73c4e23b51fae2688eaf26c from peer d9dc68ace903455d809cb1f715fe36be at {username='jenkins'} at 127.5.240.196:37222: session id = d9dc68ace903455d809cb1f715fe36be-5eaa9c06d73c4e23b51fae2688eaf26c I0703 01:42:57.976289 31537 tablet_copy_source_session.cc:202] T 5eaa9c06d73c4e23b51fae2688eaf26c P de93b58c2d67488dae12285e1806ebac: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:42:57.980865 666 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 5eaa9c06d73c4e23b51fae2688eaf26c. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:57.995129 31395 consensus_peers.cc:455] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e -> Peer de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321): Closing peer: de93b58c2d67488dae12285e1806ebac I0703 01:42:57.995395 31395 consensus_queue.cc:204] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 209, Committed index: 209, Last appended: 1.209, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 210 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } I0703 01:42:58.001282 666 ts_tablet_manager.cc:1173] T 5eaa9c06d73c4e23b51fae2688eaf26c P d9dc68ace903455d809cb1f715fe36be: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:42:58.002228 666 tablet_copy_client.cc:566] T 5eaa9c06d73c4e23b51fae2688eaf26c P d9dc68ace903455d809cb1f715fe36be: tablet copy: Starting download of 0 data blocks... I0703 01:42:58.002424 666 tablet_copy_client.cc:529] T 5eaa9c06d73c4e23b51fae2688eaf26c P d9dc68ace903455d809cb1f715fe36be: tablet copy: Starting download of 1 WAL segments... I0703 01:42:58.009047 31515 raft_consensus.cc:1081] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P de93b58c2d67488dae12285e1806ebac [term 1 FOLLOWER]: Refusing update from remote peer 2f3fdb7c089640c0b936a574bc64707e: Log matching property violated. Preceding OpId in replica: term: 1 index: 209. Preceding OpId from leader: term: 1 index: 210. (index mismatch) W0703 01:42:58.010730 31326 consensus_peers.cc:433] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 768d8de4b2ba40f180b18194c28be589 (127.5.240.198:42654): Couldn't send request to peer 768d8de4b2ba40f180b18194c28be589 for tablet 1f01bfdde47b4d37a0f77d199bc7b4d2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1f01bfdde47b4d37a0f77d199bc7b4d2. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:42:58.012856 623 consensus_queue.cc:962] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e [LEADER]: Connected to new peer: Peer: permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 210, Last known committed idx: 209, Time since last communication: 0.000s I0703 01:42:58.017421 623 raft_consensus.cc:2684] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e [term 1 LEADER]: Committing config change with OpId 1.210: config changed from index -1 to 210, VOTER 768d8de4b2ba40f180b18194c28be589 (127.5.240.198) added. New config: { opid_index: 210 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } } I0703 01:42:58.025939 31149 catalog_manager.cc:3798] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e reported cstate change: config changed from index -1 to 210, VOTER 768d8de4b2ba40f180b18194c28be589 (127.5.240.198) added. New cstate: current_term: 1 leader_uuid: "2f3fdb7c089640c0b936a574bc64707e" committed_config { opid_index: 210 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } } I0703 01:42:58.030637 618 tablet_bootstrap.cc:427] T 479104ce31404ee4ad2488d9f46509e4 P 3c1770123dc14bba80e00db5f182b585: Bootstrap replayed 1/1 log segments. Stats: ops{read=143 overwritten=0 applied=142 ignored=0} inserts{seen=791 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates W0703 01:42:58.031404 31203 consensus_peers.cc:433] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218 -> Peer 768d8de4b2ba40f180b18194c28be589 (127.5.240.198:42654): Couldn't send request to peer 768d8de4b2ba40f180b18194c28be589 for tablet a200f8a763f14d54a1e84082dbafce8f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 2 times. I0703 01:42:58.032716 31514 raft_consensus.cc:2684] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P de93b58c2d67488dae12285e1806ebac [term 1 FOLLOWER]: Committing config change with OpId 1.210: config changed from index -1 to 210, VOTER 768d8de4b2ba40f180b18194c28be589 (127.5.240.198) added. New config: { opid_index: 210 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } } I0703 01:42:58.035650 618 tablet_bootstrap.cc:427] T 479104ce31404ee4ad2488d9f46509e4 P 3c1770123dc14bba80e00db5f182b585: Bootstrap complete. I0703 01:42:58.035996 618 ts_tablet_manager.cc:1045] T 479104ce31404ee4ad2488d9f46509e4 P 3c1770123dc14bba80e00db5f182b585: Time spent bootstrapping tablet: real 0.552s user 0.221s sys 0.013s I0703 01:42:58.043145 666 tablet_copy_client.cc:405] T 5eaa9c06d73c4e23b51fae2688eaf26c P d9dc68ace903455d809cb1f715fe36be: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:42:58.046062 666 ts_tablet_manager.cc:1028] T 5eaa9c06d73c4e23b51fae2688eaf26c P d9dc68ace903455d809cb1f715fe36be: Bootstrapping tablet I0703 01:42:58.046365 666 tablet_bootstrap.cc:427] T 5eaa9c06d73c4e23b51fae2688eaf26c P d9dc68ace903455d809cb1f715fe36be: Bootstrap starting. I0703 01:42:58.048080 618 raft_consensus.cc:325] T 479104ce31404ee4ad2488d9f46509e4 P 3c1770123dc14bba80e00db5f182b585 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending transactions. Active config: opid_index: 138 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "3c1770123dc14bba80e00db5f182b585" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 44326 } } I0703 01:42:58.059212 618 raft_consensus.cc:628] T 479104ce31404ee4ad2488d9f46509e4 P 3c1770123dc14bba80e00db5f182b585 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c1770123dc14bba80e00db5f182b585, State: Initialized, Role: FOLLOWER I0703 01:42:58.059715 618 consensus_queue.cc:227] T 479104ce31404ee4ad2488d9f46509e4 P 3c1770123dc14bba80e00db5f182b585 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 142, Last appended: 1.143, Last appended by leader: 143, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 138 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "3c1770123dc14bba80e00db5f182b585" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 44326 } } I0703 01:42:58.066994 618 ts_tablet_manager.cc:1073] T 479104ce31404ee4ad2488d9f46509e4 P 3c1770123dc14bba80e00db5f182b585: Time spent starting tablet: real 0.031s user 0.010s sys 0.004s I0703 01:42:58.068641 31415 tablet_copy_service.cc:325] P 2f3fdb7c089640c0b936a574bc64707e: Request end of tablet copy session 3c1770123dc14bba80e00db5f182b585-479104ce31404ee4ad2488d9f46509e4 received from {username='jenkins'} at 127.5.240.197:38739 I0703 01:42:58.068742 31415 tablet_copy_service.cc:413] P 2f3fdb7c089640c0b936a574bc64707e: ending tablet copy session 3c1770123dc14bba80e00db5f182b585-479104ce31404ee4ad2488d9f46509e4 on tablet 479104ce31404ee4ad2488d9f46509e4 with peer 3c1770123dc14bba80e00db5f182b585 I0703 01:42:58.117272 635 tablet_bootstrap.cc:427] T a200f8a763f14d54a1e84082dbafce8f P 768d8de4b2ba40f180b18194c28be589: Bootstrap replayed 1/1 log segments. Stats: ops{read=136 overwritten=0 applied=135 ignored=0} inserts{seen=787 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates I0703 01:42:58.119390 635 tablet_bootstrap.cc:427] T a200f8a763f14d54a1e84082dbafce8f P 768d8de4b2ba40f180b18194c28be589: Bootstrap complete. I0703 01:42:58.119832 635 ts_tablet_manager.cc:1045] T a200f8a763f14d54a1e84082dbafce8f P 768d8de4b2ba40f180b18194c28be589: Time spent bootstrapping tablet: real 0.421s user 0.208s sys 0.006s I0703 01:42:58.129279 635 raft_consensus.cc:325] T a200f8a763f14d54a1e84082dbafce8f P 768d8de4b2ba40f180b18194c28be589 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending transactions. Active config: opid_index: 131 OBSOLETE_local: false peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } I0703 01:42:58.143385 635 raft_consensus.cc:628] T a200f8a763f14d54a1e84082dbafce8f P 768d8de4b2ba40f180b18194c28be589 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768d8de4b2ba40f180b18194c28be589, State: Initialized, Role: FOLLOWER I0703 01:42:58.144706 635 consensus_queue.cc:227] T a200f8a763f14d54a1e84082dbafce8f P 768d8de4b2ba40f180b18194c28be589 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 135, Last appended: 1.136, Last appended by leader: 136, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 131 OBSOLETE_local: false peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } I0703 01:42:58.167863 635 ts_tablet_manager.cc:1073] T a200f8a763f14d54a1e84082dbafce8f P 768d8de4b2ba40f180b18194c28be589: Time spent starting tablet: real 0.048s user 0.009s sys 0.009s I0703 01:42:58.170720 31292 tablet_copy_service.cc:325] P 1288ca0989104bbea36e22f38c374218: Request end of tablet copy session 768d8de4b2ba40f180b18194c28be589-a200f8a763f14d54a1e84082dbafce8f received from {username='jenkins'} at 127.5.240.198:42829 I0703 01:42:58.170805 31292 tablet_copy_service.cc:413] P 1288ca0989104bbea36e22f38c374218: ending tablet copy session 768d8de4b2ba40f180b18194c28be589-a200f8a763f14d54a1e84082dbafce8f on tablet a200f8a763f14d54a1e84082dbafce8f with peer 768d8de4b2ba40f180b18194c28be589 I0703 01:42:58.209940 696 ts_tablet_manager.cc:669] T bbe64aa4b0e04f468b035048629b969b P 56eaab17f812433a8f7880f0744e1744: Initiating tablet copy from peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521) I0703 01:42:58.210290 696 tablet_copy_client.cc:239] T bbe64aa4b0e04f468b035048629b969b P 56eaab17f812433a8f7880f0744e1744: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.193:33521 I0703 01:42:58.217800 31271 consensus_peers.cc:455] T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218 -> Peer de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321): Closing peer: de93b58c2d67488dae12285e1806ebac I0703 01:42:58.217988 31271 consensus_queue.cc:204] T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 188, Committed index: 188, Last appended: 1.188, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 189 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } I0703 01:42:58.227282 31292 tablet_copy_service.cc:135] P 1288ca0989104bbea36e22f38c374218: Received BeginTabletCopySession request for tablet bbe64aa4b0e04f468b035048629b969b from peer 56eaab17f812433a8f7880f0744e1744 ({username='jenkins'} at 127.5.240.199:50303) I0703 01:42:58.227569 31292 tablet_copy_service.cc:156] P 1288ca0989104bbea36e22f38c374218: Beginning new tablet copy session on tablet bbe64aa4b0e04f468b035048629b969b from peer 56eaab17f812433a8f7880f0744e1744 at {username='jenkins'} at 127.5.240.199:50303: session id = 56eaab17f812433a8f7880f0744e1744-bbe64aa4b0e04f468b035048629b969b I0703 01:42:58.227850 31514 raft_consensus.cc:1081] T ffe66a5cfb564964951c619e51e7c47c P de93b58c2d67488dae12285e1806ebac [term 1 FOLLOWER]: Refusing update from remote peer 1288ca0989104bbea36e22f38c374218: Log matching property violated. Preceding OpId in replica: term: 1 index: 188. Preceding OpId from leader: term: 1 index: 189. (index mismatch) I0703 01:42:58.230898 31292 tablet_copy_source_session.cc:202] T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:42:58.232515 696 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet bbe64aa4b0e04f468b035048629b969b. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:58.234747 32346 consensus_queue.cc:962] T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 189, Last known committed idx: 188, Time since last communication: 0.000s I0703 01:42:58.238241 666 log.cc:525] T 5eaa9c06d73c4e23b51fae2688eaf26c P d9dc68ace903455d809cb1f715fe36be: Log is configured to *not* fsync() on all Append() calls I0703 01:42:58.239547 696 ts_tablet_manager.cc:1173] T bbe64aa4b0e04f468b035048629b969b P 56eaab17f812433a8f7880f0744e1744: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:42:58.242925 696 tablet_copy_client.cc:566] T bbe64aa4b0e04f468b035048629b969b P 56eaab17f812433a8f7880f0744e1744: tablet copy: Starting download of 0 data blocks... I0703 01:42:58.243083 696 tablet_copy_client.cc:529] T bbe64aa4b0e04f468b035048629b969b P 56eaab17f812433a8f7880f0744e1744: tablet copy: Starting download of 1 WAL segments... I0703 01:42:58.244674 32758 raft_consensus.cc:2684] T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218 [term 1 LEADER]: Committing config change with OpId 1.189: config changed from index -1 to 189, VOTER d9dc68ace903455d809cb1f715fe36be (127.5.240.196) added. New config: { opid_index: 189 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } } I0703 01:42:58.249630 696 tablet_copy_client.cc:405] T bbe64aa4b0e04f468b035048629b969b P 56eaab17f812433a8f7880f0744e1744: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:42:58.250469 31149 catalog_manager.cc:3798] T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218 reported cstate change: config changed from index -1 to 189, VOTER d9dc68ace903455d809cb1f715fe36be (127.5.240.196) added. New cstate: current_term: 1 leader_uuid: "1288ca0989104bbea36e22f38c374218" committed_config { opid_index: 189 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } } W0703 01:42:58.251822 31205 consensus_peers.cc:433] T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218 -> Peer d9dc68ace903455d809cb1f715fe36be (127.5.240.196:56543): Couldn't send request to peer d9dc68ace903455d809cb1f715fe36be for tablet ffe66a5cfb564964951c619e51e7c47c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ffe66a5cfb564964951c619e51e7c47c. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:42:58.255435 31515 raft_consensus.cc:2684] T ffe66a5cfb564964951c619e51e7c47c P de93b58c2d67488dae12285e1806ebac [term 1 FOLLOWER]: Committing config change with OpId 1.189: config changed from index -1 to 189, VOTER d9dc68ace903455d809cb1f715fe36be (127.5.240.196) added. New config: { opid_index: 189 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } } I0703 01:42:58.260695 696 ts_tablet_manager.cc:1028] T bbe64aa4b0e04f468b035048629b969b P 56eaab17f812433a8f7880f0744e1744: Bootstrapping tablet I0703 01:42:58.266105 696 tablet_bootstrap.cc:427] T bbe64aa4b0e04f468b035048629b969b P 56eaab17f812433a8f7880f0744e1744: Bootstrap starting. I0703 01:42:58.267561 32638 raft_consensus.cc:1026] T 479104ce31404ee4ad2488d9f46509e4 P 3c1770123dc14bba80e00db5f182b585 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.137->[1.138-1.153] Dedup: 1.143->[1.144-1.153] I0703 01:42:58.282279 618 ts_tablet_manager.cc:669] T e0af316d40d143a8b387933151adf0e4 P 3c1770123dc14bba80e00db5f182b585: Initiating tablet copy from peer de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321) I0703 01:42:58.282449 618 tablet_copy_client.cc:239] T e0af316d40d143a8b387933151adf0e4 P 3c1770123dc14bba80e00db5f182b585: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.195:59321 I0703 01:42:58.335667 31537 tablet_copy_service.cc:135] P de93b58c2d67488dae12285e1806ebac: Received BeginTabletCopySession request for tablet e0af316d40d143a8b387933151adf0e4 from peer 3c1770123dc14bba80e00db5f182b585 ({username='jenkins'} at 127.5.240.197:55267) I0703 01:42:58.335850 31537 tablet_copy_service.cc:156] P de93b58c2d67488dae12285e1806ebac: Beginning new tablet copy session on tablet e0af316d40d143a8b387933151adf0e4 from peer 3c1770123dc14bba80e00db5f182b585 at {username='jenkins'} at 127.5.240.197:55267: session id = 3c1770123dc14bba80e00db5f182b585-e0af316d40d143a8b387933151adf0e4 I0703 01:42:58.338346 31537 tablet_copy_source_session.cc:202] T e0af316d40d143a8b387933151adf0e4 P de93b58c2d67488dae12285e1806ebac: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:42:58.340296 618 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet e0af316d40d143a8b387933151adf0e4. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:58.346071 618 ts_tablet_manager.cc:1173] T e0af316d40d143a8b387933151adf0e4 P 3c1770123dc14bba80e00db5f182b585: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:42:58.346457 618 tablet_copy_client.cc:566] T e0af316d40d143a8b387933151adf0e4 P 3c1770123dc14bba80e00db5f182b585: tablet copy: Starting download of 0 data blocks... I0703 01:42:58.346626 618 tablet_copy_client.cc:529] T e0af316d40d143a8b387933151adf0e4 P 3c1770123dc14bba80e00db5f182b585: tablet copy: Starting download of 1 WAL segments... I0703 01:42:58.354007 618 tablet_copy_client.cc:405] T e0af316d40d143a8b387933151adf0e4 P 3c1770123dc14bba80e00db5f182b585: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:42:58.356312 618 ts_tablet_manager.cc:1028] T e0af316d40d143a8b387933151adf0e4 P 3c1770123dc14bba80e00db5f182b585: Bootstrapping tablet I0703 01:42:58.356448 618 tablet_bootstrap.cc:427] T e0af316d40d143a8b387933151adf0e4 P 3c1770123dc14bba80e00db5f182b585: Bootstrap starting. W0703 01:42:58.490394 31450 consensus_peers.cc:433] T 5eaa9c06d73c4e23b51fae2688eaf26c P de93b58c2d67488dae12285e1806ebac -> Peer d9dc68ace903455d809cb1f715fe36be (127.5.240.196:56543): Couldn't send request to peer d9dc68ace903455d809cb1f715fe36be for tablet 5eaa9c06d73c4e23b51fae2688eaf26c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 2 times. I0703 01:42:58.508929 635 ts_tablet_manager.cc:669] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 768d8de4b2ba40f180b18194c28be589: Initiating tablet copy from peer 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194:39827) I0703 01:42:58.509109 635 tablet_copy_client.cc:239] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 768d8de4b2ba40f180b18194c28be589: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.194:39827 I0703 01:42:58.540241 31415 tablet_copy_service.cc:135] P 2f3fdb7c089640c0b936a574bc64707e: Received BeginTabletCopySession request for tablet 1f01bfdde47b4d37a0f77d199bc7b4d2 from peer 768d8de4b2ba40f180b18194c28be589 ({username='jenkins'} at 127.5.240.198:59612) I0703 01:42:58.541867 31415 tablet_copy_service.cc:156] P 2f3fdb7c089640c0b936a574bc64707e: Beginning new tablet copy session on tablet 1f01bfdde47b4d37a0f77d199bc7b4d2 from peer 768d8de4b2ba40f180b18194c28be589 at {username='jenkins'} at 127.5.240.198:59612: session id = 768d8de4b2ba40f180b18194c28be589-1f01bfdde47b4d37a0f77d199bc7b4d2 I0703 01:42:58.544325 31415 tablet_copy_source_session.cc:202] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:42:58.546252 635 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 1f01bfdde47b4d37a0f77d199bc7b4d2. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:58.551133 635 ts_tablet_manager.cc:1173] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 768d8de4b2ba40f180b18194c28be589: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:42:58.553966 635 tablet_copy_client.cc:566] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 768d8de4b2ba40f180b18194c28be589: tablet copy: Starting download of 0 data blocks... I0703 01:42:58.554742 635 tablet_copy_client.cc:529] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 768d8de4b2ba40f180b18194c28be589: tablet copy: Starting download of 1 WAL segments... I0703 01:42:58.562352 635 tablet_copy_client.cc:405] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 768d8de4b2ba40f180b18194c28be589: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:42:58.564987 635 ts_tablet_manager.cc:1028] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 768d8de4b2ba40f180b18194c28be589: Bootstrapping tablet I0703 01:42:58.572113 635 tablet_bootstrap.cc:427] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 768d8de4b2ba40f180b18194c28be589: Bootstrap starting. I0703 01:42:58.618512 300 raft_consensus.cc:1026] T a200f8a763f14d54a1e84082dbafce8f P 768d8de4b2ba40f180b18194c28be589 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.130->[1.131-1.147] Dedup: 1.136->[1.137-1.147] I0703 01:42:58.627159 31394 consensus_peers.cc:455] T 3d7a6e87780a49949cf7e024817cf3a3 P 2f3fdb7c089640c0b936a574bc64707e -> Peer de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321): Closing peer: de93b58c2d67488dae12285e1806ebac I0703 01:42:58.628216 31394 consensus_queue.cc:204] T 3d7a6e87780a49949cf7e024817cf3a3 P 2f3fdb7c089640c0b936a574bc64707e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 213, Committed index: 213, Last appended: 1.213, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 214 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } W0703 01:42:58.631111 31328 consensus_peers.cc:433] T 3d7a6e87780a49949cf7e024817cf3a3 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 56eaab17f812433a8f7880f0744e1744 (127.5.240.199:57714): Couldn't send request to peer 56eaab17f812433a8f7880f0744e1744 for tablet 3d7a6e87780a49949cf7e024817cf3a3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3d7a6e87780a49949cf7e024817cf3a3. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:42:58.638375 31514 raft_consensus.cc:1081] T 3d7a6e87780a49949cf7e024817cf3a3 P de93b58c2d67488dae12285e1806ebac [term 1 FOLLOWER]: Refusing update from remote peer 2f3fdb7c089640c0b936a574bc64707e: Log matching property violated. Preceding OpId in replica: term: 1 index: 213. Preceding OpId from leader: term: 1 index: 214. (index mismatch) I0703 01:42:58.641346 623 consensus_queue.cc:962] T 3d7a6e87780a49949cf7e024817cf3a3 P 2f3fdb7c089640c0b936a574bc64707e [LEADER]: Connected to new peer: Peer: permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 214, Last known committed idx: 213, Time since last communication: 0.000s I0703 01:42:58.653406 32580 raft_consensus.cc:2684] T 3d7a6e87780a49949cf7e024817cf3a3 P 2f3fdb7c089640c0b936a574bc64707e [term 1 LEADER]: Committing config change with OpId 1.214: config changed from index -1 to 214, VOTER 56eaab17f812433a8f7880f0744e1744 (127.5.240.199) added. New config: { opid_index: 214 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } } I0703 01:42:58.658597 31514 raft_consensus.cc:2684] T 3d7a6e87780a49949cf7e024817cf3a3 P de93b58c2d67488dae12285e1806ebac [term 1 FOLLOWER]: Committing config change with OpId 1.214: config changed from index -1 to 214, VOTER 56eaab17f812433a8f7880f0744e1744 (127.5.240.199) added. New config: { opid_index: 214 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } } I0703 01:42:58.659391 31149 catalog_manager.cc:3798] T 3d7a6e87780a49949cf7e024817cf3a3 P 2f3fdb7c089640c0b936a574bc64707e reported cstate change: config changed from index -1 to 214, VOTER 56eaab17f812433a8f7880f0744e1744 (127.5.240.199) added. New cstate: current_term: 1 leader_uuid: "2f3fdb7c089640c0b936a574bc64707e" committed_config { opid_index: 214 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } } I0703 01:42:58.741782 733 ts_tablet_manager.cc:669] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be: Initiating tablet copy from peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521) I0703 01:42:58.742000 733 tablet_copy_client.cc:239] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.193:33521 I0703 01:42:58.786375 31292 tablet_copy_service.cc:135] P 1288ca0989104bbea36e22f38c374218: Received BeginTabletCopySession request for tablet ffe66a5cfb564964951c619e51e7c47c from peer d9dc68ace903455d809cb1f715fe36be ({username='jenkins'} at 127.5.240.196:46328) I0703 01:42:58.786492 31292 tablet_copy_service.cc:156] P 1288ca0989104bbea36e22f38c374218: Beginning new tablet copy session on tablet ffe66a5cfb564964951c619e51e7c47c from peer d9dc68ace903455d809cb1f715fe36be at {username='jenkins'} at 127.5.240.196:46328: session id = d9dc68ace903455d809cb1f715fe36be-ffe66a5cfb564964951c619e51e7c47c I0703 01:42:58.789093 31292 tablet_copy_source_session.cc:202] T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:42:58.791935 733 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet ffe66a5cfb564964951c619e51e7c47c. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:58.796860 733 ts_tablet_manager.cc:1173] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:42:58.797684 733 tablet_copy_client.cc:566] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be: tablet copy: Starting download of 0 data blocks... I0703 01:42:58.804754 733 tablet_copy_client.cc:529] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be: tablet copy: Starting download of 1 WAL segments... W0703 01:42:58.826022 31205 consensus_peers.cc:433] T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218 -> Peer 56eaab17f812433a8f7880f0744e1744 (127.5.240.199:57714): Couldn't send request to peer 56eaab17f812433a8f7880f0744e1744 for tablet bbe64aa4b0e04f468b035048629b969b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 2 times. I0703 01:42:58.827992 733 tablet_copy_client.cc:405] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:42:58.839007 733 ts_tablet_manager.cc:1028] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be: Bootstrapping tablet I0703 01:42:58.839393 733 tablet_bootstrap.cc:427] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be: Bootstrap starting. W0703 01:42:58.881086 31448 consensus_peers.cc:433] T e0af316d40d143a8b387933151adf0e4 P de93b58c2d67488dae12285e1806ebac -> Peer 3c1770123dc14bba80e00db5f182b585 (127.5.240.197:44326): Couldn't send request to peer 3c1770123dc14bba80e00db5f182b585 for tablet e0af316d40d143a8b387933151adf0e4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 2 times. W0703 01:42:58.912113 31450 consensus_peers.cc:433] T 5eaa9c06d73c4e23b51fae2688eaf26c P de93b58c2d67488dae12285e1806ebac -> Peer d9dc68ace903455d809cb1f715fe36be (127.5.240.196:56543): Couldn't send request to peer d9dc68ace903455d809cb1f715fe36be for tablet 5eaa9c06d73c4e23b51fae2688eaf26c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 3 times. I0703 01:42:59.028180 755 ts_tablet_manager.cc:669] T 3d7a6e87780a49949cf7e024817cf3a3 P 56eaab17f812433a8f7880f0744e1744: Initiating tablet copy from peer 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194:39827) I0703 01:42:59.028448 755 tablet_copy_client.cc:239] T 3d7a6e87780a49949cf7e024817cf3a3 P 56eaab17f812433a8f7880f0744e1744: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.194:39827 I0703 01:42:59.064664 31395 consensus_peers.cc:455] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 56eaab17f812433a8f7880f0744e1744 (127.5.240.199:57714): Closing peer: 56eaab17f812433a8f7880f0744e1744 I0703 01:42:59.064963 31395 consensus_queue.cc:204] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } I0703 01:42:59.075963 31415 tablet_copy_service.cc:135] P 2f3fdb7c089640c0b936a574bc64707e: Received BeginTabletCopySession request for tablet 3d7a6e87780a49949cf7e024817cf3a3 from peer 56eaab17f812433a8f7880f0744e1744 ({username='jenkins'} at 127.5.240.199:50676) I0703 01:42:59.076086 31415 tablet_copy_service.cc:156] P 2f3fdb7c089640c0b936a574bc64707e: Beginning new tablet copy session on tablet 3d7a6e87780a49949cf7e024817cf3a3 from peer 56eaab17f812433a8f7880f0744e1744 at {username='jenkins'} at 127.5.240.199:50676: session id = 56eaab17f812433a8f7880f0744e1744-3d7a6e87780a49949cf7e024817cf3a3 I0703 01:42:59.078621 31415 tablet_copy_source_session.cc:202] T 3d7a6e87780a49949cf7e024817cf3a3 P 2f3fdb7c089640c0b936a574bc64707e: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:42:59.081712 755 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 3d7a6e87780a49949cf7e024817cf3a3. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:59.086757 755 ts_tablet_manager.cc:1173] T 3d7a6e87780a49949cf7e024817cf3a3 P 56eaab17f812433a8f7880f0744e1744: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:42:59.086987 755 tablet_copy_client.cc:566] T 3d7a6e87780a49949cf7e024817cf3a3 P 56eaab17f812433a8f7880f0744e1744: tablet copy: Starting download of 0 data blocks... W0703 01:42:59.091006 31205 consensus_peers.cc:433] T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218 -> Peer d9dc68ace903455d809cb1f715fe36be (127.5.240.196:56543): Couldn't send request to peer d9dc68ace903455d809cb1f715fe36be for tablet ffe66a5cfb564964951c619e51e7c47c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 2 times. I0703 01:42:59.099606 755 tablet_copy_client.cc:529] T 3d7a6e87780a49949cf7e024817cf3a3 P 56eaab17f812433a8f7880f0744e1744: tablet copy: Starting download of 1 WAL segments... I0703 01:42:59.101225 434 raft_consensus.cc:1081] T 454bab377db147a1a2f8241ecc2241f3 P 56eaab17f812433a8f7880f0744e1744 [term 1 FOLLOWER]: Refusing update from remote peer 2f3fdb7c089640c0b936a574bc64707e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0703 01:42:59.102602 623 consensus_queue.cc:962] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e [LEADER]: Connected to new peer: Peer: permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s I0703 01:42:59.107003 755 tablet_copy_client.cc:405] T 3d7a6e87780a49949cf7e024817cf3a3 P 56eaab17f812433a8f7880f0744e1744: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:42:59.110065 755 ts_tablet_manager.cc:1028] T 3d7a6e87780a49949cf7e024817cf3a3 P 56eaab17f812433a8f7880f0744e1744: Bootstrapping tablet I0703 01:42:59.110230 755 tablet_bootstrap.cc:427] T 3d7a6e87780a49949cf7e024817cf3a3 P 56eaab17f812433a8f7880f0744e1744: Bootstrap starting. I0703 01:42:59.131438 32580 raft_consensus.cc:2684] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d9dc68ace903455d809cb1f715fe36be (127.5.240.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } } W0703 01:42:59.142047 31326 consensus_peers.cc:433] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 768d8de4b2ba40f180b18194c28be589 (127.5.240.198:42654): Couldn't send request to peer 768d8de4b2ba40f180b18194c28be589 for tablet 1f01bfdde47b4d37a0f77d199bc7b4d2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 2 times. I0703 01:42:59.153455 434 raft_consensus.cc:2684] T 454bab377db147a1a2f8241ecc2241f3 P 56eaab17f812433a8f7880f0744e1744 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER d9dc68ace903455d809cb1f715fe36be (127.5.240.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } } I0703 01:42:59.160799 31149 catalog_manager.cc:3798] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e reported cstate change: config changed from index -1 to 2, VOTER d9dc68ace903455d809cb1f715fe36be (127.5.240.196) added. New cstate: current_term: 1 leader_uuid: "2f3fdb7c089640c0b936a574bc64707e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } } W0703 01:42:59.169040 31328 consensus_peers.cc:433] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e -> Peer d9dc68ace903455d809cb1f715fe36be (127.5.240.196:56543): Couldn't send request to peer d9dc68ace903455d809cb1f715fe36be for tablet 454bab377db147a1a2f8241ecc2241f3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 454bab377db147a1a2f8241ecc2241f3. Retrying in the next heartbeat period. Already tried 1 times. W0703 01:42:59.183933 31329 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.240.195:59321, user_credentials={real_user=jenkins}} blocked reactor thread for 56852.6us W0703 01:42:59.186563 31327 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.240.193:33521, user_credentials={real_user=jenkins}} blocked reactor thread for 43171.9us W0703 01:42:59.317863 31205 consensus_peers.cc:433] T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218 -> Peer 56eaab17f812433a8f7880f0744e1744 (127.5.240.199:57714): Couldn't send request to peer 56eaab17f812433a8f7880f0744e1744 for tablet bbe64aa4b0e04f468b035048629b969b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 3 times. W0703 01:42:59.351073 31450 consensus_peers.cc:433] T 5eaa9c06d73c4e23b51fae2688eaf26c P de93b58c2d67488dae12285e1806ebac -> Peer d9dc68ace903455d809cb1f715fe36be (127.5.240.196:56543): Couldn't send request to peer d9dc68ace903455d809cb1f715fe36be for tablet 5eaa9c06d73c4e23b51fae2688eaf26c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 4 times. W0703 01:42:59.402559 31448 consensus_peers.cc:433] T e0af316d40d143a8b387933151adf0e4 P de93b58c2d67488dae12285e1806ebac -> Peer 3c1770123dc14bba80e00db5f182b585 (127.5.240.197:44326): Couldn't send request to peer 3c1770123dc14bba80e00db5f182b585 for tablet e0af316d40d143a8b387933151adf0e4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 3 times. I0703 01:42:59.425202 31514 consensus_peers.cc:455] T b2c8b58f935a4d00b5cf3e2cedec4d81 P de93b58c2d67488dae12285e1806ebac -> Peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521): Closing peer: 1288ca0989104bbea36e22f38c374218 I0703 01:42:59.425454 31514 consensus_queue.cc:204] T b2c8b58f935a4d00b5cf3e2cedec4d81 P de93b58c2d67488dae12285e1806ebac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 216, Committed index: 216, Last appended: 1.216, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } I0703 01:42:59.428617 31271 raft_consensus.cc:1081] T b2c8b58f935a4d00b5cf3e2cedec4d81 P 1288ca0989104bbea36e22f38c374218 [term 1 FOLLOWER]: Refusing update from remote peer de93b58c2d67488dae12285e1806ebac: Log matching property violated. Preceding OpId in replica: term: 1 index: 216. Preceding OpId from leader: term: 1 index: 217. (index mismatch) I0703 01:42:59.429961 32688 consensus_queue.cc:962] T b2c8b58f935a4d00b5cf3e2cedec4d81 P de93b58c2d67488dae12285e1806ebac [LEADER]: Connected to new peer: Peer: permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 217, Last known committed idx: 216, Time since last communication: 0.000s I0703 01:42:59.432541 653 raft_consensus.cc:2684] T b2c8b58f935a4d00b5cf3e2cedec4d81 P de93b58c2d67488dae12285e1806ebac [term 1 LEADER]: Committing config change with OpId 1.217: config changed from index -1 to 217, VOTER 768d8de4b2ba40f180b18194c28be589 (127.5.240.198) added. New config: { opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } } I0703 01:42:59.436599 31271 raft_consensus.cc:2684] T b2c8b58f935a4d00b5cf3e2cedec4d81 P 1288ca0989104bbea36e22f38c374218 [term 1 FOLLOWER]: Committing config change with OpId 1.217: config changed from index -1 to 217, VOTER 768d8de4b2ba40f180b18194c28be589 (127.5.240.198) added. New config: { opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } } I0703 01:42:59.440920 31150 catalog_manager.cc:3798] T b2c8b58f935a4d00b5cf3e2cedec4d81 P 1288ca0989104bbea36e22f38c374218 reported cstate change: config changed from index -1 to 217, VOTER 768d8de4b2ba40f180b18194c28be589 (127.5.240.198) added. New cstate: current_term: 1 leader_uuid: "de93b58c2d67488dae12285e1806ebac" committed_config { opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } } W0703 01:42:59.475296 31328 consensus_peers.cc:433] T 3d7a6e87780a49949cf7e024817cf3a3 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 56eaab17f812433a8f7880f0744e1744 (127.5.240.199:57714): Couldn't send request to peer 56eaab17f812433a8f7880f0744e1744 for tablet 3d7a6e87780a49949cf7e024817cf3a3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 2 times. W0703 01:42:59.475498 31448 consensus_peers.cc:433] T b2c8b58f935a4d00b5cf3e2cedec4d81 P de93b58c2d67488dae12285e1806ebac -> Peer 768d8de4b2ba40f180b18194c28be589 (127.5.240.198:42654): Couldn't send request to peer 768d8de4b2ba40f180b18194c28be589 for tablet b2c8b58f935a4d00b5cf3e2cedec4d81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b2c8b58f935a4d00b5cf3e2cedec4d81. Retrying in the next heartbeat period. Already tried 1 times. W0703 01:42:59.603752 31205 consensus_peers.cc:433] T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218 -> Peer d9dc68ace903455d809cb1f715fe36be (127.5.240.196:56543): Couldn't send request to peer d9dc68ace903455d809cb1f715fe36be for tablet ffe66a5cfb564964951c619e51e7c47c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 3 times. I0703 01:42:59.660980 666 tablet_bootstrap.cc:427] T 5eaa9c06d73c4e23b51fae2688eaf26c P d9dc68ace903455d809cb1f715fe36be: Bootstrap replayed 1/1 log segments. Stats: ops{read=228 overwritten=0 applied=228 ignored=0} inserts{seen=880 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:42:59.661517 666 tablet_bootstrap.cc:427] T 5eaa9c06d73c4e23b51fae2688eaf26c P d9dc68ace903455d809cb1f715fe36be: Bootstrap complete. I0703 01:42:59.666918 666 ts_tablet_manager.cc:1045] T 5eaa9c06d73c4e23b51fae2688eaf26c P d9dc68ace903455d809cb1f715fe36be: Time spent bootstrapping tablet: real 1.621s user 0.299s sys 0.027s I0703 01:42:59.702143 666 raft_consensus.cc:325] T 5eaa9c06d73c4e23b51fae2688eaf26c P d9dc68ace903455d809cb1f715fe36be [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } I0703 01:42:59.702765 666 raft_consensus.cc:628] T 5eaa9c06d73c4e23b51fae2688eaf26c P d9dc68ace903455d809cb1f715fe36be [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9dc68ace903455d809cb1f715fe36be, State: Initialized, Role: FOLLOWER I0703 01:42:59.703495 666 consensus_queue.cc:227] T 5eaa9c06d73c4e23b51fae2688eaf26c P d9dc68ace903455d809cb1f715fe36be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 228, Last appended: 1.228, Last appended by leader: 228, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 216 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } I0703 01:42:59.707365 802 ts_tablet_manager.cc:669] T 454bab377db147a1a2f8241ecc2241f3 P d9dc68ace903455d809cb1f715fe36be: Initiating tablet copy from peer 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194:39827) I0703 01:42:59.707598 802 tablet_copy_client.cc:239] T 454bab377db147a1a2f8241ecc2241f3 P d9dc68ace903455d809cb1f715fe36be: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.194:39827 W0703 01:42:59.716260 31326 consensus_peers.cc:433] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 768d8de4b2ba40f180b18194c28be589 (127.5.240.198:42654): Couldn't send request to peer 768d8de4b2ba40f180b18194c28be589 for tablet 1f01bfdde47b4d37a0f77d199bc7b4d2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 3 times. W0703 01:42:59.731614 31205 consensus_peers.cc:433] T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218 -> Peer 56eaab17f812433a8f7880f0744e1744 (127.5.240.199:57714): Couldn't send request to peer 56eaab17f812433a8f7880f0744e1744 for tablet bbe64aa4b0e04f468b035048629b969b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 4 times. I0703 01:42:59.774134 31415 tablet_copy_service.cc:135] P 2f3fdb7c089640c0b936a574bc64707e: Received BeginTabletCopySession request for tablet 454bab377db147a1a2f8241ecc2241f3 from peer d9dc68ace903455d809cb1f715fe36be ({username='jenkins'} at 127.5.240.196:56593) I0703 01:42:59.774271 31415 tablet_copy_service.cc:156] P 2f3fdb7c089640c0b936a574bc64707e: Beginning new tablet copy session on tablet 454bab377db147a1a2f8241ecc2241f3 from peer d9dc68ace903455d809cb1f715fe36be at {username='jenkins'} at 127.5.240.196:56593: session id = d9dc68ace903455d809cb1f715fe36be-454bab377db147a1a2f8241ecc2241f3 I0703 01:42:59.778239 31415 tablet_copy_source_session.cc:202] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:42:59.781009 666 ts_tablet_manager.cc:1073] T 5eaa9c06d73c4e23b51fae2688eaf26c P d9dc68ace903455d809cb1f715fe36be: Time spent starting tablet: real 0.113s user 0.010s sys 0.010s I0703 01:42:59.782519 31537 tablet_copy_service.cc:325] P de93b58c2d67488dae12285e1806ebac: Request end of tablet copy session d9dc68ace903455d809cb1f715fe36be-5eaa9c06d73c4e23b51fae2688eaf26c received from {username='jenkins'} at 127.5.240.196:37222 I0703 01:42:59.782603 31537 tablet_copy_service.cc:413] P de93b58c2d67488dae12285e1806ebac: ending tablet copy session d9dc68ace903455d809cb1f715fe36be-5eaa9c06d73c4e23b51fae2688eaf26c on tablet 5eaa9c06d73c4e23b51fae2688eaf26c with peer d9dc68ace903455d809cb1f715fe36be I0703 01:42:59.785496 802 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 454bab377db147a1a2f8241ecc2241f3. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:42:59.793159 802 ts_tablet_manager.cc:1173] T 454bab377db147a1a2f8241ecc2241f3 P d9dc68ace903455d809cb1f715fe36be: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:42:59.793305 802 tablet_copy_client.cc:566] T 454bab377db147a1a2f8241ecc2241f3 P d9dc68ace903455d809cb1f715fe36be: tablet copy: Starting download of 0 data blocks... I0703 01:42:59.796036 802 tablet_copy_client.cc:529] T 454bab377db147a1a2f8241ecc2241f3 P d9dc68ace903455d809cb1f715fe36be: tablet copy: Starting download of 1 WAL segments... W0703 01:42:59.801964 31448 consensus_peers.cc:433] T e0af316d40d143a8b387933151adf0e4 P de93b58c2d67488dae12285e1806ebac -> Peer 3c1770123dc14bba80e00db5f182b585 (127.5.240.197:44326): Couldn't send request to peer 3c1770123dc14bba80e00db5f182b585 for tablet e0af316d40d143a8b387933151adf0e4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 4 times. I0703 01:42:59.803583 31395 consensus_peers.cc:455] T 44f06af6e8ef46d78419aea845bb38a7 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521): Closing peer: 1288ca0989104bbea36e22f38c374218 I0703 01:42:59.803812 802 tablet_copy_client.cc:405] T 454bab377db147a1a2f8241ecc2241f3 P d9dc68ace903455d809cb1f715fe36be: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:42:59.804095 31395 consensus_queue.cc:204] T 44f06af6e8ef46d78419aea845bb38a7 P 2f3fdb7c089640c0b936a574bc64707e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 201, Committed index: 201, Last appended: 1.202, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 203 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } W0703 01:42:59.806687 31328 consensus_peers.cc:433] T 44f06af6e8ef46d78419aea845bb38a7 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 56eaab17f812433a8f7880f0744e1744 (127.5.240.199:57714): Couldn't send request to peer 56eaab17f812433a8f7880f0744e1744 for tablet 44f06af6e8ef46d78419aea845bb38a7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 44f06af6e8ef46d78419aea845bb38a7. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:42:59.806903 802 ts_tablet_manager.cc:1028] T 454bab377db147a1a2f8241ecc2241f3 P d9dc68ace903455d809cb1f715fe36be: Bootstrapping tablet I0703 01:42:59.807703 802 tablet_bootstrap.cc:427] T 454bab377db147a1a2f8241ecc2241f3 P d9dc68ace903455d809cb1f715fe36be: Bootstrap starting. I0703 01:42:59.819931 31270 raft_consensus.cc:1081] T 44f06af6e8ef46d78419aea845bb38a7 P 1288ca0989104bbea36e22f38c374218 [term 1 FOLLOWER]: Refusing update from remote peer 2f3fdb7c089640c0b936a574bc64707e: Log matching property violated. Preceding OpId in replica: term: 1 index: 202. Preceding OpId from leader: term: 1 index: 203. (index mismatch) I0703 01:42:59.830186 623 consensus_queue.cc:962] T 44f06af6e8ef46d78419aea845bb38a7 P 2f3fdb7c089640c0b936a574bc64707e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 203, Last known committed idx: 201, Time since last communication: 0.000s I0703 01:42:59.840420 802 tablet_bootstrap.cc:427] T 454bab377db147a1a2f8241ecc2241f3 P d9dc68ace903455d809cb1f715fe36be: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:42:59.840773 802 tablet_bootstrap.cc:427] T 454bab377db147a1a2f8241ecc2241f3 P d9dc68ace903455d809cb1f715fe36be: Bootstrap complete. I0703 01:42:59.840934 802 ts_tablet_manager.cc:1045] T 454bab377db147a1a2f8241ecc2241f3 P d9dc68ace903455d809cb1f715fe36be: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.003s I0703 01:42:59.841737 802 raft_consensus.cc:325] T 454bab377db147a1a2f8241ecc2241f3 P d9dc68ace903455d809cb1f715fe36be [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } I0703 01:42:59.841923 802 raft_consensus.cc:628] T 454bab377db147a1a2f8241ecc2241f3 P d9dc68ace903455d809cb1f715fe36be [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9dc68ace903455d809cb1f715fe36be, State: Initialized, Role: FOLLOWER I0703 01:42:59.842082 802 consensus_queue.cc:227] T 454bab377db147a1a2f8241ecc2241f3 P d9dc68ace903455d809cb1f715fe36be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } I0703 01:42:59.842660 802 ts_tablet_manager.cc:1073] T 454bab377db147a1a2f8241ecc2241f3 P d9dc68ace903455d809cb1f715fe36be: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s I0703 01:42:59.847632 31415 tablet_copy_service.cc:325] P 2f3fdb7c089640c0b936a574bc64707e: Request end of tablet copy session d9dc68ace903455d809cb1f715fe36be-454bab377db147a1a2f8241ecc2241f3 received from {username='jenkins'} at 127.5.240.196:56593 I0703 01:42:59.847710 31415 tablet_copy_service.cc:413] P 2f3fdb7c089640c0b936a574bc64707e: ending tablet copy session d9dc68ace903455d809cb1f715fe36be-454bab377db147a1a2f8241ecc2241f3 on tablet 454bab377db147a1a2f8241ecc2241f3 with peer d9dc68ace903455d809cb1f715fe36be I0703 01:42:59.862645 763 raft_consensus.cc:2684] T 44f06af6e8ef46d78419aea845bb38a7 P 2f3fdb7c089640c0b936a574bc64707e [term 1 LEADER]: Committing config change with OpId 1.203: config changed from index -1 to 203, VOTER 56eaab17f812433a8f7880f0744e1744 (127.5.240.199) added. New config: { opid_index: 203 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } } I0703 01:42:59.873484 31150 catalog_manager.cc:3798] T 44f06af6e8ef46d78419aea845bb38a7 P 2f3fdb7c089640c0b936a574bc64707e reported cstate change: config changed from index -1 to 203, VOTER 56eaab17f812433a8f7880f0744e1744 (127.5.240.199) added. New cstate: current_term: 1 leader_uuid: "2f3fdb7c089640c0b936a574bc64707e" committed_config { opid_index: 203 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } } I0703 01:42:59.877739 32504 raft_consensus.cc:1026] T 5eaa9c06d73c4e23b51fae2688eaf26c P d9dc68ace903455d809cb1f715fe36be [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.215->[1.216-1.260] Dedup: 1.228->[1.229-1.260] I0703 01:42:59.882730 696 tablet_bootstrap.cc:427] T bbe64aa4b0e04f468b035048629b969b P 56eaab17f812433a8f7880f0744e1744: Bootstrap replayed 1/1 log segments. Stats: ops{read=164 overwritten=0 applied=164 ignored=0} inserts{seen=851 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:42:59.883157 696 tablet_bootstrap.cc:427] T bbe64aa4b0e04f468b035048629b969b P 56eaab17f812433a8f7880f0744e1744: Bootstrap complete. I0703 01:42:59.883561 696 ts_tablet_manager.cc:1045] T bbe64aa4b0e04f468b035048629b969b P 56eaab17f812433a8f7880f0744e1744: Time spent bootstrapping tablet: real 1.618s user 0.244s sys 0.021s I0703 01:42:59.885005 696 raft_consensus.cc:325] T bbe64aa4b0e04f468b035048629b969b P 56eaab17f812433a8f7880f0744e1744 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } I0703 01:42:59.885263 696 raft_consensus.cc:628] T bbe64aa4b0e04f468b035048629b969b P 56eaab17f812433a8f7880f0744e1744 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56eaab17f812433a8f7880f0744e1744, State: Initialized, Role: FOLLOWER I0703 01:42:59.885466 696 consensus_queue.cc:227] T bbe64aa4b0e04f468b035048629b969b P 56eaab17f812433a8f7880f0744e1744 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 164, Last appended: 1.164, Last appended by leader: 164, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } I0703 01:42:59.901068 31270 raft_consensus.cc:2684] T 44f06af6e8ef46d78419aea845bb38a7 P 1288ca0989104bbea36e22f38c374218 [term 1 FOLLOWER]: Committing config change with OpId 1.203: config changed from index -1 to 203, VOTER 56eaab17f812433a8f7880f0744e1744 (127.5.240.199) added. New config: { opid_index: 203 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } } I0703 01:42:59.904731 696 ts_tablet_manager.cc:1073] T bbe64aa4b0e04f468b035048629b969b P 56eaab17f812433a8f7880f0744e1744: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s I0703 01:42:59.905776 31292 tablet_copy_service.cc:325] P 1288ca0989104bbea36e22f38c374218: Request end of tablet copy session 56eaab17f812433a8f7880f0744e1744-bbe64aa4b0e04f468b035048629b969b received from {username='jenkins'} at 127.5.240.199:50303 I0703 01:42:59.905905 31292 tablet_copy_service.cc:413] P 1288ca0989104bbea36e22f38c374218: ending tablet copy session 56eaab17f812433a8f7880f0744e1744-bbe64aa4b0e04f468b035048629b969b on tablet bbe64aa4b0e04f468b035048629b969b with peer 56eaab17f812433a8f7880f0744e1744 W0703 01:42:59.908849 32550 tablet.cc:1678] T 454bab377db147a1a2f8241ecc2241f3 P d9dc68ace903455d809cb1f715fe36be: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:42:59.923357 618 tablet_bootstrap.cc:427] T e0af316d40d143a8b387933151adf0e4 P 3c1770123dc14bba80e00db5f182b585: Bootstrap replayed 1/1 log segments. Stats: ops{read=255 overwritten=0 applied=255 ignored=0} inserts{seen=907 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:42:59.923804 618 tablet_bootstrap.cc:427] T e0af316d40d143a8b387933151adf0e4 P 3c1770123dc14bba80e00db5f182b585: Bootstrap complete. I0703 01:42:59.924202 618 ts_tablet_manager.cc:1045] T e0af316d40d143a8b387933151adf0e4 P 3c1770123dc14bba80e00db5f182b585: Time spent bootstrapping tablet: real 1.568s user 0.329s sys 0.029s I0703 01:42:59.926460 618 raft_consensus.cc:325] T e0af316d40d143a8b387933151adf0e4 P 3c1770123dc14bba80e00db5f182b585 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 243 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "3c1770123dc14bba80e00db5f182b585" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 44326 } } I0703 01:42:59.927351 618 raft_consensus.cc:628] T e0af316d40d143a8b387933151adf0e4 P 3c1770123dc14bba80e00db5f182b585 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c1770123dc14bba80e00db5f182b585, State: Initialized, Role: FOLLOWER I0703 01:42:59.927618 618 consensus_queue.cc:227] T e0af316d40d143a8b387933151adf0e4 P 3c1770123dc14bba80e00db5f182b585 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 255, Last appended: 1.255, Last appended by leader: 255, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 243 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "3c1770123dc14bba80e00db5f182b585" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 44326 } } W0703 01:42:59.936434 31328 consensus_peers.cc:433] T 3d7a6e87780a49949cf7e024817cf3a3 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 56eaab17f812433a8f7880f0744e1744 (127.5.240.199:57714): Couldn't send request to peer 56eaab17f812433a8f7880f0744e1744 for tablet 3d7a6e87780a49949cf7e024817cf3a3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 3 times. I0703 01:42:59.939875 618 ts_tablet_manager.cc:1073] T e0af316d40d143a8b387933151adf0e4 P 3c1770123dc14bba80e00db5f182b585: Time spent starting tablet: real 0.016s user 0.002s sys 0.002s I0703 01:42:59.940722 31537 tablet_copy_service.cc:325] P de93b58c2d67488dae12285e1806ebac: Request end of tablet copy session 3c1770123dc14bba80e00db5f182b585-e0af316d40d143a8b387933151adf0e4 received from {username='jenkins'} at 127.5.240.197:55267 I0703 01:42:59.940800 31537 tablet_copy_service.cc:413] P de93b58c2d67488dae12285e1806ebac: ending tablet copy session 3c1770123dc14bba80e00db5f182b585-e0af316d40d143a8b387933151adf0e4 on tablet e0af316d40d143a8b387933151adf0e4 with peer 3c1770123dc14bba80e00db5f182b585 I0703 01:42:59.967633 822 ts_tablet_manager.cc:669] T b2c8b58f935a4d00b5cf3e2cedec4d81 P 768d8de4b2ba40f180b18194c28be589: Initiating tablet copy from peer de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321) I0703 01:42:59.967880 822 tablet_copy_client.cc:239] T b2c8b58f935a4d00b5cf3e2cedec4d81 P 768d8de4b2ba40f180b18194c28be589: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.195:59321 I0703 01:43:00.004657 733 tablet_bootstrap.cc:427] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be: Bootstrap replayed 1/1 log segments. Stats: ops{read=198 overwritten=0 applied=198 ignored=0} inserts{seen=850 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:43:00.005023 733 tablet_bootstrap.cc:427] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be: Bootstrap complete. I0703 01:43:00.005223 733 ts_tablet_manager.cc:1045] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be: Time spent bootstrapping tablet: real 1.166s user 0.267s sys 0.019s I0703 01:43:00.018831 31537 tablet_copy_service.cc:135] P de93b58c2d67488dae12285e1806ebac: Received BeginTabletCopySession request for tablet b2c8b58f935a4d00b5cf3e2cedec4d81 from peer 768d8de4b2ba40f180b18194c28be589 ({username='jenkins'} at 127.5.240.198:45428) I0703 01:43:00.018955 31537 tablet_copy_service.cc:156] P de93b58c2d67488dae12285e1806ebac: Beginning new tablet copy session on tablet b2c8b58f935a4d00b5cf3e2cedec4d81 from peer 768d8de4b2ba40f180b18194c28be589 at {username='jenkins'} at 127.5.240.198:45428: session id = 768d8de4b2ba40f180b18194c28be589-b2c8b58f935a4d00b5cf3e2cedec4d81 I0703 01:43:00.021306 31537 tablet_copy_source_session.cc:202] T b2c8b58f935a4d00b5cf3e2cedec4d81 P de93b58c2d67488dae12285e1806ebac: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:43:00.024475 822 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet b2c8b58f935a4d00b5cf3e2cedec4d81. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:43:00.025750 733 raft_consensus.cc:325] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 189 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } I0703 01:43:00.025975 733 raft_consensus.cc:628] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9dc68ace903455d809cb1f715fe36be, State: Initialized, Role: FOLLOWER I0703 01:43:00.026180 733 consensus_queue.cc:227] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 198, Last appended: 1.198, Last appended by leader: 198, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 189 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } I0703 01:43:00.026780 733 ts_tablet_manager.cc:1073] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be: Time spent starting tablet: real 0.021s user 0.003s sys 0.000s I0703 01:43:00.029666 31292 tablet_copy_service.cc:325] P 1288ca0989104bbea36e22f38c374218: Request end of tablet copy session d9dc68ace903455d809cb1f715fe36be-ffe66a5cfb564964951c619e51e7c47c received from {username='jenkins'} at 127.5.240.196:46328 I0703 01:43:00.029753 31292 tablet_copy_service.cc:413] P 1288ca0989104bbea36e22f38c374218: ending tablet copy session d9dc68ace903455d809cb1f715fe36be-ffe66a5cfb564964951c619e51e7c47c on tablet ffe66a5cfb564964951c619e51e7c47c with peer d9dc68ace903455d809cb1f715fe36be I0703 01:43:00.040711 32503 raft_consensus.cc:1026] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.188->[1.189-1.208] Dedup: 1.198->[1.199-1.208] I0703 01:43:00.077939 822 ts_tablet_manager.cc:1173] T b2c8b58f935a4d00b5cf3e2cedec4d81 P 768d8de4b2ba40f180b18194c28be589: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:43:00.079479 822 tablet_copy_client.cc:566] T b2c8b58f935a4d00b5cf3e2cedec4d81 P 768d8de4b2ba40f180b18194c28be589: tablet copy: Starting download of 0 data blocks... I0703 01:43:00.080742 822 tablet_copy_client.cc:529] T b2c8b58f935a4d00b5cf3e2cedec4d81 P 768d8de4b2ba40f180b18194c28be589: tablet copy: Starting download of 1 WAL segments... I0703 01:43:00.089684 822 tablet_copy_client.cc:405] T b2c8b58f935a4d00b5cf3e2cedec4d81 P 768d8de4b2ba40f180b18194c28be589: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:43:00.093647 822 ts_tablet_manager.cc:1028] T b2c8b58f935a4d00b5cf3e2cedec4d81 P 768d8de4b2ba40f180b18194c28be589: Bootstrapping tablet I0703 01:43:00.093878 822 tablet_bootstrap.cc:427] T b2c8b58f935a4d00b5cf3e2cedec4d81 P 768d8de4b2ba40f180b18194c28be589: Bootstrap starting. I0703 01:43:00.098592 32503 raft_consensus.cc:1026] T 454bab377db147a1a2f8241ecc2241f3 P d9dc68ace903455d809cb1f715fe36be [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[] I0703 01:43:00.185351 434 raft_consensus.cc:1026] T bbe64aa4b0e04f468b035048629b969b P 56eaab17f812433a8f7880f0744e1744 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.155->[1.156-1.197] Dedup: 1.164->[1.165-1.197] W0703 01:43:00.208019 31326 consensus_peers.cc:433] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 768d8de4b2ba40f180b18194c28be589 (127.5.240.198:42654): Couldn't send request to peer 768d8de4b2ba40f180b18194c28be589 for tablet 1f01bfdde47b4d37a0f77d199bc7b4d2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 4 times. I0703 01:43:00.222363 31514 consensus_peers.cc:455] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac -> Peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521): Closing peer: 1288ca0989104bbea36e22f38c374218 I0703 01:43:00.222544 31514 consensus_queue.cc:204] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 207, Committed index: 207, Last appended: 1.207, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } W0703 01:43:00.225188 31450 consensus_peers.cc:433] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac -> Peer d9dc68ace903455d809cb1f715fe36be (127.5.240.196:56543): Couldn't send request to peer d9dc68ace903455d809cb1f715fe36be for tablet 19c33e54ee9e476796de7de42ba495b6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19c33e54ee9e476796de7de42ba495b6. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:43:00.226668 31270 raft_consensus.cc:1081] T 19c33e54ee9e476796de7de42ba495b6 P 1288ca0989104bbea36e22f38c374218 [term 1 FOLLOWER]: Refusing update from remote peer de93b58c2d67488dae12285e1806ebac: Log matching property violated. Preceding OpId in replica: term: 1 index: 207. Preceding OpId from leader: term: 1 index: 208. (index mismatch) I0703 01:43:00.227880 597 consensus_queue.cc:962] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac [LEADER]: Connected to new peer: Peer: permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 208, Last known committed idx: 207, Time since last communication: 0.000s I0703 01:43:00.240442 597 raft_consensus.cc:2684] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac [term 1 LEADER]: Committing config change with OpId 1.208: config changed from index -1 to 208, VOTER d9dc68ace903455d809cb1f715fe36be (127.5.240.196) added. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } } I0703 01:43:00.245412 31150 catalog_manager.cc:3798] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac reported cstate change: config changed from index -1 to 208, VOTER d9dc68ace903455d809cb1f715fe36be (127.5.240.196) added. New cstate: current_term: 1 leader_uuid: "de93b58c2d67488dae12285e1806ebac" committed_config { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } } I0703 01:43:00.252120 31270 raft_consensus.cc:2684] T 19c33e54ee9e476796de7de42ba495b6 P 1288ca0989104bbea36e22f38c374218 [term 1 FOLLOWER]: Committing config change with OpId 1.208: config changed from index -1 to 208, VOTER d9dc68ace903455d809cb1f715fe36be (127.5.240.196) added. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } } I0703 01:43:00.363939 32638 raft_consensus.cc:1026] T e0af316d40d143a8b387933151adf0e4 P 3c1770123dc14bba80e00db5f182b585 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.242->[1.243-1.287] Dedup: 1.255->[1.256-1.287] W0703 01:43:00.400228 31328 consensus_peers.cc:433] T 3d7a6e87780a49949cf7e024817cf3a3 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 56eaab17f812433a8f7880f0744e1744 (127.5.240.199:57714): Couldn't send request to peer 56eaab17f812433a8f7880f0744e1744 for tablet 3d7a6e87780a49949cf7e024817cf3a3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 4 times. I0703 01:43:00.439921 863 ts_tablet_manager.cc:669] T 44f06af6e8ef46d78419aea845bb38a7 P 56eaab17f812433a8f7880f0744e1744: Initiating tablet copy from peer 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194:39827) I0703 01:43:00.443307 863 tablet_copy_client.cc:239] T 44f06af6e8ef46d78419aea845bb38a7 P 56eaab17f812433a8f7880f0744e1744: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.194:39827 I0703 01:43:00.444077 31415 tablet_copy_service.cc:135] P 2f3fdb7c089640c0b936a574bc64707e: Received BeginTabletCopySession request for tablet 44f06af6e8ef46d78419aea845bb38a7 from peer 56eaab17f812433a8f7880f0744e1744 ({username='jenkins'} at 127.5.240.199:50676) I0703 01:43:00.444190 31415 tablet_copy_service.cc:156] P 2f3fdb7c089640c0b936a574bc64707e: Beginning new tablet copy session on tablet 44f06af6e8ef46d78419aea845bb38a7 from peer 56eaab17f812433a8f7880f0744e1744 at {username='jenkins'} at 127.5.240.199:50676: session id = 56eaab17f812433a8f7880f0744e1744-44f06af6e8ef46d78419aea845bb38a7 I0703 01:43:00.446987 31415 tablet_copy_source_session.cc:202] T 44f06af6e8ef46d78419aea845bb38a7 P 2f3fdb7c089640c0b936a574bc64707e: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:43:00.449877 863 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 44f06af6e8ef46d78419aea845bb38a7. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:43:00.454577 863 ts_tablet_manager.cc:1173] T 44f06af6e8ef46d78419aea845bb38a7 P 56eaab17f812433a8f7880f0744e1744: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:43:00.467752 863 tablet_copy_client.cc:566] T 44f06af6e8ef46d78419aea845bb38a7 P 56eaab17f812433a8f7880f0744e1744: tablet copy: Starting download of 0 data blocks... I0703 01:43:00.467958 863 tablet_copy_client.cc:529] T 44f06af6e8ef46d78419aea845bb38a7 P 56eaab17f812433a8f7880f0744e1744: tablet copy: Starting download of 1 WAL segments... I0703 01:43:00.475270 863 tablet_copy_client.cc:405] T 44f06af6e8ef46d78419aea845bb38a7 P 56eaab17f812433a8f7880f0744e1744: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:43:00.477790 863 ts_tablet_manager.cc:1028] T 44f06af6e8ef46d78419aea845bb38a7 P 56eaab17f812433a8f7880f0744e1744: Bootstrapping tablet I0703 01:43:00.478804 863 tablet_bootstrap.cc:427] T 44f06af6e8ef46d78419aea845bb38a7 P 56eaab17f812433a8f7880f0744e1744: Bootstrap starting. W0703 01:43:00.514142 31448 consensus_peers.cc:433] T b2c8b58f935a4d00b5cf3e2cedec4d81 P de93b58c2d67488dae12285e1806ebac -> Peer 768d8de4b2ba40f180b18194c28be589 (127.5.240.198:42654): Couldn't send request to peer 768d8de4b2ba40f180b18194c28be589 for tablet b2c8b58f935a4d00b5cf3e2cedec4d81. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 2 times. I0703 01:43:00.579839 635 tablet_bootstrap.cc:427] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 768d8de4b2ba40f180b18194c28be589: Bootstrap replayed 1/1 log segments. Stats: ops{read=225 overwritten=0 applied=225 ignored=0} inserts{seen=912 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:43:00.580265 635 tablet_bootstrap.cc:427] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 768d8de4b2ba40f180b18194c28be589: Bootstrap complete. I0703 01:43:00.588098 635 ts_tablet_manager.cc:1045] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 768d8de4b2ba40f180b18194c28be589: Time spent bootstrapping tablet: real 2.016s user 0.305s sys 0.019s I0703 01:43:00.588845 635 raft_consensus.cc:325] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 768d8de4b2ba40f180b18194c28be589 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 210 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } I0703 01:43:00.589252 635 raft_consensus.cc:628] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 768d8de4b2ba40f180b18194c28be589 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768d8de4b2ba40f180b18194c28be589, State: Initialized, Role: FOLLOWER I0703 01:43:00.589442 635 consensus_queue.cc:227] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 768d8de4b2ba40f180b18194c28be589 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 225, Last appended: 1.225, Last appended by leader: 225, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 210 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } I0703 01:43:00.596809 635 ts_tablet_manager.cc:1073] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 768d8de4b2ba40f180b18194c28be589: Time spent starting tablet: real 0.009s user 0.002s sys 0.001s I0703 01:43:00.598047 31415 tablet_copy_service.cc:325] P 2f3fdb7c089640c0b936a574bc64707e: Request end of tablet copy session 768d8de4b2ba40f180b18194c28be589-1f01bfdde47b4d37a0f77d199bc7b4d2 received from {username='jenkins'} at 127.5.240.198:59612 I0703 01:43:00.598163 31415 tablet_copy_service.cc:413] P 2f3fdb7c089640c0b936a574bc64707e: ending tablet copy session 768d8de4b2ba40f180b18194c28be589-1f01bfdde47b4d37a0f77d199bc7b4d2 on tablet 1f01bfdde47b4d37a0f77d199bc7b4d2 with peer 768d8de4b2ba40f180b18194c28be589 I0703 01:43:00.657970 300 raft_consensus.cc:1026] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 768d8de4b2ba40f180b18194c28be589 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.209->[1.210-1.266] Dedup: 1.225->[1.226-1.266] I0703 01:43:00.829890 755 tablet_bootstrap.cc:427] T 3d7a6e87780a49949cf7e024817cf3a3 P 56eaab17f812433a8f7880f0744e1744: Bootstrap replayed 1/1 log segments. Stats: ops{read=227 overwritten=0 applied=226 ignored=0} inserts{seen=913 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates W0703 01:43:00.833694 31328 consensus_peers.cc:433] T 3d7a6e87780a49949cf7e024817cf3a3 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 56eaab17f812433a8f7880f0744e1744 (127.5.240.199:57714): Couldn't send request to peer 56eaab17f812433a8f7880f0744e1744 for tablet 3d7a6e87780a49949cf7e024817cf3a3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 5 times. I0703 01:43:00.846701 898 ts_tablet_manager.cc:669] T 19c33e54ee9e476796de7de42ba495b6 P d9dc68ace903455d809cb1f715fe36be: Initiating tablet copy from peer de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321) I0703 01:43:00.854753 755 tablet_bootstrap.cc:427] T 3d7a6e87780a49949cf7e024817cf3a3 P 56eaab17f812433a8f7880f0744e1744: Bootstrap complete. I0703 01:43:00.855072 755 ts_tablet_manager.cc:1045] T 3d7a6e87780a49949cf7e024817cf3a3 P 56eaab17f812433a8f7880f0744e1744: Time spent bootstrapping tablet: real 1.745s user 0.306s sys 0.025s I0703 01:43:00.855912 755 raft_consensus.cc:325] T 3d7a6e87780a49949cf7e024817cf3a3 P 56eaab17f812433a8f7880f0744e1744 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending transactions. Active config: opid_index: 214 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } I0703 01:43:00.856750 755 raft_consensus.cc:628] T 3d7a6e87780a49949cf7e024817cf3a3 P 56eaab17f812433a8f7880f0744e1744 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56eaab17f812433a8f7880f0744e1744, State: Initialized, Role: FOLLOWER I0703 01:43:00.862059 898 tablet_copy_client.cc:239] T 19c33e54ee9e476796de7de42ba495b6 P d9dc68ace903455d809cb1f715fe36be: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.195:59321 I0703 01:43:00.863203 31537 tablet_copy_service.cc:135] P de93b58c2d67488dae12285e1806ebac: Received BeginTabletCopySession request for tablet 19c33e54ee9e476796de7de42ba495b6 from peer d9dc68ace903455d809cb1f715fe36be ({username='jenkins'} at 127.5.240.196:37222) I0703 01:43:00.863317 31537 tablet_copy_service.cc:156] P de93b58c2d67488dae12285e1806ebac: Beginning new tablet copy session on tablet 19c33e54ee9e476796de7de42ba495b6 from peer d9dc68ace903455d809cb1f715fe36be at {username='jenkins'} at 127.5.240.196:37222: session id = d9dc68ace903455d809cb1f715fe36be-19c33e54ee9e476796de7de42ba495b6 I0703 01:43:00.866271 31537 tablet_copy_source_session.cc:202] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:43:00.886404 898 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 19c33e54ee9e476796de7de42ba495b6. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:43:00.857012 755 consensus_queue.cc:227] T 3d7a6e87780a49949cf7e024817cf3a3 P 56eaab17f812433a8f7880f0744e1744 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 226, Last appended: 1.227, Last appended by leader: 227, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 214 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } I0703 01:43:00.923163 755 ts_tablet_manager.cc:1073] T 3d7a6e87780a49949cf7e024817cf3a3 P 56eaab17f812433a8f7880f0744e1744: Time spent starting tablet: real 0.068s user 0.003s sys 0.001s I0703 01:43:00.928437 31415 tablet_copy_service.cc:325] P 2f3fdb7c089640c0b936a574bc64707e: Request end of tablet copy session 56eaab17f812433a8f7880f0744e1744-3d7a6e87780a49949cf7e024817cf3a3 received from {username='jenkins'} at 127.5.240.199:50676 I0703 01:43:00.928527 31415 tablet_copy_service.cc:413] P 2f3fdb7c089640c0b936a574bc64707e: ending tablet copy session 56eaab17f812433a8f7880f0744e1744-3d7a6e87780a49949cf7e024817cf3a3 on tablet 3d7a6e87780a49949cf7e024817cf3a3 with peer 56eaab17f812433a8f7880f0744e1744 I0703 01:43:00.950261 898 ts_tablet_manager.cc:1173] T 19c33e54ee9e476796de7de42ba495b6 P d9dc68ace903455d809cb1f715fe36be: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:43:00.950784 898 tablet_copy_client.cc:566] T 19c33e54ee9e476796de7de42ba495b6 P d9dc68ace903455d809cb1f715fe36be: tablet copy: Starting download of 0 data blocks... I0703 01:43:00.951009 898 tablet_copy_client.cc:529] T 19c33e54ee9e476796de7de42ba495b6 P d9dc68ace903455d809cb1f715fe36be: tablet copy: Starting download of 1 WAL segments... I0703 01:43:00.960971 898 tablet_copy_client.cc:405] T 19c33e54ee9e476796de7de42ba495b6 P d9dc68ace903455d809cb1f715fe36be: tablet copy: Tablet Copy complete. Replacing tablet superblock. W0703 01:43:00.966984 31328 consensus_peers.cc:433] T 44f06af6e8ef46d78419aea845bb38a7 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 56eaab17f812433a8f7880f0744e1744 (127.5.240.199:57714): Couldn't send request to peer 56eaab17f812433a8f7880f0744e1744 for tablet 44f06af6e8ef46d78419aea845bb38a7. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 2 times. I0703 01:43:00.968294 898 ts_tablet_manager.cc:1028] T 19c33e54ee9e476796de7de42ba495b6 P d9dc68ace903455d809cb1f715fe36be: Bootstrapping tablet I0703 01:43:00.970337 898 tablet_bootstrap.cc:427] T 19c33e54ee9e476796de7de42ba495b6 P d9dc68ace903455d809cb1f715fe36be: Bootstrap starting. W0703 01:43:01.027494 31448 consensus_peers.cc:433] T b2c8b58f935a4d00b5cf3e2cedec4d81 P de93b58c2d67488dae12285e1806ebac -> Peer 768d8de4b2ba40f180b18194c28be589 (127.5.240.198:42654): Couldn't send request to peer 768d8de4b2ba40f180b18194c28be589 for tablet b2c8b58f935a4d00b5cf3e2cedec4d81. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 3 times. I0703 01:43:01.218295 432 raft_consensus.cc:1026] T 3d7a6e87780a49949cf7e024817cf3a3 P 56eaab17f812433a8f7880f0744e1744 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.213->[1.214-1.259] Dedup: 1.227->[1.228-1.259] W0703 01:43:01.297600 31450 consensus_peers.cc:433] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac -> Peer d9dc68ace903455d809cb1f715fe36be (127.5.240.196:56543): Couldn't send request to peer d9dc68ace903455d809cb1f715fe36be for tablet 19c33e54ee9e476796de7de42ba495b6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 2 times. W0703 01:43:01.582970 31328 consensus_peers.cc:433] T 44f06af6e8ef46d78419aea845bb38a7 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 56eaab17f812433a8f7880f0744e1744 (127.5.240.199:57714): Couldn't send request to peer 56eaab17f812433a8f7880f0744e1744 for tablet 44f06af6e8ef46d78419aea845bb38a7. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 3 times. W0703 01:43:01.593147 31448 consensus_peers.cc:433] T b2c8b58f935a4d00b5cf3e2cedec4d81 P de93b58c2d67488dae12285e1806ebac -> Peer 768d8de4b2ba40f180b18194c28be589 (127.5.240.198:42654): Couldn't send request to peer 768d8de4b2ba40f180b18194c28be589 for tablet b2c8b58f935a4d00b5cf3e2cedec4d81. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 4 times. W0703 01:43:01.879606 31450 consensus_peers.cc:433] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac -> Peer d9dc68ace903455d809cb1f715fe36be (127.5.240.196:56543): Couldn't send request to peer d9dc68ace903455d809cb1f715fe36be for tablet 19c33e54ee9e476796de7de42ba495b6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 3 times. W0703 01:43:01.999050 31448 consensus_peers.cc:433] T b2c8b58f935a4d00b5cf3e2cedec4d81 P de93b58c2d67488dae12285e1806ebac -> Peer 768d8de4b2ba40f180b18194c28be589 (127.5.240.198:42654): Couldn't send request to peer 768d8de4b2ba40f180b18194c28be589 for tablet b2c8b58f935a4d00b5cf3e2cedec4d81. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 5 times. W0703 01:43:02.102656 31328 consensus_peers.cc:433] T 44f06af6e8ef46d78419aea845bb38a7 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 56eaab17f812433a8f7880f0744e1744 (127.5.240.199:57714): Couldn't send request to peer 56eaab17f812433a8f7880f0744e1744 for tablet 44f06af6e8ef46d78419aea845bb38a7. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 4 times. I0703 01:43:02.112888 822 tablet_bootstrap.cc:427] T b2c8b58f935a4d00b5cf3e2cedec4d81 P 768d8de4b2ba40f180b18194c28be589: Bootstrap replayed 1/1 log segments. Stats: ops{read=224 overwritten=0 applied=224 ignored=0} inserts{seen=911 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:43:02.113682 822 tablet_bootstrap.cc:427] T b2c8b58f935a4d00b5cf3e2cedec4d81 P 768d8de4b2ba40f180b18194c28be589: Bootstrap complete. I0703 01:43:02.114037 822 ts_tablet_manager.cc:1045] T b2c8b58f935a4d00b5cf3e2cedec4d81 P 768d8de4b2ba40f180b18194c28be589: Time spent bootstrapping tablet: real 2.020s user 0.296s sys 0.026s I0703 01:43:02.114815 822 raft_consensus.cc:325] T b2c8b58f935a4d00b5cf3e2cedec4d81 P 768d8de4b2ba40f180b18194c28be589 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } I0703 01:43:02.115082 822 raft_consensus.cc:628] T b2c8b58f935a4d00b5cf3e2cedec4d81 P 768d8de4b2ba40f180b18194c28be589 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768d8de4b2ba40f180b18194c28be589, State: Initialized, Role: FOLLOWER I0703 01:43:02.116536 822 consensus_queue.cc:227] T b2c8b58f935a4d00b5cf3e2cedec4d81 P 768d8de4b2ba40f180b18194c28be589 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 224, Last appended: 1.224, Last appended by leader: 224, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } I0703 01:43:02.128402 822 ts_tablet_manager.cc:1073] T b2c8b58f935a4d00b5cf3e2cedec4d81 P 768d8de4b2ba40f180b18194c28be589: Time spent starting tablet: real 0.014s user 0.003s sys 0.004s I0703 01:43:02.129587 31537 tablet_copy_service.cc:325] P de93b58c2d67488dae12285e1806ebac: Request end of tablet copy session 768d8de4b2ba40f180b18194c28be589-b2c8b58f935a4d00b5cf3e2cedec4d81 received from {username='jenkins'} at 127.5.240.198:45428 I0703 01:43:02.129691 31537 tablet_copy_service.cc:413] P de93b58c2d67488dae12285e1806ebac: ending tablet copy session 768d8de4b2ba40f180b18194c28be589-b2c8b58f935a4d00b5cf3e2cedec4d81 on tablet b2c8b58f935a4d00b5cf3e2cedec4d81 with peer 768d8de4b2ba40f180b18194c28be589 I0703 01:43:02.310673 863 tablet_bootstrap.cc:427] T 44f06af6e8ef46d78419aea845bb38a7 P 56eaab17f812433a8f7880f0744e1744: Bootstrap replayed 1/1 log segments. Stats: ops{read=210 overwritten=0 applied=210 ignored=0} inserts{seen=859 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:43:02.312232 863 tablet_bootstrap.cc:427] T 44f06af6e8ef46d78419aea845bb38a7 P 56eaab17f812433a8f7880f0744e1744: Bootstrap complete. I0703 01:43:02.312469 863 ts_tablet_manager.cc:1045] T 44f06af6e8ef46d78419aea845bb38a7 P 56eaab17f812433a8f7880f0744e1744: Time spent bootstrapping tablet: real 1.834s user 0.282s sys 0.024s I0703 01:43:02.313169 863 raft_consensus.cc:325] T 44f06af6e8ef46d78419aea845bb38a7 P 56eaab17f812433a8f7880f0744e1744 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 203 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } I0703 01:43:02.313446 863 raft_consensus.cc:628] T 44f06af6e8ef46d78419aea845bb38a7 P 56eaab17f812433a8f7880f0744e1744 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56eaab17f812433a8f7880f0744e1744, State: Initialized, Role: FOLLOWER I0703 01:43:02.313714 863 consensus_queue.cc:227] T 44f06af6e8ef46d78419aea845bb38a7 P 56eaab17f812433a8f7880f0744e1744 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 210, Last appended: 1.210, Last appended by leader: 210, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 203 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } I0703 01:43:02.328783 863 ts_tablet_manager.cc:1073] T 44f06af6e8ef46d78419aea845bb38a7 P 56eaab17f812433a8f7880f0744e1744: Time spent starting tablet: real 0.016s user 0.002s sys 0.001s I0703 01:43:02.332432 31415 tablet_copy_service.cc:325] P 2f3fdb7c089640c0b936a574bc64707e: Request end of tablet copy session 56eaab17f812433a8f7880f0744e1744-44f06af6e8ef46d78419aea845bb38a7 received from {username='jenkins'} at 127.5.240.199:50676 I0703 01:43:02.332520 31415 tablet_copy_service.cc:413] P 2f3fdb7c089640c0b936a574bc64707e: ending tablet copy session 56eaab17f812433a8f7880f0744e1744-44f06af6e8ef46d78419aea845bb38a7 on tablet 44f06af6e8ef46d78419aea845bb38a7 with peer 56eaab17f812433a8f7880f0744e1744 W0703 01:43:02.360999 31450 consensus_peers.cc:433] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac -> Peer d9dc68ace903455d809cb1f715fe36be (127.5.240.196:56543): Couldn't send request to peer d9dc68ace903455d809cb1f715fe36be for tablet 19c33e54ee9e476796de7de42ba495b6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 4 times. I0703 01:43:02.367156 898 tablet_bootstrap.cc:427] T 19c33e54ee9e476796de7de42ba495b6 P d9dc68ace903455d809cb1f715fe36be: Bootstrap replayed 1/1 log segments. Stats: ops{read=222 overwritten=0 applied=221 ignored=0} inserts{seen=873 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates I0703 01:43:02.367571 898 tablet_bootstrap.cc:427] T 19c33e54ee9e476796de7de42ba495b6 P d9dc68ace903455d809cb1f715fe36be: Bootstrap complete. I0703 01:43:02.367830 898 ts_tablet_manager.cc:1045] T 19c33e54ee9e476796de7de42ba495b6 P d9dc68ace903455d809cb1f715fe36be: Time spent bootstrapping tablet: real 1.398s user 0.296s sys 0.022s I0703 01:43:02.368624 898 raft_consensus.cc:325] T 19c33e54ee9e476796de7de42ba495b6 P d9dc68ace903455d809cb1f715fe36be [term 1 FOLLOWER]: Replica starting. Triggering 1 pending transactions. Active config: opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } I0703 01:43:02.369509 898 raft_consensus.cc:628] T 19c33e54ee9e476796de7de42ba495b6 P d9dc68ace903455d809cb1f715fe36be [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9dc68ace903455d809cb1f715fe36be, State: Initialized, Role: FOLLOWER I0703 01:43:02.369705 898 consensus_queue.cc:227] T 19c33e54ee9e476796de7de42ba495b6 P d9dc68ace903455d809cb1f715fe36be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 221, Last appended: 1.222, Last appended by leader: 222, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } I0703 01:43:02.371939 898 ts_tablet_manager.cc:1073] T 19c33e54ee9e476796de7de42ba495b6 P d9dc68ace903455d809cb1f715fe36be: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s I0703 01:43:02.372690 31537 tablet_copy_service.cc:325] P de93b58c2d67488dae12285e1806ebac: Request end of tablet copy session d9dc68ace903455d809cb1f715fe36be-19c33e54ee9e476796de7de42ba495b6 received from {username='jenkins'} at 127.5.240.196:37222 I0703 01:43:02.372759 31537 tablet_copy_service.cc:413] P de93b58c2d67488dae12285e1806ebac: ending tablet copy session d9dc68ace903455d809cb1f715fe36be-19c33e54ee9e476796de7de42ba495b6 on tablet 19c33e54ee9e476796de7de42ba495b6 with peer d9dc68ace903455d809cb1f715fe36be I0703 01:43:02.410830 300 raft_consensus.cc:1026] T b2c8b58f935a4d00b5cf3e2cedec4d81 P 768d8de4b2ba40f180b18194c28be589 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.216->[1.217-1.258] Dedup: 1.224->[1.225-1.258] I0703 01:43:02.701352 431 raft_consensus.cc:1026] T 44f06af6e8ef46d78419aea845bb38a7 P 56eaab17f812433a8f7880f0744e1744 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.202->[1.203-1.233] Dedup: 1.210->[1.211-1.233] I0703 01:43:02.771250 32504 raft_consensus.cc:1026] T 19c33e54ee9e476796de7de42ba495b6 P d9dc68ace903455d809cb1f715fe36be [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.207->[1.208-1.242] Dedup: 1.222->[1.223-1.242] I0703 01:43:02.961578 31393 consensus_peers.cc:455] T 479104ce31404ee4ad2488d9f46509e4 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 3c1770123dc14bba80e00db5f182b585 (127.5.240.197:44326): Closing peer: 3c1770123dc14bba80e00db5f182b585 I0703 01:43:02.961701 31393 consensus_peers.cc:455] T 479104ce31404ee4ad2488d9f46509e4 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521): Closing peer: 1288ca0989104bbea36e22f38c374218 I0703 01:43:02.961988 31393 consensus_queue.cc:204] T 479104ce31404ee4ad2488d9f46509e4 P 2f3fdb7c089640c0b936a574bc64707e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 203, Committed index: 203, Last appended: 1.203, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 204 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "3c1770123dc14bba80e00db5f182b585" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 44326 } } I0703 01:43:02.965184 32638 raft_consensus.cc:1081] T 479104ce31404ee4ad2488d9f46509e4 P 3c1770123dc14bba80e00db5f182b585 [term 1 FOLLOWER]: Refusing update from remote peer 2f3fdb7c089640c0b936a574bc64707e: Log matching property violated. Preceding OpId in replica: term: 1 index: 203. Preceding OpId from leader: term: 1 index: 204. (index mismatch) I0703 01:43:02.966786 765 consensus_queue.cc:962] T 479104ce31404ee4ad2488d9f46509e4 P 2f3fdb7c089640c0b936a574bc64707e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c1770123dc14bba80e00db5f182b585" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 44326 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 204, Last known committed idx: 203, Time since last communication: 0.002s I0703 01:43:02.971952 765 raft_consensus.cc:2684] T 479104ce31404ee4ad2488d9f46509e4 P 2f3fdb7c089640c0b936a574bc64707e [term 1 LEADER]: Committing config change with OpId 1.204: config changed from index 138 to 204, VOTER 1288ca0989104bbea36e22f38c374218 (127.5.240.193) evicted. New config: { opid_index: 204 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "3c1770123dc14bba80e00db5f182b585" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 44326 } } } I0703 01:43:02.973733 32638 raft_consensus.cc:2684] T 479104ce31404ee4ad2488d9f46509e4 P 3c1770123dc14bba80e00db5f182b585 [term 1 FOLLOWER]: Committing config change with OpId 1.204: config changed from index 138 to 204, VOTER 1288ca0989104bbea36e22f38c374218 (127.5.240.193) evicted. New config: { opid_index: 204 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "3c1770123dc14bba80e00db5f182b585" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 44326 } } } I0703 01:43:02.979676 31149 catalog_manager.cc:3798] T 479104ce31404ee4ad2488d9f46509e4 P 2f3fdb7c089640c0b936a574bc64707e reported cstate change: config changed from index 138 to 204, VOTER 1288ca0989104bbea36e22f38c374218 (127.5.240.193) evicted. New cstate: current_term: 1 leader_uuid: "2f3fdb7c089640c0b936a574bc64707e" committed_config { opid_index: 204 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "3c1770123dc14bba80e00db5f182b585" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 44326 } } } I0703 01:43:03.081827 31149 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 479104ce31404ee4ad2488d9f46509e4 on 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521) (TS 1288ca0989104bbea36e22f38c374218 not found in new config with opid_index 204) I0703 01:43:03.082774 31250 tablet_service.cc:800] Processing DeleteTablet for tablet 479104ce31404ee4ad2488d9f46509e4 with delete_type TABLET_DATA_TOMBSTONED (TS 1288ca0989104bbea36e22f38c374218 not found in new config with opid_index 204) from {username='jenkins'} at 127.5.240.193:33318 I0703 01:43:03.085122 1066 tablet_replica.cc:261] T 479104ce31404ee4ad2488d9f46509e4 P 1288ca0989104bbea36e22f38c374218: stopping tablet replica I0703 01:43:03.088407 1066 maintenance_manager.cc:237] P 1288ca0989104bbea36e22f38c374218: Unregistered op CompactRowSetsOp(479104ce31404ee4ad2488d9f46509e4) I0703 01:43:03.095161 1066 maintenance_manager.cc:237] P 1288ca0989104bbea36e22f38c374218: Unregistered op MinorDeltaCompactionOp(479104ce31404ee4ad2488d9f46509e4) I0703 01:43:03.095299 1066 maintenance_manager.cc:237] P 1288ca0989104bbea36e22f38c374218: Unregistered op MajorDeltaCompactionOp(479104ce31404ee4ad2488d9f46509e4) I0703 01:43:03.095402 1066 maintenance_manager.cc:237] P 1288ca0989104bbea36e22f38c374218: Unregistered op UndoDeltaBlockGCOp(479104ce31404ee4ad2488d9f46509e4) I0703 01:43:03.095528 1066 maintenance_manager.cc:237] P 1288ca0989104bbea36e22f38c374218: Unregistered op FlushMRSOp(479104ce31404ee4ad2488d9f46509e4) I0703 01:43:03.096458 1066 maintenance_manager.cc:237] P 1288ca0989104bbea36e22f38c374218: Unregistered op FlushDeltaMemStoresOp(479104ce31404ee4ad2488d9f46509e4) I0703 01:43:03.096541 1066 maintenance_manager.cc:237] P 1288ca0989104bbea36e22f38c374218: Unregistered op LogGCOp(479104ce31404ee4ad2488d9f46509e4) I0703 01:43:03.097007 1066 raft_consensus.cc:2040] T 479104ce31404ee4ad2488d9f46509e4 P 1288ca0989104bbea36e22f38c374218 [term 1 FOLLOWER]: Raft consensus shutting down. I0703 01:43:03.097512 1066 raft_consensus.cc:2067] T 479104ce31404ee4ad2488d9f46509e4 P 1288ca0989104bbea36e22f38c374218 [term 1 FOLLOWER]: Raft consensus is shut down! I0703 01:43:03.113947 1066 ts_tablet_manager.cc:1369] T 479104ce31404ee4ad2488d9f46509e4 P 1288ca0989104bbea36e22f38c374218: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:43:03.137312 1066 ts_tablet_manager.cc:1382] T 479104ce31404ee4ad2488d9f46509e4 P 1288ca0989104bbea36e22f38c374218: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.203 I0703 01:43:03.142012 1066 log.cc:986] T 479104ce31404ee4ad2488d9f46509e4 P 1288ca0989104bbea36e22f38c374218: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal/wals/479104ce31404ee4ad2488d9f46509e4 I0703 01:43:03.149863 31134 catalog_manager.cc:3103] TS 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521): tablet 479104ce31404ee4ad2488d9f46509e4 (table rebalance_test_table_2 [id=b1de6184513b4ad5b539fd284d2c95d9]) successfully deleted I0703 01:43:04.012593 31515 consensus_peers.cc:455] T 5eaa9c06d73c4e23b51fae2688eaf26c P de93b58c2d67488dae12285e1806ebac -> Peer d9dc68ace903455d809cb1f715fe36be (127.5.240.196:56543): Closing peer: d9dc68ace903455d809cb1f715fe36be I0703 01:43:04.012712 31515 consensus_peers.cc:455] T 5eaa9c06d73c4e23b51fae2688eaf26c P de93b58c2d67488dae12285e1806ebac -> Peer 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194:39827): Closing peer: 2f3fdb7c089640c0b936a574bc64707e I0703 01:43:04.013069 31515 consensus_queue.cc:204] T 5eaa9c06d73c4e23b51fae2688eaf26c P de93b58c2d67488dae12285e1806ebac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 336, Committed index: 336, Last appended: 1.336, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 337 OBSOLETE_local: false peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } I0703 01:43:04.014974 32504 raft_consensus.cc:1081] T 5eaa9c06d73c4e23b51fae2688eaf26c P d9dc68ace903455d809cb1f715fe36be [term 1 FOLLOWER]: Refusing update from remote peer de93b58c2d67488dae12285e1806ebac: Log matching property violated. Preceding OpId in replica: term: 1 index: 336. Preceding OpId from leader: term: 1 index: 337. (index mismatch) I0703 01:43:04.016563 954 consensus_queue.cc:962] T 5eaa9c06d73c4e23b51fae2688eaf26c P de93b58c2d67488dae12285e1806ebac [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 337, Last known committed idx: 336, Time since last communication: 0.000s I0703 01:43:04.019616 859 raft_consensus.cc:2684] T 5eaa9c06d73c4e23b51fae2688eaf26c P de93b58c2d67488dae12285e1806ebac [term 1 LEADER]: Committing config change with OpId 1.337: config changed from index 216 to 337, VOTER 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194) evicted. New config: { opid_index: 337 OBSOLETE_local: false peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } } I0703 01:43:04.024353 32502 raft_consensus.cc:2684] T 5eaa9c06d73c4e23b51fae2688eaf26c P d9dc68ace903455d809cb1f715fe36be [term 1 FOLLOWER]: Committing config change with OpId 1.337: config changed from index 216 to 337, VOTER 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194) evicted. New config: { opid_index: 337 OBSOLETE_local: false peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } } I0703 01:43:04.030117 31150 catalog_manager.cc:3798] T 5eaa9c06d73c4e23b51fae2688eaf26c P de93b58c2d67488dae12285e1806ebac reported cstate change: config changed from index 216 to 337, VOTER 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194) evicted. New cstate: current_term: 1 leader_uuid: "de93b58c2d67488dae12285e1806ebac" committed_config { opid_index: 337 OBSOLETE_local: false peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } } I0703 01:43:04.060941 31150 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 5eaa9c06d73c4e23b51fae2688eaf26c on 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194:39827) (TS 2f3fdb7c089640c0b936a574bc64707e not found in new config with opid_index 337) I0703 01:43:04.063627 31375 tablet_service.cc:800] Processing DeleteTablet for tablet 5eaa9c06d73c4e23b51fae2688eaf26c with delete_type TABLET_DATA_TOMBSTONED (TS 2f3fdb7c089640c0b936a574bc64707e not found in new config with opid_index 337) from {username='jenkins'} at 127.5.240.194:58888 I0703 01:43:04.065739 1127 tablet_replica.cc:261] T 5eaa9c06d73c4e23b51fae2688eaf26c P 2f3fdb7c089640c0b936a574bc64707e: stopping tablet replica I0703 01:43:04.065997 1127 maintenance_manager.cc:237] P 2f3fdb7c089640c0b936a574bc64707e: Unregistered op CompactRowSetsOp(5eaa9c06d73c4e23b51fae2688eaf26c) I0703 01:43:04.066112 1127 maintenance_manager.cc:237] P 2f3fdb7c089640c0b936a574bc64707e: Unregistered op MinorDeltaCompactionOp(5eaa9c06d73c4e23b51fae2688eaf26c) I0703 01:43:04.066195 1127 maintenance_manager.cc:237] P 2f3fdb7c089640c0b936a574bc64707e: Unregistered op MajorDeltaCompactionOp(5eaa9c06d73c4e23b51fae2688eaf26c) I0703 01:43:04.076524 1127 maintenance_manager.cc:237] P 2f3fdb7c089640c0b936a574bc64707e: Unregistered op UndoDeltaBlockGCOp(5eaa9c06d73c4e23b51fae2688eaf26c) I0703 01:43:04.076767 1127 maintenance_manager.cc:237] P 2f3fdb7c089640c0b936a574bc64707e: Unregistered op FlushMRSOp(5eaa9c06d73c4e23b51fae2688eaf26c) I0703 01:43:04.077114 1127 maintenance_manager.cc:237] P 2f3fdb7c089640c0b936a574bc64707e: Unregistered op FlushDeltaMemStoresOp(5eaa9c06d73c4e23b51fae2688eaf26c) I0703 01:43:04.077378 1127 maintenance_manager.cc:237] P 2f3fdb7c089640c0b936a574bc64707e: Unregistered op LogGCOp(5eaa9c06d73c4e23b51fae2688eaf26c) I0703 01:43:04.077527 1127 raft_consensus.cc:2040] T 5eaa9c06d73c4e23b51fae2688eaf26c P 2f3fdb7c089640c0b936a574bc64707e [term 1 FOLLOWER]: Raft consensus shutting down. I0703 01:43:04.079488 1127 raft_consensus.cc:2067] T 5eaa9c06d73c4e23b51fae2688eaf26c P 2f3fdb7c089640c0b936a574bc64707e [term 1 FOLLOWER]: Raft consensus is shut down! I0703 01:43:04.082276 1127 ts_tablet_manager.cc:1369] T 5eaa9c06d73c4e23b51fae2688eaf26c P 2f3fdb7c089640c0b936a574bc64707e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:43:04.093997 1127 ts_tablet_manager.cc:1382] T 5eaa9c06d73c4e23b51fae2688eaf26c P 2f3fdb7c089640c0b936a574bc64707e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.336 I0703 01:43:04.094120 1127 log.cc:986] T 5eaa9c06d73c4e23b51fae2688eaf26c P 2f3fdb7c089640c0b936a574bc64707e: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal/wals/5eaa9c06d73c4e23b51fae2688eaf26c I0703 01:43:04.099149 31133 catalog_manager.cc:3103] TS 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194:39827): tablet 5eaa9c06d73c4e23b51fae2688eaf26c (table rebalance_test_table_1 [id=7d7c285baa3a4e26a62b27d088cedcc9]) successfully deleted I0703 01:43:04.966655 31394 consensus_peers.cc:455] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 768d8de4b2ba40f180b18194c28be589 (127.5.240.198:42654): Closing peer: 768d8de4b2ba40f180b18194c28be589 I0703 01:43:04.966756 31394 consensus_peers.cc:455] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e -> Peer de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321): Closing peer: de93b58c2d67488dae12285e1806ebac I0703 01:43:04.966903 31394 consensus_queue.cc:204] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 327, Committed index: 327, Last appended: 1.327, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 328 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } I0703 01:43:04.969380 300 raft_consensus.cc:1081] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 768d8de4b2ba40f180b18194c28be589 [term 1 FOLLOWER]: Refusing update from remote peer 2f3fdb7c089640c0b936a574bc64707e: Log matching property violated. Preceding OpId in replica: term: 1 index: 327. Preceding OpId from leader: term: 1 index: 328. (index mismatch) I0703 01:43:04.971432 1150 consensus_queue.cc:962] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e [LEADER]: Connected to new peer: Peer: permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 328, Last known committed idx: 327, Time since last communication: 0.000s I0703 01:43:04.976264 1150 raft_consensus.cc:2684] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e [term 1 LEADER]: Committing config change with OpId 1.328: config changed from index 210 to 328, VOTER de93b58c2d67488dae12285e1806ebac (127.5.240.195) evicted. New config: { opid_index: 328 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } } I0703 01:43:04.982823 31149 catalog_manager.cc:3798] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e reported cstate change: config changed from index 210 to 328, VOTER de93b58c2d67488dae12285e1806ebac (127.5.240.195) evicted. New cstate: current_term: 1 leader_uuid: "2f3fdb7c089640c0b936a574bc64707e" committed_config { opid_index: 328 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } } I0703 01:43:04.982789 300 raft_consensus.cc:2684] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 768d8de4b2ba40f180b18194c28be589 [term 1 FOLLOWER]: Committing config change with OpId 1.328: config changed from index 210 to 328, VOTER de93b58c2d67488dae12285e1806ebac (127.5.240.195) evicted. New config: { opid_index: 328 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } } I0703 01:43:05.035374 31149 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 1f01bfdde47b4d37a0f77d199bc7b4d2 on de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321) (TS de93b58c2d67488dae12285e1806ebac not found in new config with opid_index 328) I0703 01:43:05.036269 31497 tablet_service.cc:800] Processing DeleteTablet for tablet 1f01bfdde47b4d37a0f77d199bc7b4d2 with delete_type TABLET_DATA_TOMBSTONED (TS de93b58c2d67488dae12285e1806ebac not found in new config with opid_index 328) from {username='jenkins'} at 127.5.240.195:46850 I0703 01:43:05.044891 1191 tablet_replica.cc:261] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P de93b58c2d67488dae12285e1806ebac: stopping tablet replica I0703 01:43:05.045192 1191 maintenance_manager.cc:237] P de93b58c2d67488dae12285e1806ebac: Unregistered op CompactRowSetsOp(1f01bfdde47b4d37a0f77d199bc7b4d2) I0703 01:43:05.045387 1191 maintenance_manager.cc:237] P de93b58c2d67488dae12285e1806ebac: Unregistered op MinorDeltaCompactionOp(1f01bfdde47b4d37a0f77d199bc7b4d2) I0703 01:43:05.045516 1191 maintenance_manager.cc:237] P de93b58c2d67488dae12285e1806ebac: Unregistered op MajorDeltaCompactionOp(1f01bfdde47b4d37a0f77d199bc7b4d2) I0703 01:43:05.053547 1191 maintenance_manager.cc:237] P de93b58c2d67488dae12285e1806ebac: Unregistered op UndoDeltaBlockGCOp(1f01bfdde47b4d37a0f77d199bc7b4d2) I0703 01:43:05.053704 1191 maintenance_manager.cc:237] P de93b58c2d67488dae12285e1806ebac: Unregistered op FlushMRSOp(1f01bfdde47b4d37a0f77d199bc7b4d2) I0703 01:43:05.053803 1191 maintenance_manager.cc:237] P de93b58c2d67488dae12285e1806ebac: Unregistered op FlushDeltaMemStoresOp(1f01bfdde47b4d37a0f77d199bc7b4d2) I0703 01:43:05.053884 1191 maintenance_manager.cc:237] P de93b58c2d67488dae12285e1806ebac: Unregistered op LogGCOp(1f01bfdde47b4d37a0f77d199bc7b4d2) I0703 01:43:05.054388 1191 raft_consensus.cc:2040] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P de93b58c2d67488dae12285e1806ebac [term 1 FOLLOWER]: Raft consensus shutting down. I0703 01:43:05.054800 1191 pending_rounds.cc:63] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P de93b58c2d67488dae12285e1806ebac: Trying to abort 1 pending transactions. I0703 01:43:05.054879 1191 pending_rounds.cc:68] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P de93b58c2d67488dae12285e1806ebac: Aborting transaction as it isn't in flight: id { term: 1 index: 327 } timestamp: 6269367848805740544 op_type: WRITE_OP write_request { tablet_id: "1f01bfdde47b4d37a0f77d199bc7b4d2" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true } } row_operations { rows: "\001\007\000\024\203;rO\216w\347\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "b8b87dd7b09bd858" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 6269367848765595648 } request_id { client_id: "917d67f52cfc466888988d31b04b3654" seq_no: 1028 first_incomplete_seq_no: 1028 attempt_no: 0 } I0703 01:43:05.055627 1191 raft_consensus.cc:2067] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P de93b58c2d67488dae12285e1806ebac [term 1 FOLLOWER]: Raft consensus is shut down! I0703 01:43:05.061177 1191 ts_tablet_manager.cc:1369] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P de93b58c2d67488dae12285e1806ebac: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:43:05.075413 1191 ts_tablet_manager.cc:1382] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P de93b58c2d67488dae12285e1806ebac: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.327 I0703 01:43:05.075651 1191 log.cc:986] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P de93b58c2d67488dae12285e1806ebac: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal/wals/1f01bfdde47b4d37a0f77d199bc7b4d2 I0703 01:43:05.081887 31136 catalog_manager.cc:3103] TS de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321): tablet 1f01bfdde47b4d37a0f77d199bc7b4d2 (table rebalance_test_table_1 [id=7d7c285baa3a4e26a62b27d088cedcc9]) successfully deleted I0703 01:43:05.611085 31272 tablet_service.cc:1140] Received LeaderStepDown RPC: tablet_id: "bbe64aa4b0e04f468b035048629b969b" dest_uuid: "1288ca0989104bbea36e22f38c374218" from {username='jenkins'} at 127.5.240.193:33490 I0703 01:43:05.611348 31272 raft_consensus.cc:552] T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218 [term 1 LEADER]: Received request to step down I0703 01:43:05.611399 31272 raft_consensus.cc:628] T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1288ca0989104bbea36e22f38c374218, State: Running, Role: LEADER I0703 01:43:05.611662 31272 consensus_queue.cc:227] T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 368, Committed index: 368, Last appended: 1.368, Last appended by leader: 368, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } I0703 01:43:05.611860 31272 consensus_peers.cc:455] T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218 -> Peer 56eaab17f812433a8f7880f0744e1744 (127.5.240.199:57714): Closing peer: 56eaab17f812433a8f7880f0744e1744 I0703 01:43:05.611922 31272 consensus_peers.cc:455] T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218 -> Peer de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321): Closing peer: de93b58c2d67488dae12285e1806ebac I0703 01:43:05.611981 31272 raft_consensus.cc:2730] T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218: Snoozing failure detection for 3.000s (explicit stepdown request) I0703 01:43:07.045816 31272 consensus_peers.cc:455] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218 -> Peer 768d8de4b2ba40f180b18194c28be589 (127.5.240.198:42654): Closing peer: 768d8de4b2ba40f180b18194c28be589 I0703 01:43:07.045888 31272 consensus_peers.cc:455] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218 -> Peer de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321): Closing peer: de93b58c2d67488dae12285e1806ebac I0703 01:43:07.046108 31272 consensus_queue.cc:204] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 251, Committed index: 251, Last appended: 1.251, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } I0703 01:43:07.048806 32767 raft_consensus.cc:1081] T a200f8a763f14d54a1e84082dbafce8f P 768d8de4b2ba40f180b18194c28be589 [term 1 FOLLOWER]: Refusing update from remote peer 1288ca0989104bbea36e22f38c374218: Log matching property violated. Preceding OpId in replica: term: 1 index: 251. Preceding OpId from leader: term: 1 index: 252. (index mismatch) I0703 01:43:07.049482 711 consensus_queue.cc:962] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 252, Last known committed idx: 251, Time since last communication: 0.000s I0703 01:43:07.054203 711 raft_consensus.cc:2684] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218 [term 1 LEADER]: Committing config change with OpId 1.252: config changed from index 131 to 252, VOTER de93b58c2d67488dae12285e1806ebac (127.5.240.195) evicted. New config: { opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } } I0703 01:43:07.055773 32766 raft_consensus.cc:2684] T a200f8a763f14d54a1e84082dbafce8f P 768d8de4b2ba40f180b18194c28be589 [term 1 FOLLOWER]: Committing config change with OpId 1.252: config changed from index 131 to 252, VOTER de93b58c2d67488dae12285e1806ebac (127.5.240.195) evicted. New config: { opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } } I0703 01:43:07.059770 31149 catalog_manager.cc:3798] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218 reported cstate change: config changed from index 131 to 252, VOTER de93b58c2d67488dae12285e1806ebac (127.5.240.195) evicted. New cstate: current_term: 1 leader_uuid: "1288ca0989104bbea36e22f38c374218" committed_config { opid_index: 252 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } } I0703 01:43:07.121060 954 raft_consensus.cc:457] T bbe64aa4b0e04f468b035048629b969b P de93b58c2d67488dae12285e1806ebac [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1288ca0989104bbea36e22f38c374218) I0703 01:43:07.121146 954 raft_consensus.cc:2730] T bbe64aa4b0e04f468b035048629b969b P de93b58c2d67488dae12285e1806ebac: Snoozing failure detection for 1.761s (starting election) I0703 01:43:07.121251 954 raft_consensus.cc:479] T bbe64aa4b0e04f468b035048629b969b P de93b58c2d67488dae12285e1806ebac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } I0703 01:43:07.121809 954 leader_election.cc:243] T bbe64aa4b0e04f468b035048629b969b P de93b58c2d67488dae12285e1806ebac [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521) I0703 01:43:07.122051 954 leader_election.cc:243] T bbe64aa4b0e04f468b035048629b969b P de93b58c2d67488dae12285e1806ebac [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 56eaab17f812433a8f7880f0744e1744 (127.5.240.199:57714) I0703 01:43:07.124316 31270 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "bbe64aa4b0e04f468b035048629b969b" candidate_uuid: "de93b58c2d67488dae12285e1806ebac" candidate_term: 2 candidate_status { last_received { term: 1 index: 367 } } ignore_live_leader: false dest_uuid: "1288ca0989104bbea36e22f38c374218" is_pre_election: true I0703 01:43:07.124667 31270 raft_consensus.cc:2190] T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate de93b58c2d67488dae12285e1806ebac for term 2 because replica has last-logged OpId of term: 1 index: 368, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 367. I0703 01:43:07.125284 31449 leader_election.cc:393] T bbe64aa4b0e04f468b035048629b969b P de93b58c2d67488dae12285e1806ebac [CANDIDATE]: Term 2 pre-election: Vote denied by peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521). Message: Invalid argument: T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate de93b58c2d67488dae12285e1806ebac for term 2 because replica has last-logged OpId of term: 1 index: 368, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 367. I0703 01:43:07.172523 1332 raft_consensus.cc:457] T bbe64aa4b0e04f468b035048629b969b P 56eaab17f812433a8f7880f0744e1744 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1288ca0989104bbea36e22f38c374218) I0703 01:43:07.175309 31149 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet a200f8a763f14d54a1e84082dbafce8f on de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321) (TS de93b58c2d67488dae12285e1806ebac not found in new config with opid_index 252) I0703 01:43:07.177206 31497 tablet_service.cc:800] Processing DeleteTablet for tablet a200f8a763f14d54a1e84082dbafce8f with delete_type TABLET_DATA_TOMBSTONED (TS de93b58c2d67488dae12285e1806ebac not found in new config with opid_index 252) from {username='jenkins'} at 127.5.240.195:46850 I0703 01:43:07.180058 1332 raft_consensus.cc:2730] T bbe64aa4b0e04f468b035048629b969b P 56eaab17f812433a8f7880f0744e1744: Snoozing failure detection for 1.941s (starting election) I0703 01:43:07.180104 1340 tablet_replica.cc:261] T a200f8a763f14d54a1e84082dbafce8f P de93b58c2d67488dae12285e1806ebac: stopping tablet replica I0703 01:43:07.180295 1340 maintenance_manager.cc:237] P de93b58c2d67488dae12285e1806ebac: Unregistered op CompactRowSetsOp(a200f8a763f14d54a1e84082dbafce8f) I0703 01:43:07.180178 1332 raft_consensus.cc:479] T bbe64aa4b0e04f468b035048629b969b P 56eaab17f812433a8f7880f0744e1744 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } I0703 01:43:07.180400 1340 maintenance_manager.cc:237] P de93b58c2d67488dae12285e1806ebac: Unregistered op MinorDeltaCompactionOp(a200f8a763f14d54a1e84082dbafce8f) I0703 01:43:07.180528 1340 maintenance_manager.cc:237] P de93b58c2d67488dae12285e1806ebac: Unregistered op MajorDeltaCompactionOp(a200f8a763f14d54a1e84082dbafce8f) I0703 01:43:07.180660 1340 maintenance_manager.cc:237] P de93b58c2d67488dae12285e1806ebac: Unregistered op UndoDeltaBlockGCOp(a200f8a763f14d54a1e84082dbafce8f) I0703 01:43:07.180815 1340 maintenance_manager.cc:237] P de93b58c2d67488dae12285e1806ebac: Unregistered op FlushMRSOp(a200f8a763f14d54a1e84082dbafce8f) I0703 01:43:07.180961 1340 maintenance_manager.cc:237] P de93b58c2d67488dae12285e1806ebac: Unregistered op FlushDeltaMemStoresOp(a200f8a763f14d54a1e84082dbafce8f) I0703 01:43:07.181071 1340 maintenance_manager.cc:237] P de93b58c2d67488dae12285e1806ebac: Unregistered op LogGCOp(a200f8a763f14d54a1e84082dbafce8f) I0703 01:43:07.181074 1332 leader_election.cc:243] T bbe64aa4b0e04f468b035048629b969b P 56eaab17f812433a8f7880f0744e1744 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521) I0703 01:43:07.181207 1340 raft_consensus.cc:2040] T a200f8a763f14d54a1e84082dbafce8f P de93b58c2d67488dae12285e1806ebac [term 1 FOLLOWER]: Raft consensus shutting down. I0703 01:43:07.175479 432 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "bbe64aa4b0e04f468b035048629b969b" candidate_uuid: "de93b58c2d67488dae12285e1806ebac" candidate_term: 2 candidate_status { last_received { term: 1 index: 367 } } ignore_live_leader: false dest_uuid: "56eaab17f812433a8f7880f0744e1744" is_pre_election: true I0703 01:43:07.181509 1340 raft_consensus.cc:2067] T a200f8a763f14d54a1e84082dbafce8f P de93b58c2d67488dae12285e1806ebac [term 1 FOLLOWER]: Raft consensus is shut down! I0703 01:43:07.181648 432 raft_consensus.cc:2190] T bbe64aa4b0e04f468b035048629b969b P 56eaab17f812433a8f7880f0744e1744 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate de93b58c2d67488dae12285e1806ebac for term 2 because replica has last-logged OpId of term: 1 index: 368, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 367. I0703 01:43:07.181977 31269 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "bbe64aa4b0e04f468b035048629b969b" candidate_uuid: "56eaab17f812433a8f7880f0744e1744" candidate_term: 2 candidate_status { last_received { term: 1 index: 368 } } ignore_live_leader: false dest_uuid: "1288ca0989104bbea36e22f38c374218" is_pre_election: true I0703 01:43:07.182147 31269 raft_consensus.cc:2730] T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218: Snoozing failure detection for 1.903s (vote granted) I0703 01:43:07.182222 31269 raft_consensus.cc:2243] T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56eaab17f812433a8f7880f0744e1744 in term 1. I0703 01:43:07.182798 1340 ts_tablet_manager.cc:1369] T a200f8a763f14d54a1e84082dbafce8f P de93b58c2d67488dae12285e1806ebac: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:43:07.183014 359 leader_election.cc:379] T bbe64aa4b0e04f468b035048629b969b P 56eaab17f812433a8f7880f0744e1744 [CANDIDATE]: Term 2 pre-election: Vote granted by peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521) I0703 01:43:07.183086 359 leader_election.cc:270] T bbe64aa4b0e04f468b035048629b969b P 56eaab17f812433a8f7880f0744e1744 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. I0703 01:43:07.183238 31450 leader_election.cc:393] T bbe64aa4b0e04f468b035048629b969b P de93b58c2d67488dae12285e1806ebac [CANDIDATE]: Term 2 pre-election: Vote denied by peer 56eaab17f812433a8f7880f0744e1744 (127.5.240.199:57714). Message: Invalid argument: T bbe64aa4b0e04f468b035048629b969b P 56eaab17f812433a8f7880f0744e1744 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate de93b58c2d67488dae12285e1806ebac for term 2 because replica has last-logged OpId of term: 1 index: 368, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 367. I0703 01:43:07.183315 31450 leader_election.cc:270] T bbe64aa4b0e04f468b035048629b969b P de93b58c2d67488dae12285e1806ebac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0703 01:43:07.187721 1307 raft_consensus.cc:2730] T bbe64aa4b0e04f468b035048629b969b P de93b58c2d67488dae12285e1806ebac: Snoozing failure detection for 2.143s (election complete) I0703 01:43:07.188109 1307 raft_consensus.cc:2485] T bbe64aa4b0e04f468b035048629b969b P de93b58c2d67488dae12285e1806ebac [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0703 01:43:07.195905 1332 leader_election.cc:243] T bbe64aa4b0e04f468b035048629b969b P 56eaab17f812433a8f7880f0744e1744 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321) I0703 01:43:07.201341 1342 raft_consensus.cc:2730] T bbe64aa4b0e04f468b035048629b969b P 56eaab17f812433a8f7880f0744e1744: Snoozing failure detection for 1.686s (election complete) I0703 01:43:07.201830 1342 raft_consensus.cc:2537] T bbe64aa4b0e04f468b035048629b969b P 56eaab17f812433a8f7880f0744e1744 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0703 01:43:07.202170 1342 raft_consensus.cc:457] T bbe64aa4b0e04f468b035048629b969b P 56eaab17f812433a8f7880f0744e1744 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1288ca0989104bbea36e22f38c374218) I0703 01:43:07.202388 1342 raft_consensus.cc:2730] T bbe64aa4b0e04f468b035048629b969b P 56eaab17f812433a8f7880f0744e1744: Snoozing failure detection for 2.195s (starting election) I0703 01:43:07.202615 1342 raft_consensus.cc:2782] T bbe64aa4b0e04f468b035048629b969b P 56eaab17f812433a8f7880f0744e1744 [term 1 FOLLOWER]: Advancing to term 2 I0703 01:43:07.208348 1340 ts_tablet_manager.cc:1382] T a200f8a763f14d54a1e84082dbafce8f P de93b58c2d67488dae12285e1806ebac: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.251 I0703 01:43:07.212044 1340 log.cc:986] T a200f8a763f14d54a1e84082dbafce8f P de93b58c2d67488dae12285e1806ebac: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal/wals/a200f8a763f14d54a1e84082dbafce8f I0703 01:43:07.213341 31136 catalog_manager.cc:3103] TS de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321): tablet a200f8a763f14d54a1e84082dbafce8f (table rebalance_test_table_2 [id=b1de6184513b4ad5b539fd284d2c95d9]) successfully deleted I0703 01:43:07.228209 1342 raft_consensus.cc:479] T bbe64aa4b0e04f468b035048629b969b P 56eaab17f812433a8f7880f0744e1744 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } I0703 01:43:07.228981 1342 leader_election.cc:243] T bbe64aa4b0e04f468b035048629b969b P 56eaab17f812433a8f7880f0744e1744 [CANDIDATE]: Term 2 election: Requesting vote from peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521) I0703 01:43:07.232483 1342 leader_election.cc:243] T bbe64aa4b0e04f468b035048629b969b P 56eaab17f812433a8f7880f0744e1744 [CANDIDATE]: Term 2 election: Requesting vote from peer de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321) I0703 01:43:07.236997 31272 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "bbe64aa4b0e04f468b035048629b969b" candidate_uuid: "56eaab17f812433a8f7880f0744e1744" candidate_term: 2 candidate_status { last_received { term: 1 index: 368 } } ignore_live_leader: false dest_uuid: "1288ca0989104bbea36e22f38c374218" I0703 01:43:07.237191 31272 raft_consensus.cc:2782] T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218 [term 1 FOLLOWER]: Advancing to term 2 I0703 01:43:07.237316 31272 raft_consensus.cc:2730] T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218: Snoozing failure detection for 1.688s (vote granted) I0703 01:43:07.241123 31272 raft_consensus.cc:2243] T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56eaab17f812433a8f7880f0744e1744 in term 2. I0703 01:43:07.242105 359 leader_election.cc:379] T bbe64aa4b0e04f468b035048629b969b P 56eaab17f812433a8f7880f0744e1744 [CANDIDATE]: Term 2 election: Vote granted by peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521) I0703 01:43:07.242168 359 leader_election.cc:270] T bbe64aa4b0e04f468b035048629b969b P 56eaab17f812433a8f7880f0744e1744 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0703 01:43:07.242388 1342 raft_consensus.cc:2730] T bbe64aa4b0e04f468b035048629b969b P 56eaab17f812433a8f7880f0744e1744: Snoozing failure detection for 2.088s (election complete) I0703 01:43:07.242511 1342 raft_consensus.cc:2537] T bbe64aa4b0e04f468b035048629b969b P 56eaab17f812433a8f7880f0744e1744 [term 2 FOLLOWER]: Leader election won for term 2 I0703 01:43:07.242637 1342 raft_consensus.cc:593] T bbe64aa4b0e04f468b035048629b969b P 56eaab17f812433a8f7880f0744e1744 [term 2 LEADER]: Becoming Leader. State: Replica: 56eaab17f812433a8f7880f0744e1744, State: Running, Role: LEADER I0703 01:43:07.242997 1342 consensus_queue.cc:204] T bbe64aa4b0e04f468b035048629b969b P 56eaab17f812433a8f7880f0744e1744 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 368, Committed index: 368, Last appended: 1.368, Last appended by leader: 368, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } I0703 01:43:07.277294 31150 catalog_manager.cc:3798] T bbe64aa4b0e04f468b035048629b969b P 56eaab17f812433a8f7880f0744e1744 reported cstate change: term changed from 1 to 2, leader changed from 1288ca0989104bbea36e22f38c374218 (127.5.240.193) to 56eaab17f812433a8f7880f0744e1744 (127.5.240.199). New cstate: current_term: 2 leader_uuid: "56eaab17f812433a8f7880f0744e1744" committed_config { opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } } I0703 01:43:07.288492 31269 raft_consensus.cc:1081] T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218 [term 2 FOLLOWER]: Refusing update from remote peer 56eaab17f812433a8f7880f0744e1744: Log matching property violated. Preceding OpId in replica: term: 1 index: 368. Preceding OpId from leader: term: 2 index: 370. (index mismatch) I0703 01:43:07.289230 1332 consensus_queue.cc:962] T bbe64aa4b0e04f468b035048629b969b P 56eaab17f812433a8f7880f0744e1744 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 369, Last known committed idx: 368, Time since last communication: 0.000s I0703 01:43:07.298215 31514 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "bbe64aa4b0e04f468b035048629b969b" candidate_uuid: "56eaab17f812433a8f7880f0744e1744" candidate_term: 2 candidate_status { last_received { term: 1 index: 368 } } ignore_live_leader: false dest_uuid: "de93b58c2d67488dae12285e1806ebac" I0703 01:43:07.298481 31514 raft_consensus.cc:2782] T bbe64aa4b0e04f468b035048629b969b P de93b58c2d67488dae12285e1806ebac [term 1 FOLLOWER]: Advancing to term 2 I0703 01:43:07.298650 31514 raft_consensus.cc:2730] T bbe64aa4b0e04f468b035048629b969b P de93b58c2d67488dae12285e1806ebac: Snoozing failure detection for 2.763s (vote granted) I0703 01:43:07.299067 31515 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "bbe64aa4b0e04f468b035048629b969b" candidate_uuid: "56eaab17f812433a8f7880f0744e1744" candidate_term: 2 candidate_status { last_received { term: 1 index: 368 } } ignore_live_leader: false dest_uuid: "de93b58c2d67488dae12285e1806ebac" is_pre_election: true I0703 01:43:07.300940 31514 raft_consensus.cc:2243] T bbe64aa4b0e04f468b035048629b969b P de93b58c2d67488dae12285e1806ebac [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56eaab17f812433a8f7880f0744e1744 in term 2. I0703 01:43:07.301120 31515 raft_consensus.cc:2218] T bbe64aa4b0e04f468b035048629b969b P de93b58c2d67488dae12285e1806ebac [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 56eaab17f812433a8f7880f0744e1744 for term 2 because replica is already servicing an update from a current leader or another vote. I0703 01:43:07.301440 31517 raft_consensus.cc:1081] T bbe64aa4b0e04f468b035048629b969b P de93b58c2d67488dae12285e1806ebac [term 2 FOLLOWER]: Refusing update from remote peer 56eaab17f812433a8f7880f0744e1744: Log matching property violated. Preceding OpId in replica: term: 1 index: 367. Preceding OpId from leader: term: 2 index: 370. (index mismatch) I0703 01:43:07.302395 361 leader_election.cc:379] T bbe64aa4b0e04f468b035048629b969b P 56eaab17f812433a8f7880f0744e1744 [CANDIDATE]: Term 2 election: Vote granted by peer de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321) I0703 01:43:07.302984 1332 consensus_queue.cc:962] T bbe64aa4b0e04f468b035048629b969b P 56eaab17f812433a8f7880f0744e1744 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 369, Last known committed idx: 366, Time since last communication: 0.000s I0703 01:43:07.303838 361 leader_election.cc:393] T bbe64aa4b0e04f468b035048629b969b P 56eaab17f812433a8f7880f0744e1744 [CANDIDATE]: Term 2 pre-election: Vote denied by peer de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321). Message: Service unavailable: T bbe64aa4b0e04f468b035048629b969b P de93b58c2d67488dae12285e1806ebac [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 56eaab17f812433a8f7880f0744e1744 for term 2 because replica is already servicing an update from a current leader or another vote. I0703 01:43:08.043434 31515 consensus_peers.cc:455] T b2c8b58f935a4d00b5cf3e2cedec4d81 P de93b58c2d67488dae12285e1806ebac -> Peer 768d8de4b2ba40f180b18194c28be589 (127.5.240.198:42654): Closing peer: 768d8de4b2ba40f180b18194c28be589 I0703 01:43:08.043509 31515 consensus_peers.cc:455] T b2c8b58f935a4d00b5cf3e2cedec4d81 P de93b58c2d67488dae12285e1806ebac -> Peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521): Closing peer: 1288ca0989104bbea36e22f38c374218 I0703 01:43:08.043783 31515 consensus_queue.cc:204] T b2c8b58f935a4d00b5cf3e2cedec4d81 P de93b58c2d67488dae12285e1806ebac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 362, Committed index: 362, Last appended: 1.362, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 363 OBSOLETE_local: false peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } I0703 01:43:08.045552 32767 raft_consensus.cc:1081] T b2c8b58f935a4d00b5cf3e2cedec4d81 P 768d8de4b2ba40f180b18194c28be589 [term 1 FOLLOWER]: Refusing update from remote peer de93b58c2d67488dae12285e1806ebac: Log matching property violated. Preceding OpId in replica: term: 1 index: 362. Preceding OpId from leader: term: 1 index: 363. (index mismatch) I0703 01:43:08.046258 1367 consensus_queue.cc:962] T b2c8b58f935a4d00b5cf3e2cedec4d81 P de93b58c2d67488dae12285e1806ebac [LEADER]: Connected to new peer: Peer: permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 363, Last known committed idx: 362, Time since last communication: 0.000s I0703 01:43:08.049599 1367 raft_consensus.cc:2684] T b2c8b58f935a4d00b5cf3e2cedec4d81 P de93b58c2d67488dae12285e1806ebac [term 1 LEADER]: Committing config change with OpId 1.363: config changed from index 217 to 363, VOTER 1288ca0989104bbea36e22f38c374218 (127.5.240.193) evicted. New config: { opid_index: 363 OBSOLETE_local: false peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } } I0703 01:43:08.051486 32767 raft_consensus.cc:2684] T b2c8b58f935a4d00b5cf3e2cedec4d81 P 768d8de4b2ba40f180b18194c28be589 [term 1 FOLLOWER]: Committing config change with OpId 1.363: config changed from index 217 to 363, VOTER 1288ca0989104bbea36e22f38c374218 (127.5.240.193) evicted. New config: { opid_index: 363 OBSOLETE_local: false peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } } I0703 01:43:08.059294 31150 catalog_manager.cc:3798] T b2c8b58f935a4d00b5cf3e2cedec4d81 P de93b58c2d67488dae12285e1806ebac reported cstate change: config changed from index 217 to 363, VOTER 1288ca0989104bbea36e22f38c374218 (127.5.240.193) evicted. New cstate: current_term: 1 leader_uuid: "de93b58c2d67488dae12285e1806ebac" committed_config { opid_index: 363 OBSOLETE_local: false peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } } I0703 01:43:08.103399 31150 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet b2c8b58f935a4d00b5cf3e2cedec4d81 on 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521) (TS 1288ca0989104bbea36e22f38c374218 not found in new config with opid_index 363) I0703 01:43:08.105163 31250 tablet_service.cc:800] Processing DeleteTablet for tablet b2c8b58f935a4d00b5cf3e2cedec4d81 with delete_type TABLET_DATA_TOMBSTONED (TS 1288ca0989104bbea36e22f38c374218 not found in new config with opid_index 363) from {username='jenkins'} at 127.5.240.193:33318 I0703 01:43:08.108366 1409 tablet_replica.cc:261] T b2c8b58f935a4d00b5cf3e2cedec4d81 P 1288ca0989104bbea36e22f38c374218: stopping tablet replica I0703 01:43:08.108717 1409 maintenance_manager.cc:237] P 1288ca0989104bbea36e22f38c374218: Unregistered op CompactRowSetsOp(b2c8b58f935a4d00b5cf3e2cedec4d81) I0703 01:43:08.108829 1409 maintenance_manager.cc:237] P 1288ca0989104bbea36e22f38c374218: Unregistered op MinorDeltaCompactionOp(b2c8b58f935a4d00b5cf3e2cedec4d81) I0703 01:43:08.109845 1409 maintenance_manager.cc:237] P 1288ca0989104bbea36e22f38c374218: Unregistered op MajorDeltaCompactionOp(b2c8b58f935a4d00b5cf3e2cedec4d81) I0703 01:43:08.109953 1409 maintenance_manager.cc:237] P 1288ca0989104bbea36e22f38c374218: Unregistered op UndoDeltaBlockGCOp(b2c8b58f935a4d00b5cf3e2cedec4d81) I0703 01:43:08.110057 1409 maintenance_manager.cc:237] P 1288ca0989104bbea36e22f38c374218: Unregistered op FlushMRSOp(b2c8b58f935a4d00b5cf3e2cedec4d81) I0703 01:43:08.110144 1409 maintenance_manager.cc:237] P 1288ca0989104bbea36e22f38c374218: Unregistered op FlushDeltaMemStoresOp(b2c8b58f935a4d00b5cf3e2cedec4d81) I0703 01:43:08.110219 1409 maintenance_manager.cc:237] P 1288ca0989104bbea36e22f38c374218: Unregistered op LogGCOp(b2c8b58f935a4d00b5cf3e2cedec4d81) I0703 01:43:08.111806 1409 raft_consensus.cc:2040] T b2c8b58f935a4d00b5cf3e2cedec4d81 P 1288ca0989104bbea36e22f38c374218 [term 1 FOLLOWER]: Raft consensus shutting down. I0703 01:43:08.112078 1409 raft_consensus.cc:2067] T b2c8b58f935a4d00b5cf3e2cedec4d81 P 1288ca0989104bbea36e22f38c374218 [term 1 FOLLOWER]: Raft consensus is shut down! I0703 01:43:08.114439 1409 ts_tablet_manager.cc:1369] T b2c8b58f935a4d00b5cf3e2cedec4d81 P 1288ca0989104bbea36e22f38c374218: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:43:08.138079 1409 ts_tablet_manager.cc:1382] T b2c8b58f935a4d00b5cf3e2cedec4d81 P 1288ca0989104bbea36e22f38c374218: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.362 I0703 01:43:08.145073 1409 log.cc:986] T b2c8b58f935a4d00b5cf3e2cedec4d81 P 1288ca0989104bbea36e22f38c374218: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal/wals/b2c8b58f935a4d00b5cf3e2cedec4d81 I0703 01:43:08.145819 31134 catalog_manager.cc:3103] TS 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521): tablet b2c8b58f935a4d00b5cf3e2cedec4d81 (table rebalance_test_table_0 [id=9d68c0e8280a4d18af7d08099d85f9a3]) successfully deleted W0703 01:43:08.753126 31205 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.240.196:56543, user_credentials={real_user=jenkins}} blocked reactor thread for 47530us I0703 01:43:08.973449 31517 consensus_peers.cc:455] T e0af316d40d143a8b387933151adf0e4 P de93b58c2d67488dae12285e1806ebac -> Peer 3c1770123dc14bba80e00db5f182b585 (127.5.240.197:44326): Closing peer: 3c1770123dc14bba80e00db5f182b585 I0703 01:43:08.973572 31517 consensus_peers.cc:455] T e0af316d40d143a8b387933151adf0e4 P de93b58c2d67488dae12285e1806ebac -> Peer 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194:39827): Closing peer: 2f3fdb7c089640c0b936a574bc64707e I0703 01:43:08.973863 31517 consensus_queue.cc:204] T e0af316d40d143a8b387933151adf0e4 P de93b58c2d67488dae12285e1806ebac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 431, Committed index: 431, Last appended: 1.431, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 432 OBSOLETE_local: false peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "3c1770123dc14bba80e00db5f182b585" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 44326 } } I0703 01:43:08.977648 32638 raft_consensus.cc:1081] T e0af316d40d143a8b387933151adf0e4 P 3c1770123dc14bba80e00db5f182b585 [term 1 FOLLOWER]: Refusing update from remote peer de93b58c2d67488dae12285e1806ebac: Log matching property violated. Preceding OpId in replica: term: 1 index: 431. Preceding OpId from leader: term: 1 index: 432. (index mismatch) I0703 01:43:08.978323 1367 consensus_queue.cc:962] T e0af316d40d143a8b387933151adf0e4 P de93b58c2d67488dae12285e1806ebac [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c1770123dc14bba80e00db5f182b585" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 44326 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 432, Last known committed idx: 431, Time since last communication: 0.000s I0703 01:43:08.981946 1367 raft_consensus.cc:2684] T e0af316d40d143a8b387933151adf0e4 P de93b58c2d67488dae12285e1806ebac [term 1 LEADER]: Committing config change with OpId 1.432: config changed from index 243 to 432, VOTER 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194) evicted. New config: { opid_index: 432 OBSOLETE_local: false peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "3c1770123dc14bba80e00db5f182b585" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 44326 } } } I0703 01:43:08.985406 32637 raft_consensus.cc:2684] T e0af316d40d143a8b387933151adf0e4 P 3c1770123dc14bba80e00db5f182b585 [term 1 FOLLOWER]: Committing config change with OpId 1.432: config changed from index 243 to 432, VOTER 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194) evicted. New config: { opid_index: 432 OBSOLETE_local: false peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "3c1770123dc14bba80e00db5f182b585" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 44326 } } } I0703 01:43:08.989595 31149 catalog_manager.cc:3798] T e0af316d40d143a8b387933151adf0e4 P de93b58c2d67488dae12285e1806ebac reported cstate change: config changed from index 243 to 432, VOTER 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194) evicted. New cstate: current_term: 1 leader_uuid: "de93b58c2d67488dae12285e1806ebac" committed_config { opid_index: 432 OBSOLETE_local: false peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "3c1770123dc14bba80e00db5f182b585" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 44326 } } } W0703 01:43:09.063676 31205 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.240.196:56543, user_credentials={real_user=jenkins}} blocked reactor thread for 44440.3us I0703 01:43:09.069926 31149 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet e0af316d40d143a8b387933151adf0e4 on 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194:39827) (TS 2f3fdb7c089640c0b936a574bc64707e not found in new config with opid_index 432) I0703 01:43:09.071990 31375 tablet_service.cc:800] Processing DeleteTablet for tablet e0af316d40d143a8b387933151adf0e4 with delete_type TABLET_DATA_TOMBSTONED (TS 2f3fdb7c089640c0b936a574bc64707e not found in new config with opid_index 432) from {username='jenkins'} at 127.5.240.194:58888 I0703 01:43:09.086154 1480 tablet_replica.cc:261] T e0af316d40d143a8b387933151adf0e4 P 2f3fdb7c089640c0b936a574bc64707e: stopping tablet replica I0703 01:43:09.091199 1480 maintenance_manager.cc:237] P 2f3fdb7c089640c0b936a574bc64707e: Unregistered op CompactRowSetsOp(e0af316d40d143a8b387933151adf0e4) I0703 01:43:09.091346 1480 maintenance_manager.cc:237] P 2f3fdb7c089640c0b936a574bc64707e: Unregistered op MinorDeltaCompactionOp(e0af316d40d143a8b387933151adf0e4) I0703 01:43:09.091475 1480 maintenance_manager.cc:237] P 2f3fdb7c089640c0b936a574bc64707e: Unregistered op MajorDeltaCompactionOp(e0af316d40d143a8b387933151adf0e4) I0703 01:43:09.091596 1480 maintenance_manager.cc:237] P 2f3fdb7c089640c0b936a574bc64707e: Unregistered op UndoDeltaBlockGCOp(e0af316d40d143a8b387933151adf0e4) I0703 01:43:09.091702 1480 maintenance_manager.cc:237] P 2f3fdb7c089640c0b936a574bc64707e: Unregistered op FlushMRSOp(e0af316d40d143a8b387933151adf0e4) I0703 01:43:09.091814 1480 maintenance_manager.cc:237] P 2f3fdb7c089640c0b936a574bc64707e: Unregistered op FlushDeltaMemStoresOp(e0af316d40d143a8b387933151adf0e4) I0703 01:43:09.091924 1480 maintenance_manager.cc:237] P 2f3fdb7c089640c0b936a574bc64707e: Unregistered op LogGCOp(e0af316d40d143a8b387933151adf0e4) I0703 01:43:09.092043 1480 raft_consensus.cc:2040] T e0af316d40d143a8b387933151adf0e4 P 2f3fdb7c089640c0b936a574bc64707e [term 1 FOLLOWER]: Raft consensus shutting down. I0703 01:43:09.092285 1480 raft_consensus.cc:2067] T e0af316d40d143a8b387933151adf0e4 P 2f3fdb7c089640c0b936a574bc64707e [term 1 FOLLOWER]: Raft consensus is shut down! I0703 01:43:09.096745 1480 ts_tablet_manager.cc:1369] T e0af316d40d143a8b387933151adf0e4 P 2f3fdb7c089640c0b936a574bc64707e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:43:09.125748 1480 ts_tablet_manager.cc:1382] T e0af316d40d143a8b387933151adf0e4 P 2f3fdb7c089640c0b936a574bc64707e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.431 I0703 01:43:09.129665 1480 log.cc:986] T e0af316d40d143a8b387933151adf0e4 P 2f3fdb7c089640c0b936a574bc64707e: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal/wals/e0af316d40d143a8b387933151adf0e4 I0703 01:43:09.132778 31133 catalog_manager.cc:3103] TS 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194:39827): tablet e0af316d40d143a8b387933151adf0e4 (table rebalance_test_table_3 [id=b06785934be6432b91c9f50ee8c104d2]) successfully deleted I0703 01:43:09.602887 31269 tablet_service.cc:1140] Received LeaderStepDown RPC: tablet_id: "ffe66a5cfb564964951c619e51e7c47c" dest_uuid: "1288ca0989104bbea36e22f38c374218" from {username='jenkins'} at 127.5.240.193:33557 I0703 01:43:09.603036 31269 raft_consensus.cc:552] T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218 [term 1 LEADER]: Received request to step down I0703 01:43:09.603123 31269 raft_consensus.cc:628] T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1288ca0989104bbea36e22f38c374218, State: Running, Role: LEADER I0703 01:43:09.603426 31269 consensus_queue.cc:227] T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 459, Committed index: 459, Last appended: 1.459, Last appended by leader: 459, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 189 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } I0703 01:43:09.635856 31269 consensus_peers.cc:455] T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218 -> Peer d9dc68ace903455d809cb1f715fe36be (127.5.240.196:56543): Closing peer: d9dc68ace903455d809cb1f715fe36be I0703 01:43:09.635913 31269 consensus_peers.cc:455] T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218 -> Peer de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321): Closing peer: de93b58c2d67488dae12285e1806ebac I0703 01:43:09.635951 31269 raft_consensus.cc:2730] T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218: Snoozing failure detection for 3.000s (explicit stepdown request) W0703 01:43:09.635951 1454 consensus_queue.cc:1041] T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "de93b58c2d67488dae12285e1806ebac" responder_term: 1 status { last_received { term: 1 index: 458 } last_committed_idx: 457 last_received_current_leader { term: 1 index: 458 } } I0703 01:43:10.828538 31393 consensus_peers.cc:455] T 3d7a6e87780a49949cf7e024817cf3a3 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 56eaab17f812433a8f7880f0744e1744 (127.5.240.199:57714): Closing peer: 56eaab17f812433a8f7880f0744e1744 I0703 01:43:10.828651 31393 consensus_peers.cc:455] T 3d7a6e87780a49949cf7e024817cf3a3 P 2f3fdb7c089640c0b936a574bc64707e -> Peer de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321): Closing peer: de93b58c2d67488dae12285e1806ebac I0703 01:43:10.828928 31393 consensus_queue.cc:204] T 3d7a6e87780a49949cf7e024817cf3a3 P 2f3fdb7c089640c0b936a574bc64707e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 437, Committed index: 437, Last appended: 1.437, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 438 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } I0703 01:43:10.830595 431 raft_consensus.cc:1081] T 3d7a6e87780a49949cf7e024817cf3a3 P 56eaab17f812433a8f7880f0744e1744 [term 1 FOLLOWER]: Refusing update from remote peer 2f3fdb7c089640c0b936a574bc64707e: Log matching property violated. Preceding OpId in replica: term: 1 index: 437. Preceding OpId from leader: term: 1 index: 438. (index mismatch) I0703 01:43:10.834182 1150 consensus_queue.cc:962] T 3d7a6e87780a49949cf7e024817cf3a3 P 2f3fdb7c089640c0b936a574bc64707e [LEADER]: Connected to new peer: Peer: permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 438, Last known committed idx: 437, Time since last communication: 0.000s I0703 01:43:10.842042 763 raft_consensus.cc:2684] T 3d7a6e87780a49949cf7e024817cf3a3 P 2f3fdb7c089640c0b936a574bc64707e [term 1 LEADER]: Committing config change with OpId 1.438: config changed from index 214 to 438, VOTER de93b58c2d67488dae12285e1806ebac (127.5.240.195) evicted. New config: { opid_index: 438 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } } I0703 01:43:10.852211 431 raft_consensus.cc:2684] T 3d7a6e87780a49949cf7e024817cf3a3 P 56eaab17f812433a8f7880f0744e1744 [term 1 FOLLOWER]: Committing config change with OpId 1.438: config changed from index 214 to 438, VOTER de93b58c2d67488dae12285e1806ebac (127.5.240.195) evicted. New config: { opid_index: 438 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } } I0703 01:43:10.857275 31152 catalog_manager.cc:3798] T 3d7a6e87780a49949cf7e024817cf3a3 P 56eaab17f812433a8f7880f0744e1744 reported cstate change: config changed from index 214 to 438, VOTER de93b58c2d67488dae12285e1806ebac (127.5.240.195) evicted. New cstate: current_term: 1 leader_uuid: "2f3fdb7c089640c0b936a574bc64707e" committed_config { opid_index: 438 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } } I0703 01:43:10.929064 31152 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 3d7a6e87780a49949cf7e024817cf3a3 on de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321) (TS de93b58c2d67488dae12285e1806ebac not found in new config with opid_index 438) I0703 01:43:10.930192 31497 tablet_service.cc:800] Processing DeleteTablet for tablet 3d7a6e87780a49949cf7e024817cf3a3 with delete_type TABLET_DATA_TOMBSTONED (TS de93b58c2d67488dae12285e1806ebac not found in new config with opid_index 438) from {username='jenkins'} at 127.5.240.195:46850 I0703 01:43:10.941642 1620 tablet_replica.cc:261] T 3d7a6e87780a49949cf7e024817cf3a3 P de93b58c2d67488dae12285e1806ebac: stopping tablet replica I0703 01:43:10.941790 1620 maintenance_manager.cc:237] P de93b58c2d67488dae12285e1806ebac: Unregistered op CompactRowSetsOp(3d7a6e87780a49949cf7e024817cf3a3) I0703 01:43:10.941906 1620 maintenance_manager.cc:237] P de93b58c2d67488dae12285e1806ebac: Unregistered op MinorDeltaCompactionOp(3d7a6e87780a49949cf7e024817cf3a3) I0703 01:43:10.942374 1620 maintenance_manager.cc:237] P de93b58c2d67488dae12285e1806ebac: Unregistered op MajorDeltaCompactionOp(3d7a6e87780a49949cf7e024817cf3a3) I0703 01:43:10.942479 1620 maintenance_manager.cc:237] P de93b58c2d67488dae12285e1806ebac: Unregistered op UndoDeltaBlockGCOp(3d7a6e87780a49949cf7e024817cf3a3) I0703 01:43:10.942580 1620 maintenance_manager.cc:237] P de93b58c2d67488dae12285e1806ebac: Unregistered op FlushMRSOp(3d7a6e87780a49949cf7e024817cf3a3) I0703 01:43:10.942689 1620 maintenance_manager.cc:237] P de93b58c2d67488dae12285e1806ebac: Unregistered op FlushDeltaMemStoresOp(3d7a6e87780a49949cf7e024817cf3a3) I0703 01:43:10.942770 1620 maintenance_manager.cc:237] P de93b58c2d67488dae12285e1806ebac: Unregistered op LogGCOp(3d7a6e87780a49949cf7e024817cf3a3) I0703 01:43:10.942859 1620 raft_consensus.cc:2040] T 3d7a6e87780a49949cf7e024817cf3a3 P de93b58c2d67488dae12285e1806ebac [term 1 FOLLOWER]: Raft consensus shutting down. I0703 01:43:10.943012 1620 raft_consensus.cc:2067] T 3d7a6e87780a49949cf7e024817cf3a3 P de93b58c2d67488dae12285e1806ebac [term 1 FOLLOWER]: Raft consensus is shut down! I0703 01:43:10.945273 1620 ts_tablet_manager.cc:1369] T 3d7a6e87780a49949cf7e024817cf3a3 P de93b58c2d67488dae12285e1806ebac: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:43:10.951186 1620 ts_tablet_manager.cc:1382] T 3d7a6e87780a49949cf7e024817cf3a3 P de93b58c2d67488dae12285e1806ebac: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.437 I0703 01:43:10.951766 1620 log.cc:986] T 3d7a6e87780a49949cf7e024817cf3a3 P de93b58c2d67488dae12285e1806ebac: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal/wals/3d7a6e87780a49949cf7e024817cf3a3 I0703 01:43:10.952692 31136 catalog_manager.cc:3103] TS de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321): tablet 3d7a6e87780a49949cf7e024817cf3a3 (table rebalance_test_table_3 [id=b06785934be6432b91c9f50ee8c104d2]) successfully deleted I0703 01:43:11.099337 1545 raft_consensus.cc:457] T ffe66a5cfb564964951c619e51e7c47c P de93b58c2d67488dae12285e1806ebac [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1288ca0989104bbea36e22f38c374218) I0703 01:43:11.099421 1545 raft_consensus.cc:2730] T ffe66a5cfb564964951c619e51e7c47c P de93b58c2d67488dae12285e1806ebac: Snoozing failure detection for 1.587s (starting election) I0703 01:43:11.099505 1545 raft_consensus.cc:479] T ffe66a5cfb564964951c619e51e7c47c P de93b58c2d67488dae12285e1806ebac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 189 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } I0703 01:43:11.099998 1545 leader_election.cc:243] T ffe66a5cfb564964951c619e51e7c47c P de93b58c2d67488dae12285e1806ebac [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521) I0703 01:43:11.100141 1545 leader_election.cc:243] T ffe66a5cfb564964951c619e51e7c47c P de93b58c2d67488dae12285e1806ebac [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer d9dc68ace903455d809cb1f715fe36be (127.5.240.196:56543) I0703 01:43:11.103651 31269 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "ffe66a5cfb564964951c619e51e7c47c" candidate_uuid: "de93b58c2d67488dae12285e1806ebac" candidate_term: 2 candidate_status { last_received { term: 1 index: 458 } } ignore_live_leader: false dest_uuid: "1288ca0989104bbea36e22f38c374218" is_pre_election: true I0703 01:43:11.103852 31269 raft_consensus.cc:2190] T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate de93b58c2d67488dae12285e1806ebac for term 2 because replica has last-logged OpId of term: 1 index: 459, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 458. I0703 01:43:11.107791 31449 leader_election.cc:393] T ffe66a5cfb564964951c619e51e7c47c P de93b58c2d67488dae12285e1806ebac [CANDIDATE]: Term 2 pre-election: Vote denied by peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521). Message: Invalid argument: T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate de93b58c2d67488dae12285e1806ebac for term 2 because replica has last-logged OpId of term: 1 index: 459, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 458. I0703 01:43:11.107791 32502 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "ffe66a5cfb564964951c619e51e7c47c" candidate_uuid: "de93b58c2d67488dae12285e1806ebac" candidate_term: 2 candidate_status { last_received { term: 1 index: 458 } } ignore_live_leader: false dest_uuid: "d9dc68ace903455d809cb1f715fe36be" is_pre_election: true I0703 01:43:11.108093 32502 raft_consensus.cc:2190] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate de93b58c2d67488dae12285e1806ebac for term 2 because replica has last-logged OpId of term: 1 index: 459, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 458. I0703 01:43:11.110471 31450 leader_election.cc:393] T ffe66a5cfb564964951c619e51e7c47c P de93b58c2d67488dae12285e1806ebac [CANDIDATE]: Term 2 pre-election: Vote denied by peer d9dc68ace903455d809cb1f715fe36be (127.5.240.196:56543). Message: Invalid argument: T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate de93b58c2d67488dae12285e1806ebac for term 2 because replica has last-logged OpId of term: 1 index: 459, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 458. I0703 01:43:11.113245 31450 leader_election.cc:270] T ffe66a5cfb564964951c619e51e7c47c P de93b58c2d67488dae12285e1806ebac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0703 01:43:11.113788 1307 raft_consensus.cc:2730] T ffe66a5cfb564964951c619e51e7c47c P de93b58c2d67488dae12285e1806ebac: Snoozing failure detection for 2.188s (election complete) I0703 01:43:11.113921 1307 raft_consensus.cc:2485] T ffe66a5cfb564964951c619e51e7c47c P de93b58c2d67488dae12285e1806ebac [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0703 01:43:11.124472 1638 raft_consensus.cc:457] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1288ca0989104bbea36e22f38c374218) I0703 01:43:11.124640 1638 raft_consensus.cc:2730] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be: Snoozing failure detection for 2.017s (starting election) I0703 01:43:11.136837 1638 raft_consensus.cc:479] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 189 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } I0703 01:43:11.139608 1638 leader_election.cc:243] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521) I0703 01:43:11.140748 31269 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "ffe66a5cfb564964951c619e51e7c47c" candidate_uuid: "d9dc68ace903455d809cb1f715fe36be" candidate_term: 2 candidate_status { last_received { term: 1 index: 459 } } ignore_live_leader: false dest_uuid: "1288ca0989104bbea36e22f38c374218" is_pre_election: true I0703 01:43:11.140877 31269 raft_consensus.cc:2730] T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218: Snoozing failure detection for 1.591s (vote granted) I0703 01:43:11.140918 31269 raft_consensus.cc:2243] T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9dc68ace903455d809cb1f715fe36be in term 1. I0703 01:43:11.141638 32433 leader_election.cc:379] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be [CANDIDATE]: Term 2 pre-election: Vote granted by peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521) I0703 01:43:11.141805 32433 leader_election.cc:270] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. I0703 01:43:11.159356 1638 leader_election.cc:243] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321) I0703 01:43:11.159636 1638 raft_consensus.cc:2730] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be: Snoozing failure detection for 1.880s (election complete) I0703 01:43:11.160779 1638 raft_consensus.cc:2537] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be [term 1 FOLLOWER]: Leader pre-election won for term 2 I0703 01:43:11.161583 1638 raft_consensus.cc:457] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1288ca0989104bbea36e22f38c374218) I0703 01:43:11.163084 1638 raft_consensus.cc:2730] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be: Snoozing failure detection for 2.206s (starting election) I0703 01:43:11.164160 1638 raft_consensus.cc:2782] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be [term 1 FOLLOWER]: Advancing to term 2 I0703 01:43:11.167361 1638 raft_consensus.cc:479] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be [term 2 FOLLOWER]: Starting leader election with config: opid_index: 189 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } I0703 01:43:11.167866 1638 leader_election.cc:243] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be [CANDIDATE]: Term 2 election: Requesting vote from peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521) I0703 01:43:11.170660 31514 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "ffe66a5cfb564964951c619e51e7c47c" candidate_uuid: "d9dc68ace903455d809cb1f715fe36be" candidate_term: 2 candidate_status { last_received { term: 1 index: 459 } } ignore_live_leader: false dest_uuid: "de93b58c2d67488dae12285e1806ebac" is_pre_election: true I0703 01:43:11.170816 31514 raft_consensus.cc:2730] T ffe66a5cfb564964951c619e51e7c47c P de93b58c2d67488dae12285e1806ebac: Snoozing failure detection for 2.444s (vote granted) I0703 01:43:11.170948 31514 raft_consensus.cc:2243] T ffe66a5cfb564964951c619e51e7c47c P de93b58c2d67488dae12285e1806ebac [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9dc68ace903455d809cb1f715fe36be in term 1. I0703 01:43:11.172237 32435 leader_election.cc:379] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be [CANDIDATE]: Term 2 pre-election: Vote granted by peer de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321) I0703 01:43:11.173920 1638 leader_election.cc:243] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be [CANDIDATE]: Term 2 election: Requesting vote from peer de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321) I0703 01:43:11.174232 31269 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "ffe66a5cfb564964951c619e51e7c47c" candidate_uuid: "d9dc68ace903455d809cb1f715fe36be" candidate_term: 2 candidate_status { last_received { term: 1 index: 459 } } ignore_live_leader: false dest_uuid: "1288ca0989104bbea36e22f38c374218" I0703 01:43:11.174381 31269 raft_consensus.cc:2782] T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218 [term 1 FOLLOWER]: Advancing to term 2 I0703 01:43:11.174438 31269 raft_consensus.cc:2730] T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218: Snoozing failure detection for 1.929s (vote granted) I0703 01:43:11.177104 31514 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "ffe66a5cfb564964951c619e51e7c47c" candidate_uuid: "d9dc68ace903455d809cb1f715fe36be" candidate_term: 2 candidate_status { last_received { term: 1 index: 459 } } ignore_live_leader: false dest_uuid: "de93b58c2d67488dae12285e1806ebac" I0703 01:43:11.177273 31514 raft_consensus.cc:2782] T ffe66a5cfb564964951c619e51e7c47c P de93b58c2d67488dae12285e1806ebac [term 1 FOLLOWER]: Advancing to term 2 I0703 01:43:11.177376 31514 raft_consensus.cc:2730] T ffe66a5cfb564964951c619e51e7c47c P de93b58c2d67488dae12285e1806ebac: Snoozing failure detection for 2.461s (vote granted) I0703 01:43:11.177742 31269 raft_consensus.cc:2243] T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9dc68ace903455d809cb1f715fe36be in term 2. I0703 01:43:11.178170 32433 leader_election.cc:379] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be [CANDIDATE]: Term 2 election: Vote granted by peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521) I0703 01:43:11.178237 32433 leader_election.cc:270] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0703 01:43:11.178503 1638 raft_consensus.cc:2730] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be: Snoozing failure detection for 2.123s (election complete) I0703 01:43:11.178606 1638 raft_consensus.cc:2537] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be [term 2 FOLLOWER]: Leader election won for term 2 I0703 01:43:11.178717 1638 raft_consensus.cc:593] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be [term 2 LEADER]: Becoming Leader. State: Replica: d9dc68ace903455d809cb1f715fe36be, State: Running, Role: LEADER I0703 01:43:11.179075 1638 consensus_queue.cc:204] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 459, Committed index: 459, Last appended: 1.459, Last appended by leader: 459, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 189 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } I0703 01:43:11.179823 31514 raft_consensus.cc:2243] T ffe66a5cfb564964951c619e51e7c47c P de93b58c2d67488dae12285e1806ebac [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9dc68ace903455d809cb1f715fe36be in term 2. I0703 01:43:11.187675 31149 catalog_manager.cc:3798] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be reported cstate change: term changed from 1 to 2, leader changed from 1288ca0989104bbea36e22f38c374218 (127.5.240.193) to d9dc68ace903455d809cb1f715fe36be (127.5.240.196). New cstate: current_term: 2 leader_uuid: "d9dc68ace903455d809cb1f715fe36be" committed_config { opid_index: 189 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } } I0703 01:43:11.190310 32435 leader_election.cc:379] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be [CANDIDATE]: Term 2 election: Vote granted by peer de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321) I0703 01:43:11.201656 31269 raft_consensus.cc:1081] T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218 [term 2 FOLLOWER]: Refusing update from remote peer d9dc68ace903455d809cb1f715fe36be: Log matching property violated. Preceding OpId in replica: term: 1 index: 459. Preceding OpId from leader: term: 2 index: 461. (index mismatch) I0703 01:43:11.202574 1639 consensus_queue.cc:962] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be [LEADER]: Connected to new peer: Peer: permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 460, Last known committed idx: 459, Time since last communication: 0.000s I0703 01:43:11.203735 31514 raft_consensus.cc:1081] T ffe66a5cfb564964951c619e51e7c47c P de93b58c2d67488dae12285e1806ebac [term 2 FOLLOWER]: Refusing update from remote peer d9dc68ace903455d809cb1f715fe36be: Log matching property violated. Preceding OpId in replica: term: 1 index: 458. Preceding OpId from leader: term: 2 index: 461. (index mismatch) I0703 01:43:11.214625 1639 consensus_queue.cc:962] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be [LEADER]: Connected to new peer: Peer: permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 460, Last known committed idx: 457, Time since last communication: 0.000s I0703 01:43:11.473459 31393 tablet_service.cc:1140] Received LeaderStepDown RPC: tablet_id: "454bab377db147a1a2f8241ecc2241f3" dest_uuid: "2f3fdb7c089640c0b936a574bc64707e" from {username='jenkins'} at 127.5.240.194:59173 I0703 01:43:11.473640 31393 raft_consensus.cc:552] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e [term 1 LEADER]: Received request to step down I0703 01:43:11.473676 31393 raft_consensus.cc:628] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2f3fdb7c089640c0b936a574bc64707e, State: Running, Role: LEADER I0703 01:43:11.473860 31393 consensus_queue.cc:227] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } I0703 01:43:11.474093 31393 consensus_peers.cc:455] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e -> Peer d9dc68ace903455d809cb1f715fe36be (127.5.240.196:56543): Closing peer: d9dc68ace903455d809cb1f715fe36be I0703 01:43:11.474133 31393 consensus_peers.cc:455] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 56eaab17f812433a8f7880f0744e1744 (127.5.240.199:57714): Closing peer: 56eaab17f812433a8f7880f0744e1744 I0703 01:43:11.474273 31393 raft_consensus.cc:2730] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e: Snoozing failure detection for 3.000s (explicit stepdown request) I0703 01:43:12.669616 1682 raft_consensus.cc:457] T 454bab377db147a1a2f8241ecc2241f3 P d9dc68ace903455d809cb1f715fe36be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2f3fdb7c089640c0b936a574bc64707e) I0703 01:43:12.669710 1682 raft_consensus.cc:2730] T 454bab377db147a1a2f8241ecc2241f3 P d9dc68ace903455d809cb1f715fe36be: Snoozing failure detection for 1.718s (starting election) I0703 01:43:12.669818 1682 raft_consensus.cc:479] T 454bab377db147a1a2f8241ecc2241f3 P d9dc68ace903455d809cb1f715fe36be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } I0703 01:43:12.670398 1682 leader_election.cc:243] T 454bab377db147a1a2f8241ecc2241f3 P d9dc68ace903455d809cb1f715fe36be [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194:39827) I0703 01:43:12.670603 1682 leader_election.cc:243] T 454bab377db147a1a2f8241ecc2241f3 P d9dc68ace903455d809cb1f715fe36be [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 56eaab17f812433a8f7880f0744e1744 (127.5.240.199:57714) I0703 01:43:12.671742 31393 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "454bab377db147a1a2f8241ecc2241f3" candidate_uuid: "d9dc68ace903455d809cb1f715fe36be" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2f3fdb7c089640c0b936a574bc64707e" is_pre_election: true I0703 01:43:12.671893 31393 raft_consensus.cc:2730] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e: Snoozing failure detection for 1.565s (vote granted) I0703 01:43:12.671953 31393 raft_consensus.cc:2243] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9dc68ace903455d809cb1f715fe36be in term 1. I0703 01:43:12.673843 32432 leader_election.cc:379] T 454bab377db147a1a2f8241ecc2241f3 P d9dc68ace903455d809cb1f715fe36be [CANDIDATE]: Term 2 pre-election: Vote granted by peer 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194:39827) I0703 01:43:12.673996 32432 leader_election.cc:270] T 454bab377db147a1a2f8241ecc2241f3 P d9dc68ace903455d809cb1f715fe36be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. I0703 01:43:12.674249 1682 raft_consensus.cc:2730] T 454bab377db147a1a2f8241ecc2241f3 P d9dc68ace903455d809cb1f715fe36be: Snoozing failure detection for 2.235s (election complete) I0703 01:43:12.674324 1682 raft_consensus.cc:2537] T 454bab377db147a1a2f8241ecc2241f3 P d9dc68ace903455d809cb1f715fe36be [term 1 FOLLOWER]: Leader pre-election won for term 2 I0703 01:43:12.674374 1682 raft_consensus.cc:457] T 454bab377db147a1a2f8241ecc2241f3 P d9dc68ace903455d809cb1f715fe36be [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2f3fdb7c089640c0b936a574bc64707e) I0703 01:43:12.674458 1682 raft_consensus.cc:2730] T 454bab377db147a1a2f8241ecc2241f3 P d9dc68ace903455d809cb1f715fe36be: Snoozing failure detection for 1.697s (starting election) I0703 01:43:12.674556 1682 raft_consensus.cc:2782] T 454bab377db147a1a2f8241ecc2241f3 P d9dc68ace903455d809cb1f715fe36be [term 1 FOLLOWER]: Advancing to term 2 I0703 01:43:12.697986 1682 raft_consensus.cc:479] T 454bab377db147a1a2f8241ecc2241f3 P d9dc68ace903455d809cb1f715fe36be [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } I0703 01:43:12.698611 1682 leader_election.cc:243] T 454bab377db147a1a2f8241ecc2241f3 P d9dc68ace903455d809cb1f715fe36be [CANDIDATE]: Term 2 election: Requesting vote from peer 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194:39827) I0703 01:43:12.715536 31393 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "454bab377db147a1a2f8241ecc2241f3" candidate_uuid: "d9dc68ace903455d809cb1f715fe36be" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2f3fdb7c089640c0b936a574bc64707e" I0703 01:43:12.715703 31393 raft_consensus.cc:2782] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e [term 1 FOLLOWER]: Advancing to term 2 I0703 01:43:12.715795 31393 raft_consensus.cc:2730] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e: Snoozing failure detection for 1.670s (vote granted) I0703 01:43:12.716372 1682 leader_election.cc:243] T 454bab377db147a1a2f8241ecc2241f3 P d9dc68ace903455d809cb1f715fe36be [CANDIDATE]: Term 2 election: Requesting vote from peer 56eaab17f812433a8f7880f0744e1744 (127.5.240.199:57714) I0703 01:43:12.717728 31393 raft_consensus.cc:2243] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9dc68ace903455d809cb1f715fe36be in term 2. I0703 01:43:12.725231 32432 leader_election.cc:379] T 454bab377db147a1a2f8241ecc2241f3 P d9dc68ace903455d809cb1f715fe36be [CANDIDATE]: Term 2 election: Vote granted by peer 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194:39827) I0703 01:43:12.725325 32432 leader_election.cc:270] T 454bab377db147a1a2f8241ecc2241f3 P d9dc68ace903455d809cb1f715fe36be [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0703 01:43:12.734423 1564 raft_consensus.cc:457] T 454bab377db147a1a2f8241ecc2241f3 P 56eaab17f812433a8f7880f0744e1744 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2f3fdb7c089640c0b936a574bc64707e) I0703 01:43:12.734525 1564 raft_consensus.cc:2730] T 454bab377db147a1a2f8241ecc2241f3 P 56eaab17f812433a8f7880f0744e1744: Snoozing failure detection for 2.214s (starting election) I0703 01:43:12.734614 1564 raft_consensus.cc:479] T 454bab377db147a1a2f8241ecc2241f3 P 56eaab17f812433a8f7880f0744e1744 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } I0703 01:43:12.735117 1564 leader_election.cc:243] T 454bab377db147a1a2f8241ecc2241f3 P 56eaab17f812433a8f7880f0744e1744 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194:39827) I0703 01:43:12.735266 1564 leader_election.cc:243] T 454bab377db147a1a2f8241ecc2241f3 P 56eaab17f812433a8f7880f0744e1744 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer d9dc68ace903455d809cb1f715fe36be (127.5.240.196:56543) I0703 01:43:12.736521 31393 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "454bab377db147a1a2f8241ecc2241f3" candidate_uuid: "56eaab17f812433a8f7880f0744e1744" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2f3fdb7c089640c0b936a574bc64707e" is_pre_election: true I0703 01:43:12.736726 31393 raft_consensus.cc:2173] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 56eaab17f812433a8f7880f0744e1744 in current term 2: Already voted for candidate d9dc68ace903455d809cb1f715fe36be in this term. I0703 01:43:12.738041 358 leader_election.cc:393] T 454bab377db147a1a2f8241ecc2241f3 P 56eaab17f812433a8f7880f0744e1744 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194:39827). Message: Invalid argument: T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 56eaab17f812433a8f7880f0744e1744 in current term 2: Already voted for candidate d9dc68ace903455d809cb1f715fe36be in this term. I0703 01:43:12.754479 1752 raft_consensus.cc:2730] T 454bab377db147a1a2f8241ecc2241f3 P d9dc68ace903455d809cb1f715fe36be: Snoozing failure detection for 1.600s (election complete) I0703 01:43:12.754583 1752 raft_consensus.cc:2537] T 454bab377db147a1a2f8241ecc2241f3 P d9dc68ace903455d809cb1f715fe36be [term 2 FOLLOWER]: Leader election won for term 2 I0703 01:43:12.754989 1752 raft_consensus.cc:593] T 454bab377db147a1a2f8241ecc2241f3 P d9dc68ace903455d809cb1f715fe36be [term 2 LEADER]: Becoming Leader. State: Replica: d9dc68ace903455d809cb1f715fe36be, State: Running, Role: LEADER I0703 01:43:12.755193 1752 consensus_queue.cc:204] T 454bab377db147a1a2f8241ecc2241f3 P d9dc68ace903455d809cb1f715fe36be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } I0703 01:43:12.776314 432 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "454bab377db147a1a2f8241ecc2241f3" candidate_uuid: "d9dc68ace903455d809cb1f715fe36be" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "56eaab17f812433a8f7880f0744e1744" I0703 01:43:12.776597 432 raft_consensus.cc:2782] T 454bab377db147a1a2f8241ecc2241f3 P 56eaab17f812433a8f7880f0744e1744 [term 1 FOLLOWER]: Advancing to term 2 I0703 01:43:12.776736 432 raft_consensus.cc:2730] T 454bab377db147a1a2f8241ecc2241f3 P 56eaab17f812433a8f7880f0744e1744: Snoozing failure detection for 1.654s (vote granted) I0703 01:43:12.776703 434 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "454bab377db147a1a2f8241ecc2241f3" candidate_uuid: "d9dc68ace903455d809cb1f715fe36be" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "56eaab17f812433a8f7880f0744e1744" is_pre_election: true I0703 01:43:12.778851 432 raft_consensus.cc:2243] T 454bab377db147a1a2f8241ecc2241f3 P 56eaab17f812433a8f7880f0744e1744 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9dc68ace903455d809cb1f715fe36be in term 2. I0703 01:43:12.779019 434 raft_consensus.cc:2218] T 454bab377db147a1a2f8241ecc2241f3 P 56eaab17f812433a8f7880f0744e1744 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9dc68ace903455d809cb1f715fe36be for term 2 because replica is already servicing an update from a current leader or another vote. I0703 01:43:12.781692 32434 leader_election.cc:379] T 454bab377db147a1a2f8241ecc2241f3 P d9dc68ace903455d809cb1f715fe36be [CANDIDATE]: Term 2 election: Vote granted by peer 56eaab17f812433a8f7880f0744e1744 (127.5.240.199:57714) I0703 01:43:12.782052 32434 leader_election.cc:393] T 454bab377db147a1a2f8241ecc2241f3 P d9dc68ace903455d809cb1f715fe36be [CANDIDATE]: Term 2 pre-election: Vote denied by peer 56eaab17f812433a8f7880f0744e1744 (127.5.240.199:57714). Message: Service unavailable: T 454bab377db147a1a2f8241ecc2241f3 P 56eaab17f812433a8f7880f0744e1744 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9dc68ace903455d809cb1f715fe36be for term 2 because replica is already servicing an update from a current leader or another vote. I0703 01:43:12.805572 31149 catalog_manager.cc:3798] T 454bab377db147a1a2f8241ecc2241f3 P d9dc68ace903455d809cb1f715fe36be reported cstate change: term changed from 1 to 2, leader changed from 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194) to d9dc68ace903455d809cb1f715fe36be (127.5.240.196). New cstate: current_term: 2 leader_uuid: "d9dc68ace903455d809cb1f715fe36be" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } } I0703 01:43:12.824760 32503 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "454bab377db147a1a2f8241ecc2241f3" candidate_uuid: "56eaab17f812433a8f7880f0744e1744" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d9dc68ace903455d809cb1f715fe36be" is_pre_election: true I0703 01:43:12.825074 32503 raft_consensus.cc:2204] T 454bab377db147a1a2f8241ecc2241f3 P d9dc68ace903455d809cb1f715fe36be [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 56eaab17f812433a8f7880f0744e1744 for term 2 because replica is either leader or believes a valid leader to be alive. I0703 01:43:12.826712 360 leader_election.cc:393] T 454bab377db147a1a2f8241ecc2241f3 P 56eaab17f812433a8f7880f0744e1744 [CANDIDATE]: Term 2 pre-election: Vote denied by peer d9dc68ace903455d809cb1f715fe36be (127.5.240.196:56543). Message: Invalid argument: T 454bab377db147a1a2f8241ecc2241f3 P d9dc68ace903455d809cb1f715fe36be [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 56eaab17f812433a8f7880f0744e1744 for term 2 because replica is either leader or believes a valid leader to be alive. I0703 01:43:12.826797 360 leader_election.cc:270] T 454bab377db147a1a2f8241ecc2241f3 P 56eaab17f812433a8f7880f0744e1744 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0703 01:43:12.827060 1564 raft_consensus.cc:2730] T 454bab377db147a1a2f8241ecc2241f3 P 56eaab17f812433a8f7880f0744e1744: Snoozing failure detection for 1.817s (election complete) I0703 01:43:12.827153 1564 raft_consensus.cc:2485] T 454bab377db147a1a2f8241ecc2241f3 P 56eaab17f812433a8f7880f0744e1744 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0703 01:43:12.895550 31514 tablet_service.cc:1140] Received LeaderStepDown RPC: tablet_id: "19c33e54ee9e476796de7de42ba495b6" dest_uuid: "de93b58c2d67488dae12285e1806ebac" from {username='jenkins'} at 127.5.240.195:47147 I0703 01:43:12.895726 31514 raft_consensus.cc:552] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac [term 1 LEADER]: Received request to step down I0703 01:43:12.895761 31514 raft_consensus.cc:628] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac [term 1 LEADER]: Becoming Follower/Learner. State: Replica: de93b58c2d67488dae12285e1806ebac, State: Running, Role: LEADER I0703 01:43:12.895952 31514 consensus_queue.cc:227] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 485, Committed index: 485, Last appended: 1.485, Last appended by leader: 485, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } I0703 01:43:12.896152 31514 consensus_peers.cc:455] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac -> Peer d9dc68ace903455d809cb1f715fe36be (127.5.240.196:56543): Closing peer: d9dc68ace903455d809cb1f715fe36be I0703 01:43:12.896194 31514 consensus_peers.cc:455] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac -> Peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521): Closing peer: 1288ca0989104bbea36e22f38c374218 I0703 01:43:12.896333 31514 raft_consensus.cc:2730] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac: Snoozing failure detection for 3.000s (explicit stepdown request) I0703 01:43:13.171885 31393 raft_consensus.cc:1081] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e [term 2 FOLLOWER]: Refusing update from remote peer d9dc68ace903455d809cb1f715fe36be: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch) I0703 01:43:13.175351 1752 consensus_queue.cc:962] T 454bab377db147a1a2f8241ecc2241f3 P d9dc68ace903455d809cb1f715fe36be [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s I0703 01:43:13.199800 431 raft_consensus.cc:1081] T 454bab377db147a1a2f8241ecc2241f3 P 56eaab17f812433a8f7880f0744e1744 [term 2 FOLLOWER]: Refusing update from remote peer d9dc68ace903455d809cb1f715fe36be: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch) I0703 01:43:13.202111 1755 consensus_queue.cc:962] T 454bab377db147a1a2f8241ecc2241f3 P d9dc68ace903455d809cb1f715fe36be [LEADER]: Connected to new peer: Peer: permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s I0703 01:43:14.286316 31394 consensus_peers.cc:455] T 44f06af6e8ef46d78419aea845bb38a7 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 56eaab17f812433a8f7880f0744e1744 (127.5.240.199:57714): Closing peer: 56eaab17f812433a8f7880f0744e1744 I0703 01:43:14.286379 31394 consensus_peers.cc:455] T 44f06af6e8ef46d78419aea845bb38a7 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521): Closing peer: 1288ca0989104bbea36e22f38c374218 I0703 01:43:14.286640 31394 consensus_queue.cc:204] T 44f06af6e8ef46d78419aea845bb38a7 P 2f3fdb7c089640c0b936a574bc64707e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 483, Committed index: 483, Last appended: 1.483, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 484 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } I0703 01:43:14.300096 434 raft_consensus.cc:1081] T 44f06af6e8ef46d78419aea845bb38a7 P 56eaab17f812433a8f7880f0744e1744 [term 1 FOLLOWER]: Refusing update from remote peer 2f3fdb7c089640c0b936a574bc64707e: Log matching property violated. Preceding OpId in replica: term: 1 index: 483. Preceding OpId from leader: term: 1 index: 484. (index mismatch) I0703 01:43:14.301190 765 consensus_queue.cc:962] T 44f06af6e8ef46d78419aea845bb38a7 P 2f3fdb7c089640c0b936a574bc64707e [LEADER]: Connected to new peer: Peer: permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 484, Last known committed idx: 483, Time since last communication: 0.000s I0703 01:43:14.340051 763 raft_consensus.cc:2684] T 44f06af6e8ef46d78419aea845bb38a7 P 2f3fdb7c089640c0b936a574bc64707e [term 1 LEADER]: Committing config change with OpId 1.484: config changed from index 203 to 484, VOTER 1288ca0989104bbea36e22f38c374218 (127.5.240.193) evicted. New config: { opid_index: 484 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } } I0703 01:43:14.340930 434 raft_consensus.cc:2684] T 44f06af6e8ef46d78419aea845bb38a7 P 56eaab17f812433a8f7880f0744e1744 [term 1 FOLLOWER]: Committing config change with OpId 1.484: config changed from index 203 to 484, VOTER 1288ca0989104bbea36e22f38c374218 (127.5.240.193) evicted. New config: { opid_index: 484 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } } I0703 01:43:14.345966 31149 catalog_manager.cc:3798] T 44f06af6e8ef46d78419aea845bb38a7 P 56eaab17f812433a8f7880f0744e1744 reported cstate change: config changed from index 203 to 484, VOTER 1288ca0989104bbea36e22f38c374218 (127.5.240.193) evicted. New cstate: current_term: 1 leader_uuid: "2f3fdb7c089640c0b936a574bc64707e" committed_config { opid_index: 484 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } } I0703 01:43:14.351877 1639 raft_consensus.cc:457] T 19c33e54ee9e476796de7de42ba495b6 P d9dc68ace903455d809cb1f715fe36be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader de93b58c2d67488dae12285e1806ebac) I0703 01:43:14.351948 1639 raft_consensus.cc:2730] T 19c33e54ee9e476796de7de42ba495b6 P d9dc68ace903455d809cb1f715fe36be: Snoozing failure detection for 1.798s (starting election) I0703 01:43:14.352039 1639 raft_consensus.cc:479] T 19c33e54ee9e476796de7de42ba495b6 P d9dc68ace903455d809cb1f715fe36be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } I0703 01:43:14.352525 1639 leader_election.cc:243] T 19c33e54ee9e476796de7de42ba495b6 P d9dc68ace903455d809cb1f715fe36be [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321) I0703 01:43:14.352697 1639 leader_election.cc:243] T 19c33e54ee9e476796de7de42ba495b6 P d9dc68ace903455d809cb1f715fe36be [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521) I0703 01:43:14.354408 31517 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "19c33e54ee9e476796de7de42ba495b6" candidate_uuid: "d9dc68ace903455d809cb1f715fe36be" candidate_term: 2 candidate_status { last_received { term: 1 index: 485 } } ignore_live_leader: false dest_uuid: "de93b58c2d67488dae12285e1806ebac" is_pre_election: true I0703 01:43:14.354563 31517 raft_consensus.cc:2730] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac: Snoozing failure detection for 2.086s (vote granted) I0703 01:43:14.354655 31517 raft_consensus.cc:2243] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9dc68ace903455d809cb1f715fe36be in term 1. I0703 01:43:14.356335 32435 leader_election.cc:379] T 19c33e54ee9e476796de7de42ba495b6 P d9dc68ace903455d809cb1f715fe36be [CANDIDATE]: Term 2 pre-election: Vote granted by peer de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321) I0703 01:43:14.356429 32435 leader_election.cc:270] T 19c33e54ee9e476796de7de42ba495b6 P d9dc68ace903455d809cb1f715fe36be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. I0703 01:43:14.356667 1752 raft_consensus.cc:2730] T 19c33e54ee9e476796de7de42ba495b6 P d9dc68ace903455d809cb1f715fe36be: Snoozing failure detection for 1.849s (election complete) I0703 01:43:14.356745 1752 raft_consensus.cc:2537] T 19c33e54ee9e476796de7de42ba495b6 P d9dc68ace903455d809cb1f715fe36be [term 1 FOLLOWER]: Leader pre-election won for term 2 I0703 01:43:14.356794 1752 raft_consensus.cc:457] T 19c33e54ee9e476796de7de42ba495b6 P d9dc68ace903455d809cb1f715fe36be [term 1 FOLLOWER]: Starting leader election (detected failure of leader de93b58c2d67488dae12285e1806ebac) I0703 01:43:14.356855 1752 raft_consensus.cc:2730] T 19c33e54ee9e476796de7de42ba495b6 P d9dc68ace903455d809cb1f715fe36be: Snoozing failure detection for 2.205s (starting election) I0703 01:43:14.356907 1752 raft_consensus.cc:2782] T 19c33e54ee9e476796de7de42ba495b6 P d9dc68ace903455d809cb1f715fe36be [term 1 FOLLOWER]: Advancing to term 2 I0703 01:43:14.359304 1752 raft_consensus.cc:479] T 19c33e54ee9e476796de7de42ba495b6 P d9dc68ace903455d809cb1f715fe36be [term 2 FOLLOWER]: Starting leader election with config: opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } I0703 01:43:14.359782 1752 leader_election.cc:243] T 19c33e54ee9e476796de7de42ba495b6 P d9dc68ace903455d809cb1f715fe36be [CANDIDATE]: Term 2 election: Requesting vote from peer de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321) I0703 01:43:14.359925 1752 leader_election.cc:243] T 19c33e54ee9e476796de7de42ba495b6 P d9dc68ace903455d809cb1f715fe36be [CANDIDATE]: Term 2 election: Requesting vote from peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521) I0703 01:43:14.361138 31517 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "19c33e54ee9e476796de7de42ba495b6" candidate_uuid: "d9dc68ace903455d809cb1f715fe36be" candidate_term: 2 candidate_status { last_received { term: 1 index: 485 } } ignore_live_leader: false dest_uuid: "de93b58c2d67488dae12285e1806ebac" I0703 01:43:14.361279 31517 raft_consensus.cc:2782] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac [term 1 FOLLOWER]: Advancing to term 2 I0703 01:43:14.361344 31517 raft_consensus.cc:2730] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac: Snoozing failure detection for 1.985s (vote granted) I0703 01:43:14.361671 31272 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "19c33e54ee9e476796de7de42ba495b6" candidate_uuid: "d9dc68ace903455d809cb1f715fe36be" candidate_term: 2 candidate_status { last_received { term: 1 index: 485 } } ignore_live_leader: false dest_uuid: "1288ca0989104bbea36e22f38c374218" is_pre_election: true I0703 01:43:14.361837 31272 raft_consensus.cc:2204] T 19c33e54ee9e476796de7de42ba495b6 P 1288ca0989104bbea36e22f38c374218 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9dc68ace903455d809cb1f715fe36be for term 2 because replica is either leader or believes a valid leader to be alive. I0703 01:43:14.362687 31272 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "19c33e54ee9e476796de7de42ba495b6" candidate_uuid: "d9dc68ace903455d809cb1f715fe36be" candidate_term: 2 candidate_status { last_received { term: 1 index: 485 } } ignore_live_leader: false dest_uuid: "1288ca0989104bbea36e22f38c374218" I0703 01:43:14.362862 31272 raft_consensus.cc:2204] T 19c33e54ee9e476796de7de42ba495b6 P 1288ca0989104bbea36e22f38c374218 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d9dc68ace903455d809cb1f715fe36be for term 2 because replica is either leader or believes a valid leader to be alive. I0703 01:43:14.363318 31517 raft_consensus.cc:2243] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9dc68ace903455d809cb1f715fe36be in term 2. I0703 01:43:14.364277 32433 leader_election.cc:393] T 19c33e54ee9e476796de7de42ba495b6 P d9dc68ace903455d809cb1f715fe36be [CANDIDATE]: Term 2 pre-election: Vote denied by peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521). Message: Invalid argument: T 19c33e54ee9e476796de7de42ba495b6 P 1288ca0989104bbea36e22f38c374218 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d9dc68ace903455d809cb1f715fe36be for term 2 because replica is either leader or believes a valid leader to be alive. I0703 01:43:14.364678 32433 leader_election.cc:393] T 19c33e54ee9e476796de7de42ba495b6 P d9dc68ace903455d809cb1f715fe36be [CANDIDATE]: Term 2 election: Vote denied by peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521). Message: Invalid argument: T 19c33e54ee9e476796de7de42ba495b6 P 1288ca0989104bbea36e22f38c374218 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d9dc68ace903455d809cb1f715fe36be for term 2 because replica is either leader or believes a valid leader to be alive. I0703 01:43:14.364964 32435 leader_election.cc:379] T 19c33e54ee9e476796de7de42ba495b6 P d9dc68ace903455d809cb1f715fe36be [CANDIDATE]: Term 2 election: Vote granted by peer de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321) I0703 01:43:14.365029 32435 leader_election.cc:270] T 19c33e54ee9e476796de7de42ba495b6 P d9dc68ace903455d809cb1f715fe36be [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0703 01:43:14.365581 1639 raft_consensus.cc:2730] T 19c33e54ee9e476796de7de42ba495b6 P d9dc68ace903455d809cb1f715fe36be: Snoozing failure detection for 1.512s (election complete) I0703 01:43:14.365689 1639 raft_consensus.cc:2537] T 19c33e54ee9e476796de7de42ba495b6 P d9dc68ace903455d809cb1f715fe36be [term 2 FOLLOWER]: Leader election won for term 2 I0703 01:43:14.365798 1639 raft_consensus.cc:593] T 19c33e54ee9e476796de7de42ba495b6 P d9dc68ace903455d809cb1f715fe36be [term 2 LEADER]: Becoming Leader. State: Replica: d9dc68ace903455d809cb1f715fe36be, State: Running, Role: LEADER I0703 01:43:14.366070 1639 consensus_queue.cc:204] T 19c33e54ee9e476796de7de42ba495b6 P d9dc68ace903455d809cb1f715fe36be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 485, Committed index: 485, Last appended: 1.485, Last appended by leader: 485, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } I0703 01:43:14.374754 1081 raft_consensus.cc:457] T 19c33e54ee9e476796de7de42ba495b6 P 1288ca0989104bbea36e22f38c374218 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader de93b58c2d67488dae12285e1806ebac) I0703 01:43:14.374842 1081 raft_consensus.cc:2730] T 19c33e54ee9e476796de7de42ba495b6 P 1288ca0989104bbea36e22f38c374218: Snoozing failure detection for 1.755s (starting election) I0703 01:43:14.374912 1081 raft_consensus.cc:479] T 19c33e54ee9e476796de7de42ba495b6 P 1288ca0989104bbea36e22f38c374218 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } I0703 01:43:14.375378 1081 leader_election.cc:243] T 19c33e54ee9e476796de7de42ba495b6 P 1288ca0989104bbea36e22f38c374218 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321) I0703 01:43:14.375494 1081 leader_election.cc:243] T 19c33e54ee9e476796de7de42ba495b6 P 1288ca0989104bbea36e22f38c374218 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer d9dc68ace903455d809cb1f715fe36be (127.5.240.196:56543) I0703 01:43:14.376312 32503 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "19c33e54ee9e476796de7de42ba495b6" candidate_uuid: "1288ca0989104bbea36e22f38c374218" candidate_term: 2 candidate_status { last_received { term: 1 index: 485 } } ignore_live_leader: false dest_uuid: "d9dc68ace903455d809cb1f715fe36be" is_pre_election: true I0703 01:43:14.386737 31517 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "19c33e54ee9e476796de7de42ba495b6" candidate_uuid: "1288ca0989104bbea36e22f38c374218" candidate_term: 2 candidate_status { last_received { term: 1 index: 485 } } ignore_live_leader: false dest_uuid: "de93b58c2d67488dae12285e1806ebac" is_pre_election: true I0703 01:43:14.388661 31517 raft_consensus.cc:2173] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1288ca0989104bbea36e22f38c374218 in current term 2: Already voted for candidate d9dc68ace903455d809cb1f715fe36be in this term. I0703 01:43:14.392110 31206 leader_election.cc:393] T 19c33e54ee9e476796de7de42ba495b6 P 1288ca0989104bbea36e22f38c374218 [CANDIDATE]: Term 2 pre-election: Vote denied by peer de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321). Message: Invalid argument: T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1288ca0989104bbea36e22f38c374218 in current term 2: Already voted for candidate d9dc68ace903455d809cb1f715fe36be in this term. I0703 01:43:14.392354 32503 raft_consensus.cc:2204] T 19c33e54ee9e476796de7de42ba495b6 P d9dc68ace903455d809cb1f715fe36be [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 1288ca0989104bbea36e22f38c374218 for term 2 because replica is either leader or believes a valid leader to be alive. I0703 01:43:14.393159 31205 leader_election.cc:393] T 19c33e54ee9e476796de7de42ba495b6 P 1288ca0989104bbea36e22f38c374218 [CANDIDATE]: Term 2 pre-election: Vote denied by peer d9dc68ace903455d809cb1f715fe36be (127.5.240.196:56543). Message: Invalid argument: T 19c33e54ee9e476796de7de42ba495b6 P d9dc68ace903455d809cb1f715fe36be [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 1288ca0989104bbea36e22f38c374218 for term 2 because replica is either leader or believes a valid leader to be alive. I0703 01:43:14.393232 31205 leader_election.cc:270] T 19c33e54ee9e476796de7de42ba495b6 P 1288ca0989104bbea36e22f38c374218 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0703 01:43:14.393393 1587 raft_consensus.cc:2730] T 19c33e54ee9e476796de7de42ba495b6 P 1288ca0989104bbea36e22f38c374218: Snoozing failure detection for 2.191s (election complete) I0703 01:43:14.393527 1587 raft_consensus.cc:2782] T 19c33e54ee9e476796de7de42ba495b6 P 1288ca0989104bbea36e22f38c374218 [term 1 FOLLOWER]: Advancing to term 2 I0703 01:43:14.395634 31149 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 44f06af6e8ef46d78419aea845bb38a7 on 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521) (TS 1288ca0989104bbea36e22f38c374218 not found in new config with opid_index 484) I0703 01:43:14.395855 1587 raft_consensus.cc:2485] T 19c33e54ee9e476796de7de42ba495b6 P 1288ca0989104bbea36e22f38c374218 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0703 01:43:14.397281 31250 tablet_service.cc:800] Processing DeleteTablet for tablet 44f06af6e8ef46d78419aea845bb38a7 with delete_type TABLET_DATA_TOMBSTONED (TS 1288ca0989104bbea36e22f38c374218 not found in new config with opid_index 484) from {username='jenkins'} at 127.5.240.193:33318 I0703 01:43:14.398185 1872 tablet_replica.cc:261] T 44f06af6e8ef46d78419aea845bb38a7 P 1288ca0989104bbea36e22f38c374218: stopping tablet replica I0703 01:43:14.399255 1872 maintenance_manager.cc:237] P 1288ca0989104bbea36e22f38c374218: Unregistered op CompactRowSetsOp(44f06af6e8ef46d78419aea845bb38a7) I0703 01:43:14.399386 1872 maintenance_manager.cc:237] P 1288ca0989104bbea36e22f38c374218: Unregistered op MinorDeltaCompactionOp(44f06af6e8ef46d78419aea845bb38a7) I0703 01:43:14.399471 1872 maintenance_manager.cc:237] P 1288ca0989104bbea36e22f38c374218: Unregistered op MajorDeltaCompactionOp(44f06af6e8ef46d78419aea845bb38a7) I0703 01:43:14.399582 1872 maintenance_manager.cc:237] P 1288ca0989104bbea36e22f38c374218: Unregistered op UndoDeltaBlockGCOp(44f06af6e8ef46d78419aea845bb38a7) I0703 01:43:14.399667 1872 maintenance_manager.cc:237] P 1288ca0989104bbea36e22f38c374218: Unregistered op FlushMRSOp(44f06af6e8ef46d78419aea845bb38a7) I0703 01:43:14.399739 1872 maintenance_manager.cc:237] P 1288ca0989104bbea36e22f38c374218: Unregistered op FlushDeltaMemStoresOp(44f06af6e8ef46d78419aea845bb38a7) I0703 01:43:14.399806 1872 maintenance_manager.cc:237] P 1288ca0989104bbea36e22f38c374218: Unregistered op LogGCOp(44f06af6e8ef46d78419aea845bb38a7) I0703 01:43:14.399871 1872 raft_consensus.cc:2040] T 44f06af6e8ef46d78419aea845bb38a7 P 1288ca0989104bbea36e22f38c374218 [term 1 FOLLOWER]: Raft consensus shutting down. I0703 01:43:14.400792 1872 raft_consensus.cc:2067] T 44f06af6e8ef46d78419aea845bb38a7 P 1288ca0989104bbea36e22f38c374218 [term 1 FOLLOWER]: Raft consensus is shut down! I0703 01:43:14.401801 1872 ts_tablet_manager.cc:1369] T 44f06af6e8ef46d78419aea845bb38a7 P 1288ca0989104bbea36e22f38c374218: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:43:14.402671 31147 catalog_manager.cc:3798] T 19c33e54ee9e476796de7de42ba495b6 P d9dc68ace903455d809cb1f715fe36be reported cstate change: term changed from 1 to 2, leader changed from de93b58c2d67488dae12285e1806ebac (127.5.240.195) to d9dc68ace903455d809cb1f715fe36be (127.5.240.196). New cstate: current_term: 2 leader_uuid: "d9dc68ace903455d809cb1f715fe36be" committed_config { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } } I0703 01:43:14.412919 1872 ts_tablet_manager.cc:1382] T 44f06af6e8ef46d78419aea845bb38a7 P 1288ca0989104bbea36e22f38c374218: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.483 I0703 01:43:14.413017 1872 log.cc:986] T 44f06af6e8ef46d78419aea845bb38a7 P 1288ca0989104bbea36e22f38c374218: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal/wals/44f06af6e8ef46d78419aea845bb38a7 I0703 01:43:14.414573 31134 catalog_manager.cc:3103] TS 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521): tablet 44f06af6e8ef46d78419aea845bb38a7 (table rebalance_test_table_0 [id=9d68c0e8280a4d18af7d08099d85f9a3]) successfully deleted I0703 01:43:14.484793 31268 raft_consensus.cc:1081] T 19c33e54ee9e476796de7de42ba495b6 P 1288ca0989104bbea36e22f38c374218 [term 2 FOLLOWER]: Refusing update from remote peer d9dc68ace903455d809cb1f715fe36be: Log matching property violated. Preceding OpId in replica: term: 1 index: 485. Preceding OpId from leader: term: 2 index: 487. (index mismatch) I0703 01:43:14.485177 31514 raft_consensus.cc:1081] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac [term 2 FOLLOWER]: Refusing update from remote peer d9dc68ace903455d809cb1f715fe36be: Log matching property violated. Preceding OpId in replica: term: 1 index: 485. Preceding OpId from leader: term: 2 index: 487. (index mismatch) I0703 01:43:14.485886 1639 consensus_queue.cc:962] T 19c33e54ee9e476796de7de42ba495b6 P d9dc68ace903455d809cb1f715fe36be [LEADER]: Connected to new peer: Peer: permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 486, Last known committed idx: 485, Time since last communication: 0.000s I0703 01:43:14.486940 1639 consensus_queue.cc:962] T 19c33e54ee9e476796de7de42ba495b6 P d9dc68ace903455d809cb1f715fe36be [LEADER]: Connected to new peer: Peer: permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 486, Last known committed idx: 485, Time since last communication: 0.000s I0703 01:43:15.284608 434 consensus_peers.cc:455] T bbe64aa4b0e04f468b035048629b969b P 56eaab17f812433a8f7880f0744e1744 -> Peer de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321): Closing peer: de93b58c2d67488dae12285e1806ebac I0703 01:43:15.284713 434 consensus_peers.cc:455] T bbe64aa4b0e04f468b035048629b969b P 56eaab17f812433a8f7880f0744e1744 -> Peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521): Closing peer: 1288ca0989104bbea36e22f38c374218 I0703 01:43:15.285087 434 consensus_queue.cc:204] T bbe64aa4b0e04f468b035048629b969b P 56eaab17f812433a8f7880f0744e1744 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 528, Committed index: 528, Last appended: 2.528, Last appended by leader: 368, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 529 OBSOLETE_local: false peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } I0703 01:43:15.287015 31517 raft_consensus.cc:1081] T bbe64aa4b0e04f468b035048629b969b P de93b58c2d67488dae12285e1806ebac [term 2 FOLLOWER]: Refusing update from remote peer 56eaab17f812433a8f7880f0744e1744: Log matching property violated. Preceding OpId in replica: term: 2 index: 528. Preceding OpId from leader: term: 2 index: 529. (index mismatch) I0703 01:43:15.287698 1790 consensus_queue.cc:962] T bbe64aa4b0e04f468b035048629b969b P 56eaab17f812433a8f7880f0744e1744 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 529, Last known committed idx: 528, Time since last communication: 0.000s I0703 01:43:15.290758 1790 raft_consensus.cc:2684] T bbe64aa4b0e04f468b035048629b969b P 56eaab17f812433a8f7880f0744e1744 [term 2 LEADER]: Committing config change with OpId 2.529: config changed from index 156 to 529, VOTER 1288ca0989104bbea36e22f38c374218 (127.5.240.193) evicted. New config: { opid_index: 529 OBSOLETE_local: false peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } } I0703 01:43:15.291494 31514 raft_consensus.cc:2684] T bbe64aa4b0e04f468b035048629b969b P de93b58c2d67488dae12285e1806ebac [term 2 FOLLOWER]: Committing config change with OpId 2.529: config changed from index 156 to 529, VOTER 1288ca0989104bbea36e22f38c374218 (127.5.240.193) evicted. New config: { opid_index: 529 OBSOLETE_local: false peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } } I0703 01:43:15.296044 31152 catalog_manager.cc:3798] T bbe64aa4b0e04f468b035048629b969b P de93b58c2d67488dae12285e1806ebac reported cstate change: config changed from index 156 to 529, VOTER 1288ca0989104bbea36e22f38c374218 (127.5.240.193) evicted. New cstate: current_term: 2 leader_uuid: "56eaab17f812433a8f7880f0744e1744" committed_config { opid_index: 529 OBSOLETE_local: false peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } } I0703 01:43:15.325218 31152 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet bbe64aa4b0e04f468b035048629b969b on 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521) (TS 1288ca0989104bbea36e22f38c374218 not found in new config with opid_index 529) I0703 01:43:15.326207 31250 tablet_service.cc:800] Processing DeleteTablet for tablet bbe64aa4b0e04f468b035048629b969b with delete_type TABLET_DATA_TOMBSTONED (TS 1288ca0989104bbea36e22f38c374218 not found in new config with opid_index 529) from {username='jenkins'} at 127.5.240.193:33318 I0703 01:43:15.357060 1950 tablet_replica.cc:261] T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218: stopping tablet replica I0703 01:43:15.358491 1950 maintenance_manager.cc:237] P 1288ca0989104bbea36e22f38c374218: Unregistered op CompactRowSetsOp(bbe64aa4b0e04f468b035048629b969b) I0703 01:43:15.358603 1950 maintenance_manager.cc:237] P 1288ca0989104bbea36e22f38c374218: Unregistered op MinorDeltaCompactionOp(bbe64aa4b0e04f468b035048629b969b) I0703 01:43:15.358729 1950 maintenance_manager.cc:237] P 1288ca0989104bbea36e22f38c374218: Unregistered op MajorDeltaCompactionOp(bbe64aa4b0e04f468b035048629b969b) I0703 01:43:15.358994 1950 maintenance_manager.cc:237] P 1288ca0989104bbea36e22f38c374218: Unregistered op UndoDeltaBlockGCOp(bbe64aa4b0e04f468b035048629b969b) I0703 01:43:15.359102 1950 maintenance_manager.cc:237] P 1288ca0989104bbea36e22f38c374218: Unregistered op FlushMRSOp(bbe64aa4b0e04f468b035048629b969b) I0703 01:43:15.359913 1950 maintenance_manager.cc:237] P 1288ca0989104bbea36e22f38c374218: Unregistered op FlushDeltaMemStoresOp(bbe64aa4b0e04f468b035048629b969b) I0703 01:43:15.362507 1950 maintenance_manager.cc:237] P 1288ca0989104bbea36e22f38c374218: Unregistered op LogGCOp(bbe64aa4b0e04f468b035048629b969b) I0703 01:43:15.363885 1950 raft_consensus.cc:2040] T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218 [term 2 FOLLOWER]: Raft consensus shutting down. I0703 01:43:15.364702 1950 raft_consensus.cc:2067] T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218 [term 2 FOLLOWER]: Raft consensus is shut down! I0703 01:43:15.369105 1950 ts_tablet_manager.cc:1369] T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:43:15.385933 1950 ts_tablet_manager.cc:1382] T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.528 I0703 01:43:15.386077 1950 log.cc:986] T bbe64aa4b0e04f468b035048629b969b P 1288ca0989104bbea36e22f38c374218: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal/wals/bbe64aa4b0e04f468b035048629b969b I0703 01:43:15.386890 31134 catalog_manager.cc:3103] TS 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521): tablet bbe64aa4b0e04f468b035048629b969b (table rebalance_test_table_4 [id=b0493a1d5ef04606a6d4de013749c53b]) successfully deleted I0703 01:43:16.527794 32503 consensus_peers.cc:455] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be -> Peer de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321): Closing peer: de93b58c2d67488dae12285e1806ebac I0703 01:43:16.527881 32503 consensus_peers.cc:455] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be -> Peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521): Closing peer: 1288ca0989104bbea36e22f38c374218 I0703 01:43:16.528138 32503 consensus_queue.cc:204] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 610, Committed index: 610, Last appended: 2.610, Last appended by leader: 459, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 611 OBSOLETE_local: false peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } I0703 01:43:16.546416 31516 raft_consensus.cc:1081] T ffe66a5cfb564964951c619e51e7c47c P de93b58c2d67488dae12285e1806ebac [term 2 FOLLOWER]: Refusing update from remote peer d9dc68ace903455d809cb1f715fe36be: Log matching property violated. Preceding OpId in replica: term: 2 index: 610. Preceding OpId from leader: term: 2 index: 611. (index mismatch) I0703 01:43:16.551717 2014 consensus_queue.cc:962] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be [LEADER]: Connected to new peer: Peer: permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 611, Last known committed idx: 610, Time since last communication: 0.000s I0703 01:43:16.556984 1639 raft_consensus.cc:2684] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be [term 2 LEADER]: Committing config change with OpId 2.611: config changed from index 189 to 611, VOTER 1288ca0989104bbea36e22f38c374218 (127.5.240.193) evicted. New config: { opid_index: 611 OBSOLETE_local: false peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } } I0703 01:43:16.557957 31516 raft_consensus.cc:2684] T ffe66a5cfb564964951c619e51e7c47c P de93b58c2d67488dae12285e1806ebac [term 2 FOLLOWER]: Committing config change with OpId 2.611: config changed from index 189 to 611, VOTER 1288ca0989104bbea36e22f38c374218 (127.5.240.193) evicted. New config: { opid_index: 611 OBSOLETE_local: false peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } } I0703 01:43:16.561579 31147 catalog_manager.cc:3798] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be reported cstate change: config changed from index 189 to 611, VOTER 1288ca0989104bbea36e22f38c374218 (127.5.240.193) evicted. New cstate: current_term: 2 leader_uuid: "d9dc68ace903455d809cb1f715fe36be" committed_config { opid_index: 611 OBSOLETE_local: false peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } } I0703 01:43:16.618525 31147 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet ffe66a5cfb564964951c619e51e7c47c on 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521) (TS 1288ca0989104bbea36e22f38c374218 not found in new config with opid_index 611) I0703 01:43:16.620488 31250 tablet_service.cc:800] Processing DeleteTablet for tablet ffe66a5cfb564964951c619e51e7c47c with delete_type TABLET_DATA_TOMBSTONED (TS 1288ca0989104bbea36e22f38c374218 not found in new config with opid_index 611) from {username='jenkins'} at 127.5.240.193:33318 I0703 01:43:16.625946 2042 tablet_replica.cc:261] T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218: stopping tablet replica I0703 01:43:16.628742 2042 maintenance_manager.cc:237] P 1288ca0989104bbea36e22f38c374218: Unregistered op CompactRowSetsOp(ffe66a5cfb564964951c619e51e7c47c) I0703 01:43:16.628859 2042 maintenance_manager.cc:237] P 1288ca0989104bbea36e22f38c374218: Unregistered op MinorDeltaCompactionOp(ffe66a5cfb564964951c619e51e7c47c) I0703 01:43:16.629567 2042 maintenance_manager.cc:237] P 1288ca0989104bbea36e22f38c374218: Unregistered op MajorDeltaCompactionOp(ffe66a5cfb564964951c619e51e7c47c) I0703 01:43:16.629707 2042 maintenance_manager.cc:237] P 1288ca0989104bbea36e22f38c374218: Unregistered op UndoDeltaBlockGCOp(ffe66a5cfb564964951c619e51e7c47c) I0703 01:43:16.629848 2042 maintenance_manager.cc:237] P 1288ca0989104bbea36e22f38c374218: Unregistered op FlushMRSOp(ffe66a5cfb564964951c619e51e7c47c) I0703 01:43:16.631108 2042 maintenance_manager.cc:237] P 1288ca0989104bbea36e22f38c374218: Unregistered op FlushDeltaMemStoresOp(ffe66a5cfb564964951c619e51e7c47c) I0703 01:43:16.631192 2042 maintenance_manager.cc:237] P 1288ca0989104bbea36e22f38c374218: Unregistered op LogGCOp(ffe66a5cfb564964951c619e51e7c47c) I0703 01:43:16.631273 2042 raft_consensus.cc:2040] T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218 [term 2 FOLLOWER]: Raft consensus shutting down. I0703 01:43:16.631422 2042 raft_consensus.cc:2067] T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218 [term 2 FOLLOWER]: Raft consensus is shut down! I0703 01:43:16.632361 2042 ts_tablet_manager.cc:1369] T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:43:16.651310 2042 ts_tablet_manager.cc:1382] T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.610 I0703 01:43:16.651432 2042 log.cc:986] T ffe66a5cfb564964951c619e51e7c47c P 1288ca0989104bbea36e22f38c374218: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal/wals/ffe66a5cfb564964951c619e51e7c47c I0703 01:43:16.652241 31134 catalog_manager.cc:3103] TS 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521): tablet ffe66a5cfb564964951c619e51e7c47c (table rebalance_test_table_4 [id=b0493a1d5ef04606a6d4de013749c53b]) successfully deleted W0703 01:43:16.848665 32379 outbound_call.cc:287] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.5.240.194:39827, user_credentials={real_user=jenkins}} blocked reactor thread for 52716us I0703 01:43:17.499315 32503 consensus_peers.cc:455] T 454bab377db147a1a2f8241ecc2241f3 P d9dc68ace903455d809cb1f715fe36be -> Peer 56eaab17f812433a8f7880f0744e1744 (127.5.240.199:57714): Closing peer: 56eaab17f812433a8f7880f0744e1744 I0703 01:43:17.499512 32503 consensus_peers.cc:455] T 454bab377db147a1a2f8241ecc2241f3 P d9dc68ace903455d809cb1f715fe36be -> Peer 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194:39827): Closing peer: 2f3fdb7c089640c0b936a574bc64707e I0703 01:43:17.499866 32503 consensus_queue.cc:204] T 454bab377db147a1a2f8241ecc2241f3 P d9dc68ace903455d809cb1f715fe36be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } I0703 01:43:17.525069 434 raft_consensus.cc:1081] T 454bab377db147a1a2f8241ecc2241f3 P 56eaab17f812433a8f7880f0744e1744 [term 2 FOLLOWER]: Refusing update from remote peer d9dc68ace903455d809cb1f715fe36be: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch) I0703 01:43:17.532415 1639 consensus_queue.cc:962] T 454bab377db147a1a2f8241ecc2241f3 P d9dc68ace903455d809cb1f715fe36be [LEADER]: Connected to new peer: Peer: permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I0703 01:43:17.548995 2014 raft_consensus.cc:2684] T 454bab377db147a1a2f8241ecc2241f3 P d9dc68ace903455d809cb1f715fe36be [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } } I0703 01:43:17.553462 434 raft_consensus.cc:2684] T 454bab377db147a1a2f8241ecc2241f3 P 56eaab17f812433a8f7880f0744e1744 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } } I0703 01:43:17.553714 31147 catalog_manager.cc:3798] T 454bab377db147a1a2f8241ecc2241f3 P d9dc68ace903455d809cb1f715fe36be reported cstate change: config changed from index 2 to 4, VOTER 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194) evicted. New cstate: current_term: 2 leader_uuid: "d9dc68ace903455d809cb1f715fe36be" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } } I0703 01:43:17.588213 31147 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 454bab377db147a1a2f8241ecc2241f3 on 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194:39827) (TS 2f3fdb7c089640c0b936a574bc64707e not found in new config with opid_index 4) I0703 01:43:17.596043 31375 tablet_service.cc:800] Processing DeleteTablet for tablet 454bab377db147a1a2f8241ecc2241f3 with delete_type TABLET_DATA_TOMBSTONED (TS 2f3fdb7c089640c0b936a574bc64707e not found in new config with opid_index 4) from {username='jenkins'} at 127.5.240.194:58888 I0703 01:43:17.600365 2112 tablet_replica.cc:261] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e: stopping tablet replica I0703 01:43:17.600579 2112 maintenance_manager.cc:237] P 2f3fdb7c089640c0b936a574bc64707e: Unregistered op CompactRowSetsOp(454bab377db147a1a2f8241ecc2241f3) I0703 01:43:17.600934 2112 maintenance_manager.cc:237] P 2f3fdb7c089640c0b936a574bc64707e: Unregistered op MinorDeltaCompactionOp(454bab377db147a1a2f8241ecc2241f3) I0703 01:43:17.601061 2112 maintenance_manager.cc:237] P 2f3fdb7c089640c0b936a574bc64707e: Unregistered op MajorDeltaCompactionOp(454bab377db147a1a2f8241ecc2241f3) I0703 01:43:17.601481 2112 maintenance_manager.cc:237] P 2f3fdb7c089640c0b936a574bc64707e: Unregistered op UndoDeltaBlockGCOp(454bab377db147a1a2f8241ecc2241f3) I0703 01:43:17.601620 2112 maintenance_manager.cc:237] P 2f3fdb7c089640c0b936a574bc64707e: Unregistered op FlushMRSOp(454bab377db147a1a2f8241ecc2241f3) I0703 01:43:17.601734 2112 maintenance_manager.cc:237] P 2f3fdb7c089640c0b936a574bc64707e: Unregistered op FlushDeltaMemStoresOp(454bab377db147a1a2f8241ecc2241f3) I0703 01:43:17.601843 2112 maintenance_manager.cc:237] P 2f3fdb7c089640c0b936a574bc64707e: Unregistered op LogGCOp(454bab377db147a1a2f8241ecc2241f3) I0703 01:43:17.601954 2112 raft_consensus.cc:2040] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e [term 2 FOLLOWER]: Raft consensus shutting down. I0703 01:43:17.602159 2112 raft_consensus.cc:2067] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e [term 2 FOLLOWER]: Raft consensus is shut down! I0703 01:43:17.604439 2112 ts_tablet_manager.cc:1369] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:43:17.645151 2112 ts_tablet_manager.cc:1382] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3 I0703 01:43:17.651399 2112 log.cc:986] T 454bab377db147a1a2f8241ecc2241f3 P 2f3fdb7c089640c0b936a574bc64707e: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal/wals/454bab377db147a1a2f8241ecc2241f3 I0703 01:43:17.663187 31133 catalog_manager.cc:3103] TS 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194:39827): tablet 454bab377db147a1a2f8241ecc2241f3 (table TestTable [id=fc16aaadf7d64bcc93cb3a3e710b1316]) successfully deleted W0703 01:43:18.306732 31203 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.240.194:39827, user_credentials={real_user=jenkins}} blocked reactor thread for 47440.1us I0703 01:43:18.549103 32503 consensus_peers.cc:455] T 19c33e54ee9e476796de7de42ba495b6 P d9dc68ace903455d809cb1f715fe36be -> Peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521): Closing peer: 1288ca0989104bbea36e22f38c374218 I0703 01:43:18.549214 32503 consensus_peers.cc:455] T 19c33e54ee9e476796de7de42ba495b6 P d9dc68ace903455d809cb1f715fe36be -> Peer de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321): Closing peer: de93b58c2d67488dae12285e1806ebac I0703 01:43:18.549480 32503 consensus_queue.cc:204] T 19c33e54ee9e476796de7de42ba495b6 P d9dc68ace903455d809cb1f715fe36be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 600, Committed index: 600, Last appended: 2.600, Last appended by leader: 485, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 601 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } I0703 01:43:18.564051 31271 raft_consensus.cc:1081] T 19c33e54ee9e476796de7de42ba495b6 P 1288ca0989104bbea36e22f38c374218 [term 2 FOLLOWER]: Refusing update from remote peer d9dc68ace903455d809cb1f715fe36be: Log matching property violated. Preceding OpId in replica: term: 2 index: 600. Preceding OpId from leader: term: 2 index: 601. (index mismatch) I0703 01:43:18.574928 2014 consensus_queue.cc:962] T 19c33e54ee9e476796de7de42ba495b6 P d9dc68ace903455d809cb1f715fe36be [LEADER]: Connected to new peer: Peer: permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 601, Last known committed idx: 599, Time since last communication: 0.000s I0703 01:43:18.588083 2014 raft_consensus.cc:2684] T 19c33e54ee9e476796de7de42ba495b6 P d9dc68ace903455d809cb1f715fe36be [term 2 LEADER]: Committing config change with OpId 2.601: config changed from index 208 to 601, VOTER de93b58c2d67488dae12285e1806ebac (127.5.240.195) evicted. New config: { opid_index: 601 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } } I0703 01:43:18.590785 31268 raft_consensus.cc:2684] T 19c33e54ee9e476796de7de42ba495b6 P 1288ca0989104bbea36e22f38c374218 [term 2 FOLLOWER]: Committing config change with OpId 2.601: config changed from index 208 to 601, VOTER de93b58c2d67488dae12285e1806ebac (127.5.240.195) evicted. New config: { opid_index: 601 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } } I0703 01:43:18.595451 31147 catalog_manager.cc:3798] T 19c33e54ee9e476796de7de42ba495b6 P d9dc68ace903455d809cb1f715fe36be reported cstate change: config changed from index 208 to 601, VOTER de93b58c2d67488dae12285e1806ebac (127.5.240.195) evicted. New cstate: current_term: 2 leader_uuid: "d9dc68ace903455d809cb1f715fe36be" committed_config { opid_index: 601 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } } I0703 01:43:18.628477 31147 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 19c33e54ee9e476796de7de42ba495b6 on de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321) (TS de93b58c2d67488dae12285e1806ebac not found in new config with opid_index 601) I0703 01:43:18.628981 31497 tablet_service.cc:800] Processing DeleteTablet for tablet 19c33e54ee9e476796de7de42ba495b6 with delete_type TABLET_DATA_TOMBSTONED (TS de93b58c2d67488dae12285e1806ebac not found in new config with opid_index 601) from {username='jenkins'} at 127.5.240.195:46850 I0703 01:43:18.635478 2183 tablet_replica.cc:261] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac: stopping tablet replica I0703 01:43:18.637871 2183 maintenance_manager.cc:237] P de93b58c2d67488dae12285e1806ebac: Unregistered op CompactRowSetsOp(19c33e54ee9e476796de7de42ba495b6) I0703 01:43:18.643543 2183 maintenance_manager.cc:237] P de93b58c2d67488dae12285e1806ebac: Unregistered op MinorDeltaCompactionOp(19c33e54ee9e476796de7de42ba495b6) I0703 01:43:18.643667 2183 maintenance_manager.cc:237] P de93b58c2d67488dae12285e1806ebac: Unregistered op MajorDeltaCompactionOp(19c33e54ee9e476796de7de42ba495b6) I0703 01:43:18.643764 2183 maintenance_manager.cc:237] P de93b58c2d67488dae12285e1806ebac: Unregistered op UndoDeltaBlockGCOp(19c33e54ee9e476796de7de42ba495b6) I0703 01:43:18.643859 2183 maintenance_manager.cc:237] P de93b58c2d67488dae12285e1806ebac: Unregistered op FlushMRSOp(19c33e54ee9e476796de7de42ba495b6) I0703 01:43:18.643945 2183 maintenance_manager.cc:237] P de93b58c2d67488dae12285e1806ebac: Unregistered op FlushDeltaMemStoresOp(19c33e54ee9e476796de7de42ba495b6) I0703 01:43:18.644623 2183 maintenance_manager.cc:237] P de93b58c2d67488dae12285e1806ebac: Unregistered op LogGCOp(19c33e54ee9e476796de7de42ba495b6) I0703 01:43:18.644711 2183 raft_consensus.cc:2040] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac [term 2 FOLLOWER]: Raft consensus shutting down. I0703 01:43:18.644887 2183 raft_consensus.cc:2067] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac [term 2 FOLLOWER]: Raft consensus is shut down! I0703 01:43:18.685181 2183 ts_tablet_manager.cc:1369] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:43:18.700024 2183 ts_tablet_manager.cc:1382] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.600 I0703 01:43:18.701058 2183 log.cc:986] T 19c33e54ee9e476796de7de42ba495b6 P de93b58c2d67488dae12285e1806ebac: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal/wals/19c33e54ee9e476796de7de42ba495b6 I0703 01:43:18.701851 31136 catalog_manager.cc:3103] TS de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321): tablet 19c33e54ee9e476796de7de42ba495b6 (table rebalance_test_table_0 [id=9d68c0e8280a4d18af7d08099d85f9a3]) successfully deleted I0703 01:43:19.984292 31271 consensus_peers.cc:455] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218 -> Peer 768d8de4b2ba40f180b18194c28be589 (127.5.240.198:42654): Closing peer: 768d8de4b2ba40f180b18194c28be589 I0703 01:43:19.984525 31271 consensus_queue.cc:204] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 516, Committed index: 516, Last appended: 1.516, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 517 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } I0703 01:43:19.986762 32767 raft_consensus.cc:1081] T a200f8a763f14d54a1e84082dbafce8f P 768d8de4b2ba40f180b18194c28be589 [term 1 FOLLOWER]: Refusing update from remote peer 1288ca0989104bbea36e22f38c374218: Log matching property violated. Preceding OpId in replica: term: 1 index: 516. Preceding OpId from leader: term: 1 index: 517. (index mismatch) W0703 01:43:19.988225 31205 consensus_peers.cc:433] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218 -> Peer d9dc68ace903455d809cb1f715fe36be (127.5.240.196:56543): Couldn't send request to peer d9dc68ace903455d809cb1f715fe36be for tablet a200f8a763f14d54a1e84082dbafce8f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a200f8a763f14d54a1e84082dbafce8f. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:43:19.994812 1911 consensus_queue.cc:962] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218 [LEADER]: Connected to new peer: Peer: permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 517, Last known committed idx: 516, Time since last communication: 0.000s I0703 01:43:19.998091 2085 raft_consensus.cc:2684] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218 [term 1 LEADER]: Committing config change with OpId 1.517: config changed from index 252 to 517, VOTER d9dc68ace903455d809cb1f715fe36be (127.5.240.196) added. New config: { opid_index: 517 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } } I0703 01:43:19.999161 32767 raft_consensus.cc:2684] T a200f8a763f14d54a1e84082dbafce8f P 768d8de4b2ba40f180b18194c28be589 [term 1 FOLLOWER]: Committing config change with OpId 1.517: config changed from index 252 to 517, VOTER d9dc68ace903455d809cb1f715fe36be (127.5.240.196) added. New config: { opid_index: 517 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } } I0703 01:43:20.006264 31147 catalog_manager.cc:3798] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218 reported cstate change: config changed from index 252 to 517, VOTER d9dc68ace903455d809cb1f715fe36be (127.5.240.196) added. New cstate: current_term: 1 leader_uuid: "1288ca0989104bbea36e22f38c374218" committed_config { opid_index: 517 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } } I0703 01:43:20.260577 31393 consensus_peers.cc:455] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 768d8de4b2ba40f180b18194c28be589 (127.5.240.198:42654): Closing peer: 768d8de4b2ba40f180b18194c28be589 I0703 01:43:20.260804 31393 consensus_queue.cc:204] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 745, Committed index: 745, Last appended: 1.745, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 746 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } peers { permanent_uuid: "3c1770123dc14bba80e00db5f182b585" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 44326 } } I0703 01:43:20.263190 32767 raft_consensus.cc:1081] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 768d8de4b2ba40f180b18194c28be589 [term 1 FOLLOWER]: Refusing update from remote peer 2f3fdb7c089640c0b936a574bc64707e: Log matching property violated. Preceding OpId in replica: term: 1 index: 745. Preceding OpId from leader: term: 1 index: 746. (index mismatch) I0703 01:43:20.263934 2168 consensus_queue.cc:962] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e [LEADER]: Connected to new peer: Peer: permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 746, Last known committed idx: 745, Time since last communication: 0.000s I0703 01:43:20.272454 1916 raft_consensus.cc:2684] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e [term 1 LEADER]: Committing config change with OpId 1.746: config changed from index 328 to 746, VOTER 3c1770123dc14bba80e00db5f182b585 (127.5.240.197) added. New config: { opid_index: 746 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } peers { permanent_uuid: "3c1770123dc14bba80e00db5f182b585" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 44326 } } } I0703 01:43:20.275698 32767 raft_consensus.cc:2684] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 768d8de4b2ba40f180b18194c28be589 [term 1 FOLLOWER]: Committing config change with OpId 1.746: config changed from index 328 to 746, VOTER 3c1770123dc14bba80e00db5f182b585 (127.5.240.197) added. New config: { opid_index: 746 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } peers { permanent_uuid: "3c1770123dc14bba80e00db5f182b585" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 44326 } } } I0703 01:43:20.285126 31147 catalog_manager.cc:3798] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e reported cstate change: config changed from index 328 to 746, VOTER 3c1770123dc14bba80e00db5f182b585 (127.5.240.197) added. New cstate: current_term: 1 leader_uuid: "2f3fdb7c089640c0b936a574bc64707e" committed_config { opid_index: 746 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } peers { permanent_uuid: "3c1770123dc14bba80e00db5f182b585" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 44326 } } } W0703 01:43:20.287861 31326 consensus_peers.cc:433] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 3c1770123dc14bba80e00db5f182b585 (127.5.240.197:44326): Couldn't send request to peer 3c1770123dc14bba80e00db5f182b585 for tablet 1f01bfdde47b4d37a0f77d199bc7b4d2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1f01bfdde47b4d37a0f77d199bc7b4d2. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:43:20.429493 31393 consensus_peers.cc:455] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521): Closing peer: 1288ca0989104bbea36e22f38c374218 I0703 01:43:20.429759 31393 consensus_queue.cc:204] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 663, Committed index: 663, Last appended: 1.663, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 664 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } W0703 01:43:20.432241 31326 consensus_peers.cc:433] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 768d8de4b2ba40f180b18194c28be589 (127.5.240.198:42654): Couldn't send request to peer 768d8de4b2ba40f180b18194c28be589 for tablet f9f1c40150da4144acb68186f9aad763. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f9f1c40150da4144acb68186f9aad763. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:43:20.435083 31271 raft_consensus.cc:1081] T f9f1c40150da4144acb68186f9aad763 P 1288ca0989104bbea36e22f38c374218 [term 1 FOLLOWER]: Refusing update from remote peer 2f3fdb7c089640c0b936a574bc64707e: Log matching property violated. Preceding OpId in replica: term: 1 index: 663. Preceding OpId from leader: term: 1 index: 664. (index mismatch) I0703 01:43:20.440438 1150 consensus_queue.cc:962] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 664, Last known committed idx: 663, Time since last communication: 0.000s I0703 01:43:20.445004 1150 raft_consensus.cc:2684] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e [term 1 LEADER]: Committing config change with OpId 1.664: config changed from index -1 to 664, VOTER 768d8de4b2ba40f180b18194c28be589 (127.5.240.198) added. New config: { opid_index: 664 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } } I0703 01:43:20.452002 31271 raft_consensus.cc:2684] T f9f1c40150da4144acb68186f9aad763 P 1288ca0989104bbea36e22f38c374218 [term 1 FOLLOWER]: Committing config change with OpId 1.664: config changed from index -1 to 664, VOTER 768d8de4b2ba40f180b18194c28be589 (127.5.240.198) added. New config: { opid_index: 664 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } } I0703 01:43:20.456676 31149 catalog_manager.cc:3798] T f9f1c40150da4144acb68186f9aad763 P 1288ca0989104bbea36e22f38c374218 reported cstate change: config changed from index -1 to 664, VOTER 768d8de4b2ba40f180b18194c28be589 (127.5.240.198) added. New cstate: current_term: 1 leader_uuid: "2f3fdb7c089640c0b936a574bc64707e" committed_config { opid_index: 664 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } } I0703 01:43:20.601680 2318 ts_tablet_manager.cc:669] T a200f8a763f14d54a1e84082dbafce8f P d9dc68ace903455d809cb1f715fe36be: Initiating tablet copy from peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521) I0703 01:43:20.601841 2318 tablet_copy_client.cc:239] T a200f8a763f14d54a1e84082dbafce8f P d9dc68ace903455d809cb1f715fe36be: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.193:33521 I0703 01:43:20.602562 31292 tablet_copy_service.cc:135] P 1288ca0989104bbea36e22f38c374218: Received BeginTabletCopySession request for tablet a200f8a763f14d54a1e84082dbafce8f from peer d9dc68ace903455d809cb1f715fe36be ({username='jenkins'} at 127.5.240.196:46328) I0703 01:43:20.602665 31292 tablet_copy_service.cc:156] P 1288ca0989104bbea36e22f38c374218: Beginning new tablet copy session on tablet a200f8a763f14d54a1e84082dbafce8f from peer d9dc68ace903455d809cb1f715fe36be at {username='jenkins'} at 127.5.240.196:46328: session id = d9dc68ace903455d809cb1f715fe36be-a200f8a763f14d54a1e84082dbafce8f I0703 01:43:20.604960 31292 tablet_copy_source_session.cc:202] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:43:20.606643 2318 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet a200f8a763f14d54a1e84082dbafce8f. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:43:20.613696 2318 ts_tablet_manager.cc:1173] T a200f8a763f14d54a1e84082dbafce8f P d9dc68ace903455d809cb1f715fe36be: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:43:20.620029 2318 tablet_copy_client.cc:566] T a200f8a763f14d54a1e84082dbafce8f P d9dc68ace903455d809cb1f715fe36be: tablet copy: Starting download of 0 data blocks... I0703 01:43:20.620260 2318 tablet_copy_client.cc:529] T a200f8a763f14d54a1e84082dbafce8f P d9dc68ace903455d809cb1f715fe36be: tablet copy: Starting download of 1 WAL segments... I0703 01:43:20.633026 2318 tablet_copy_client.cc:405] T a200f8a763f14d54a1e84082dbafce8f P d9dc68ace903455d809cb1f715fe36be: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:43:20.635699 2318 ts_tablet_manager.cc:1028] T a200f8a763f14d54a1e84082dbafce8f P d9dc68ace903455d809cb1f715fe36be: Bootstrapping tablet I0703 01:43:20.635818 2318 tablet_bootstrap.cc:427] T a200f8a763f14d54a1e84082dbafce8f P d9dc68ace903455d809cb1f715fe36be: Bootstrap starting. I0703 01:43:20.683043 32503 consensus_peers.cc:455] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be -> Peer de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321): Closing peer: de93b58c2d67488dae12285e1806ebac I0703 01:43:20.683291 32503 consensus_queue.cc:204] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 703, Committed index: 703, Last appended: 2.703, Last appended by leader: 459, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 704 OBSOLETE_local: false peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } peers { permanent_uuid: "3c1770123dc14bba80e00db5f182b585" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 44326 } } I0703 01:43:20.686466 31517 raft_consensus.cc:1081] T ffe66a5cfb564964951c619e51e7c47c P de93b58c2d67488dae12285e1806ebac [term 2 FOLLOWER]: Refusing update from remote peer d9dc68ace903455d809cb1f715fe36be: Log matching property violated. Preceding OpId in replica: term: 2 index: 703. Preceding OpId from leader: term: 2 index: 704. (index mismatch) I0703 01:43:20.688433 2014 consensus_queue.cc:962] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be [LEADER]: Connected to new peer: Peer: permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 704, Last known committed idx: 703, Time since last communication: 0.000s I0703 01:43:20.691454 2248 raft_consensus.cc:2684] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be [term 2 LEADER]: Committing config change with OpId 2.704: config changed from index 611 to 704, VOTER 3c1770123dc14bba80e00db5f182b585 (127.5.240.197) added. New config: { opid_index: 704 OBSOLETE_local: false peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } peers { permanent_uuid: "3c1770123dc14bba80e00db5f182b585" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 44326 } } } I0703 01:43:20.695152 31517 raft_consensus.cc:2684] T ffe66a5cfb564964951c619e51e7c47c P de93b58c2d67488dae12285e1806ebac [term 2 FOLLOWER]: Committing config change with OpId 2.704: config changed from index 611 to 704, VOTER 3c1770123dc14bba80e00db5f182b585 (127.5.240.197) added. New config: { opid_index: 704 OBSOLETE_local: false peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } peers { permanent_uuid: "3c1770123dc14bba80e00db5f182b585" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 44326 } } } I0703 01:43:20.706200 31147 catalog_manager.cc:3798] T ffe66a5cfb564964951c619e51e7c47c P de93b58c2d67488dae12285e1806ebac reported cstate change: config changed from index 611 to 704, VOTER 3c1770123dc14bba80e00db5f182b585 (127.5.240.197) added. New cstate: current_term: 2 leader_uuid: "d9dc68ace903455d809cb1f715fe36be" committed_config { opid_index: 704 OBSOLETE_local: false peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } peers { permanent_uuid: "3c1770123dc14bba80e00db5f182b585" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 44326 } } } W0703 01:43:20.758955 32432 consensus_peers.cc:433] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be -> Peer 3c1770123dc14bba80e00db5f182b585 (127.5.240.197:44326): Couldn't send request to peer 3c1770123dc14bba80e00db5f182b585 for tablet ffe66a5cfb564964951c619e51e7c47c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ffe66a5cfb564964951c619e51e7c47c. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:43:20.818891 2333 ts_tablet_manager.cc:669] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 3c1770123dc14bba80e00db5f182b585: Initiating tablet copy from peer 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194:39827) I0703 01:43:20.819145 2333 tablet_copy_client.cc:239] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 3c1770123dc14bba80e00db5f182b585: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.194:39827 I0703 01:43:20.820638 31415 tablet_copy_service.cc:135] P 2f3fdb7c089640c0b936a574bc64707e: Received BeginTabletCopySession request for tablet 1f01bfdde47b4d37a0f77d199bc7b4d2 from peer 3c1770123dc14bba80e00db5f182b585 ({username='jenkins'} at 127.5.240.197:38739) I0703 01:43:20.820791 31415 tablet_copy_service.cc:156] P 2f3fdb7c089640c0b936a574bc64707e: Beginning new tablet copy session on tablet 1f01bfdde47b4d37a0f77d199bc7b4d2 from peer 3c1770123dc14bba80e00db5f182b585 at {username='jenkins'} at 127.5.240.197:38739: session id = 3c1770123dc14bba80e00db5f182b585-1f01bfdde47b4d37a0f77d199bc7b4d2 I0703 01:43:20.823287 31415 tablet_copy_source_session.cc:202] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:43:20.825662 2333 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 1f01bfdde47b4d37a0f77d199bc7b4d2. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:43:20.838716 2333 ts_tablet_manager.cc:1173] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 3c1770123dc14bba80e00db5f182b585: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:43:20.840651 2333 tablet_copy_client.cc:566] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 3c1770123dc14bba80e00db5f182b585: tablet copy: Starting download of 0 data blocks... I0703 01:43:20.840869 2333 tablet_copy_client.cc:529] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 3c1770123dc14bba80e00db5f182b585: tablet copy: Starting download of 1 WAL segments... I0703 01:43:20.851821 2333 tablet_copy_client.cc:405] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 3c1770123dc14bba80e00db5f182b585: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:43:20.857141 2333 ts_tablet_manager.cc:1028] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 3c1770123dc14bba80e00db5f182b585: Bootstrapping tablet I0703 01:43:20.857511 2333 tablet_bootstrap.cc:427] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 3c1770123dc14bba80e00db5f182b585: Bootstrap starting. I0703 01:43:20.940862 2343 ts_tablet_manager.cc:669] T f9f1c40150da4144acb68186f9aad763 P 768d8de4b2ba40f180b18194c28be589: Initiating tablet copy from peer 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194:39827) I0703 01:43:20.941048 2343 tablet_copy_client.cc:239] T f9f1c40150da4144acb68186f9aad763 P 768d8de4b2ba40f180b18194c28be589: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.194:39827 I0703 01:43:20.942171 31415 tablet_copy_service.cc:135] P 2f3fdb7c089640c0b936a574bc64707e: Received BeginTabletCopySession request for tablet f9f1c40150da4144acb68186f9aad763 from peer 768d8de4b2ba40f180b18194c28be589 ({username='jenkins'} at 127.5.240.198:59612) I0703 01:43:20.942312 31415 tablet_copy_service.cc:156] P 2f3fdb7c089640c0b936a574bc64707e: Beginning new tablet copy session on tablet f9f1c40150da4144acb68186f9aad763 from peer 768d8de4b2ba40f180b18194c28be589 at {username='jenkins'} at 127.5.240.198:59612: session id = 768d8de4b2ba40f180b18194c28be589-f9f1c40150da4144acb68186f9aad763 I0703 01:43:20.944664 31415 tablet_copy_source_session.cc:202] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:43:20.946131 2343 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet f9f1c40150da4144acb68186f9aad763. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:43:20.953336 2343 ts_tablet_manager.cc:1173] T f9f1c40150da4144acb68186f9aad763 P 768d8de4b2ba40f180b18194c28be589: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:43:20.955061 2343 tablet_copy_client.cc:566] T f9f1c40150da4144acb68186f9aad763 P 768d8de4b2ba40f180b18194c28be589: tablet copy: Starting download of 0 data blocks... I0703 01:43:20.956127 2343 tablet_copy_client.cc:529] T f9f1c40150da4144acb68186f9aad763 P 768d8de4b2ba40f180b18194c28be589: tablet copy: Starting download of 1 WAL segments... I0703 01:43:20.979806 2343 tablet_copy_client.cc:405] T f9f1c40150da4144acb68186f9aad763 P 768d8de4b2ba40f180b18194c28be589: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:43:20.986016 2343 ts_tablet_manager.cc:1028] T f9f1c40150da4144acb68186f9aad763 P 768d8de4b2ba40f180b18194c28be589: Bootstrapping tablet I0703 01:43:20.986397 2343 tablet_bootstrap.cc:427] T f9f1c40150da4144acb68186f9aad763 P 768d8de4b2ba40f180b18194c28be589: Bootstrap starting. W0703 01:43:21.111868 31205 consensus_peers.cc:433] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218 -> Peer d9dc68ace903455d809cb1f715fe36be (127.5.240.196:56543): Couldn't send request to peer d9dc68ace903455d809cb1f715fe36be for tablet a200f8a763f14d54a1e84082dbafce8f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 2 times. I0703 01:43:21.286783 2364 ts_tablet_manager.cc:669] T ffe66a5cfb564964951c619e51e7c47c P 3c1770123dc14bba80e00db5f182b585: Initiating tablet copy from peer d9dc68ace903455d809cb1f715fe36be (127.5.240.196:56543) I0703 01:43:21.288506 2364 tablet_copy_client.cc:239] T ffe66a5cfb564964951c619e51e7c47c P 3c1770123dc14bba80e00db5f182b585: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.196:56543 W0703 01:43:21.350746 31326 consensus_peers.cc:433] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 3c1770123dc14bba80e00db5f182b585 (127.5.240.197:44326): Couldn't send request to peer 3c1770123dc14bba80e00db5f182b585 for tablet 1f01bfdde47b4d37a0f77d199bc7b4d2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 2 times. I0703 01:43:21.387985 32525 tablet_copy_service.cc:135] P d9dc68ace903455d809cb1f715fe36be: Received BeginTabletCopySession request for tablet ffe66a5cfb564964951c619e51e7c47c from peer 3c1770123dc14bba80e00db5f182b585 ({username='jenkins'} at 127.5.240.197:60550) I0703 01:43:21.388181 32525 tablet_copy_service.cc:156] P d9dc68ace903455d809cb1f715fe36be: Beginning new tablet copy session on tablet ffe66a5cfb564964951c619e51e7c47c from peer 3c1770123dc14bba80e00db5f182b585 at {username='jenkins'} at 127.5.240.197:60550: session id = 3c1770123dc14bba80e00db5f182b585-ffe66a5cfb564964951c619e51e7c47c I0703 01:43:21.391049 32525 tablet_copy_source_session.cc:202] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:43:21.394142 2364 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet ffe66a5cfb564964951c619e51e7c47c. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:43:21.409118 2364 ts_tablet_manager.cc:1173] T ffe66a5cfb564964951c619e51e7c47c P 3c1770123dc14bba80e00db5f182b585: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:43:21.409684 2364 tablet_copy_client.cc:566] T ffe66a5cfb564964951c619e51e7c47c P 3c1770123dc14bba80e00db5f182b585: tablet copy: Starting download of 0 data blocks... I0703 01:43:21.410347 2364 tablet_copy_client.cc:529] T ffe66a5cfb564964951c619e51e7c47c P 3c1770123dc14bba80e00db5f182b585: tablet copy: Starting download of 1 WAL segments... W0703 01:43:21.460647 31326 consensus_peers.cc:433] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 768d8de4b2ba40f180b18194c28be589 (127.5.240.198:42654): Couldn't send request to peer 768d8de4b2ba40f180b18194c28be589 for tablet f9f1c40150da4144acb68186f9aad763. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 2 times. I0703 01:43:21.583529 2364 tablet_copy_client.cc:405] T ffe66a5cfb564964951c619e51e7c47c P 3c1770123dc14bba80e00db5f182b585: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:43:21.586025 2364 ts_tablet_manager.cc:1028] T ffe66a5cfb564964951c619e51e7c47c P 3c1770123dc14bba80e00db5f182b585: Bootstrapping tablet I0703 01:43:21.586151 2364 tablet_bootstrap.cc:427] T ffe66a5cfb564964951c619e51e7c47c P 3c1770123dc14bba80e00db5f182b585: Bootstrap starting. W0703 01:43:21.602571 31205 consensus_peers.cc:433] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218 -> Peer d9dc68ace903455d809cb1f715fe36be (127.5.240.196:56543): Couldn't send request to peer d9dc68ace903455d809cb1f715fe36be for tablet a200f8a763f14d54a1e84082dbafce8f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 3 times. W0703 01:43:21.767864 31326 consensus_peers.cc:433] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 3c1770123dc14bba80e00db5f182b585 (127.5.240.197:44326): Couldn't send request to peer 3c1770123dc14bba80e00db5f182b585 for tablet 1f01bfdde47b4d37a0f77d199bc7b4d2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 3 times. W0703 01:43:21.800024 32432 consensus_peers.cc:433] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be -> Peer 3c1770123dc14bba80e00db5f182b585 (127.5.240.197:44326): Couldn't send request to peer 3c1770123dc14bba80e00db5f182b585 for tablet ffe66a5cfb564964951c619e51e7c47c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 2 times. W0703 01:43:21.952057 31326 consensus_peers.cc:433] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 768d8de4b2ba40f180b18194c28be589 (127.5.240.198:42654): Couldn't send request to peer 768d8de4b2ba40f180b18194c28be589 for tablet f9f1c40150da4144acb68186f9aad763. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 3 times. W0703 01:43:21.981134 31205 consensus_peers.cc:433] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218 -> Peer d9dc68ace903455d809cb1f715fe36be (127.5.240.196:56543): Couldn't send request to peer d9dc68ace903455d809cb1f715fe36be for tablet a200f8a763f14d54a1e84082dbafce8f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 4 times. W0703 01:43:22.233245 32432 consensus_peers.cc:433] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be -> Peer 3c1770123dc14bba80e00db5f182b585 (127.5.240.197:44326): Couldn't send request to peer 3c1770123dc14bba80e00db5f182b585 for tablet ffe66a5cfb564964951c619e51e7c47c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 3 times. W0703 01:43:22.338783 31326 consensus_peers.cc:433] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 3c1770123dc14bba80e00db5f182b585 (127.5.240.197:44326): Couldn't send request to peer 3c1770123dc14bba80e00db5f182b585 for tablet 1f01bfdde47b4d37a0f77d199bc7b4d2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 4 times. W0703 01:43:22.418557 31205 consensus_peers.cc:433] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218 -> Peer d9dc68ace903455d809cb1f715fe36be (127.5.240.196:56543): Couldn't send request to peer d9dc68ace903455d809cb1f715fe36be for tablet a200f8a763f14d54a1e84082dbafce8f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 5 times. W0703 01:43:22.437532 31326 consensus_peers.cc:433] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 768d8de4b2ba40f180b18194c28be589 (127.5.240.198:42654): Couldn't send request to peer 768d8de4b2ba40f180b18194c28be589 for tablet f9f1c40150da4144acb68186f9aad763. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 4 times. W0703 01:43:22.633072 32432 consensus_peers.cc:433] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be -> Peer 3c1770123dc14bba80e00db5f182b585 (127.5.240.197:44326): Couldn't send request to peer 3c1770123dc14bba80e00db5f182b585 for tablet ffe66a5cfb564964951c619e51e7c47c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 4 times. W0703 01:43:22.737051 31326 consensus_peers.cc:433] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 3c1770123dc14bba80e00db5f182b585 (127.5.240.197:44326): Couldn't send request to peer 3c1770123dc14bba80e00db5f182b585 for tablet 1f01bfdde47b4d37a0f77d199bc7b4d2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 5 times. W0703 01:43:22.868269 31205 consensus_peers.cc:433] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218 -> Peer d9dc68ace903455d809cb1f715fe36be (127.5.240.196:56543): Couldn't send request to peer d9dc68ace903455d809cb1f715fe36be for tablet a200f8a763f14d54a1e84082dbafce8f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 6 times. W0703 01:43:23.029089 31326 consensus_peers.cc:433] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 768d8de4b2ba40f180b18194c28be589 (127.5.240.198:42654): Couldn't send request to peer 768d8de4b2ba40f180b18194c28be589 for tablet f9f1c40150da4144acb68186f9aad763. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 5 times. W0703 01:43:23.130241 31326 consensus_peers.cc:433] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 3c1770123dc14bba80e00db5f182b585 (127.5.240.197:44326): Couldn't send request to peer 3c1770123dc14bba80e00db5f182b585 for tablet 1f01bfdde47b4d37a0f77d199bc7b4d2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 6 times. W0703 01:43:23.245272 32432 consensus_peers.cc:433] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be -> Peer 3c1770123dc14bba80e00db5f182b585 (127.5.240.197:44326): Couldn't send request to peer 3c1770123dc14bba80e00db5f182b585 for tablet ffe66a5cfb564964951c619e51e7c47c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 5 times. W0703 01:43:23.446563 31205 consensus_peers.cc:433] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218 -> Peer d9dc68ace903455d809cb1f715fe36be (127.5.240.196:56543): Couldn't send request to peer d9dc68ace903455d809cb1f715fe36be for tablet a200f8a763f14d54a1e84082dbafce8f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 7 times. W0703 01:43:23.552549 31326 consensus_peers.cc:433] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 768d8de4b2ba40f180b18194c28be589 (127.5.240.198:42654): Couldn't send request to peer 768d8de4b2ba40f180b18194c28be589 for tablet f9f1c40150da4144acb68186f9aad763. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 6 times. W0703 01:43:23.561539 31326 consensus_peers.cc:433] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 3c1770123dc14bba80e00db5f182b585 (127.5.240.197:44326): Couldn't send request to peer 3c1770123dc14bba80e00db5f182b585 for tablet 1f01bfdde47b4d37a0f77d199bc7b4d2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 7 times. W0703 01:43:23.693544 32432 consensus_peers.cc:433] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be -> Peer 3c1770123dc14bba80e00db5f182b585 (127.5.240.197:44326): Couldn't send request to peer 3c1770123dc14bba80e00db5f182b585 for tablet ffe66a5cfb564964951c619e51e7c47c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 6 times. W0703 01:43:24.063745 31205 consensus_peers.cc:433] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218 -> Peer d9dc68ace903455d809cb1f715fe36be (127.5.240.196:56543): Couldn't send request to peer d9dc68ace903455d809cb1f715fe36be for tablet a200f8a763f14d54a1e84082dbafce8f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 8 times. W0703 01:43:24.099249 32432 consensus_peers.cc:433] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be -> Peer 3c1770123dc14bba80e00db5f182b585 (127.5.240.197:44326): Couldn't send request to peer 3c1770123dc14bba80e00db5f182b585 for tablet ffe66a5cfb564964951c619e51e7c47c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 7 times. W0703 01:43:24.310653 31326 consensus_peers.cc:433] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 768d8de4b2ba40f180b18194c28be589 (127.5.240.198:42654): Couldn't send request to peer 768d8de4b2ba40f180b18194c28be589 for tablet f9f1c40150da4144acb68186f9aad763. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 7 times. W0703 01:43:24.318660 31326 consensus_peers.cc:433] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 3c1770123dc14bba80e00db5f182b585 (127.5.240.197:44326): Couldn't send request to peer 3c1770123dc14bba80e00db5f182b585 for tablet 1f01bfdde47b4d37a0f77d199bc7b4d2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 8 times. I0703 01:43:24.458654 2318 tablet_bootstrap.cc:427] T a200f8a763f14d54a1e84082dbafce8f P d9dc68ace903455d809cb1f715fe36be: Bootstrap replayed 1/1 log segments. Stats: ops{read=526 overwritten=0 applied=526 ignored=0} inserts{seen=1176 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:43:24.462174 2318 tablet_bootstrap.cc:427] T a200f8a763f14d54a1e84082dbafce8f P d9dc68ace903455d809cb1f715fe36be: Bootstrap complete. I0703 01:43:24.462456 2318 ts_tablet_manager.cc:1045] T a200f8a763f14d54a1e84082dbafce8f P d9dc68ace903455d809cb1f715fe36be: Time spent bootstrapping tablet: real 3.827s user 0.595s sys 0.057s I0703 01:43:24.466137 2318 raft_consensus.cc:325] T a200f8a763f14d54a1e84082dbafce8f P d9dc68ace903455d809cb1f715fe36be [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 517 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } I0703 01:43:24.466410 2318 raft_consensus.cc:628] T a200f8a763f14d54a1e84082dbafce8f P d9dc68ace903455d809cb1f715fe36be [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9dc68ace903455d809cb1f715fe36be, State: Initialized, Role: FOLLOWER I0703 01:43:24.466630 2318 consensus_queue.cc:227] T a200f8a763f14d54a1e84082dbafce8f P d9dc68ace903455d809cb1f715fe36be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 526, Last appended: 1.526, Last appended by leader: 526, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 517 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } I0703 01:43:24.471438 2318 ts_tablet_manager.cc:1073] T a200f8a763f14d54a1e84082dbafce8f P d9dc68ace903455d809cb1f715fe36be: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s I0703 01:43:24.472354 31292 tablet_copy_service.cc:325] P 1288ca0989104bbea36e22f38c374218: Request end of tablet copy session d9dc68ace903455d809cb1f715fe36be-a200f8a763f14d54a1e84082dbafce8f received from {username='jenkins'} at 127.5.240.196:46328 I0703 01:43:24.472424 31292 tablet_copy_service.cc:413] P 1288ca0989104bbea36e22f38c374218: ending tablet copy session d9dc68ace903455d809cb1f715fe36be-a200f8a763f14d54a1e84082dbafce8f on tablet a200f8a763f14d54a1e84082dbafce8f with peer d9dc68ace903455d809cb1f715fe36be W0703 01:43:24.504762 32432 consensus_peers.cc:433] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be -> Peer 3c1770123dc14bba80e00db5f182b585 (127.5.240.197:44326): Couldn't send request to peer 3c1770123dc14bba80e00db5f182b585 for tablet ffe66a5cfb564964951c619e51e7c47c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 8 times. I0703 01:43:24.532245 32503 raft_consensus.cc:1026] T a200f8a763f14d54a1e84082dbafce8f P d9dc68ace903455d809cb1f715fe36be [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.516->[1.517-1.593] Dedup: 1.526->[1.527-1.593] W0703 01:43:24.652199 31326 consensus_peers.cc:433] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 768d8de4b2ba40f180b18194c28be589 (127.5.240.198:42654): Couldn't send request to peer 768d8de4b2ba40f180b18194c28be589 for tablet f9f1c40150da4144acb68186f9aad763. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 8 times. W0703 01:43:24.714808 31326 consensus_peers.cc:433] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 3c1770123dc14bba80e00db5f182b585 (127.5.240.197:44326): Couldn't send request to peer 3c1770123dc14bba80e00db5f182b585 for tablet 1f01bfdde47b4d37a0f77d199bc7b4d2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 9 times. W0703 01:43:24.928027 32432 consensus_peers.cc:433] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be -> Peer 3c1770123dc14bba80e00db5f182b585 (127.5.240.197:44326): Couldn't send request to peer 3c1770123dc14bba80e00db5f182b585 for tablet ffe66a5cfb564964951c619e51e7c47c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 9 times. W0703 01:43:25.124534 31326 consensus_peers.cc:433] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 3c1770123dc14bba80e00db5f182b585 (127.5.240.197:44326): Couldn't send request to peer 3c1770123dc14bba80e00db5f182b585 for tablet 1f01bfdde47b4d37a0f77d199bc7b4d2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 10 times. W0703 01:43:25.164008 31326 consensus_peers.cc:433] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 768d8de4b2ba40f180b18194c28be589 (127.5.240.198:42654): Couldn't send request to peer 768d8de4b2ba40f180b18194c28be589 for tablet f9f1c40150da4144acb68186f9aad763. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 9 times. W0703 01:43:25.485815 32432 consensus_peers.cc:433] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be -> Peer 3c1770123dc14bba80e00db5f182b585 (127.5.240.197:44326): Couldn't send request to peer 3c1770123dc14bba80e00db5f182b585 for tablet ffe66a5cfb564964951c619e51e7c47c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 10 times. I0703 01:43:25.515018 31268 tablet_service.cc:1140] Received LeaderStepDown RPC: tablet_id: "a200f8a763f14d54a1e84082dbafce8f" dest_uuid: "1288ca0989104bbea36e22f38c374218" from {username='jenkins'} at 127.5.240.193:33797 I0703 01:43:25.515158 31268 raft_consensus.cc:552] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218 [term 1 LEADER]: Received request to step down I0703 01:43:25.515202 31268 raft_consensus.cc:628] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1288ca0989104bbea36e22f38c374218, State: Running, Role: LEADER I0703 01:43:25.515434 31268 consensus_queue.cc:227] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 613, Committed index: 613, Last appended: 1.613, Last appended by leader: 613, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 517 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } I0703 01:43:25.515703 31268 consensus_peers.cc:455] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218 -> Peer d9dc68ace903455d809cb1f715fe36be (127.5.240.196:56543): Closing peer: d9dc68ace903455d809cb1f715fe36be I0703 01:43:25.515786 31268 consensus_peers.cc:455] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218 -> Peer 768d8de4b2ba40f180b18194c28be589 (127.5.240.198:42654): Closing peer: 768d8de4b2ba40f180b18194c28be589 I0703 01:43:25.515980 31268 raft_consensus.cc:2730] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218: Snoozing failure detection for 3.000s (explicit stepdown request) W0703 01:43:25.601323 31326 consensus_peers.cc:433] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 768d8de4b2ba40f180b18194c28be589 (127.5.240.198:42654): Couldn't send request to peer 768d8de4b2ba40f180b18194c28be589 for tablet f9f1c40150da4144acb68186f9aad763. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 10 times. W0703 01:43:25.701933 31326 consensus_peers.cc:433] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 3c1770123dc14bba80e00db5f182b585 (127.5.240.197:44326): Couldn't send request to peer 3c1770123dc14bba80e00db5f182b585 for tablet 1f01bfdde47b4d37a0f77d199bc7b4d2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 11 times. W0703 01:43:26.059159 32432 consensus_peers.cc:433] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be -> Peer 3c1770123dc14bba80e00db5f182b585 (127.5.240.197:44326): Couldn't send request to peer 3c1770123dc14bba80e00db5f182b585 for tablet ffe66a5cfb564964951c619e51e7c47c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 11 times. W0703 01:43:26.127605 31326 consensus_peers.cc:433] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 3c1770123dc14bba80e00db5f182b585 (127.5.240.197:44326): Couldn't send request to peer 3c1770123dc14bba80e00db5f182b585 for tablet 1f01bfdde47b4d37a0f77d199bc7b4d2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 12 times. W0703 01:43:26.203119 31326 consensus_peers.cc:433] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 768d8de4b2ba40f180b18194c28be589 (127.5.240.198:42654): Couldn't send request to peer 768d8de4b2ba40f180b18194c28be589 for tablet f9f1c40150da4144acb68186f9aad763. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 11 times. I0703 01:43:26.399060 2343 tablet_bootstrap.cc:427] T f9f1c40150da4144acb68186f9aad763 P 768d8de4b2ba40f180b18194c28be589: Bootstrap replayed 1/1 log segments. Stats: ops{read=671 overwritten=0 applied=671 ignored=0} inserts{seen=1320 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:43:26.399480 2343 tablet_bootstrap.cc:427] T f9f1c40150da4144acb68186f9aad763 P 768d8de4b2ba40f180b18194c28be589: Bootstrap complete. I0703 01:43:26.399773 2343 ts_tablet_manager.cc:1045] T f9f1c40150da4144acb68186f9aad763 P 768d8de4b2ba40f180b18194c28be589: Time spent bootstrapping tablet: real 5.413s user 0.752s sys 0.058s I0703 01:43:26.406692 2343 raft_consensus.cc:325] T f9f1c40150da4144acb68186f9aad763 P 768d8de4b2ba40f180b18194c28be589 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 664 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } I0703 01:43:26.410177 2343 raft_consensus.cc:628] T f9f1c40150da4144acb68186f9aad763 P 768d8de4b2ba40f180b18194c28be589 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768d8de4b2ba40f180b18194c28be589, State: Initialized, Role: FOLLOWER I0703 01:43:26.410401 2343 consensus_queue.cc:227] T f9f1c40150da4144acb68186f9aad763 P 768d8de4b2ba40f180b18194c28be589 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 671, Last appended: 1.671, Last appended by leader: 671, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 664 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } I0703 01:43:26.431995 2343 ts_tablet_manager.cc:1073] T f9f1c40150da4144acb68186f9aad763 P 768d8de4b2ba40f180b18194c28be589: Time spent starting tablet: real 0.032s user 0.002s sys 0.005s I0703 01:43:26.433082 31415 tablet_copy_service.cc:325] P 2f3fdb7c089640c0b936a574bc64707e: Request end of tablet copy session 768d8de4b2ba40f180b18194c28be589-f9f1c40150da4144acb68186f9aad763 received from {username='jenkins'} at 127.5.240.198:59612 I0703 01:43:26.433152 31415 tablet_copy_service.cc:413] P 2f3fdb7c089640c0b936a574bc64707e: ending tablet copy session 768d8de4b2ba40f180b18194c28be589-f9f1c40150da4144acb68186f9aad763 on tablet f9f1c40150da4144acb68186f9aad763 with peer 768d8de4b2ba40f180b18194c28be589 W0703 01:43:26.639050 32432 consensus_peers.cc:433] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be -> Peer 3c1770123dc14bba80e00db5f182b585 (127.5.240.197:44326): Couldn't send request to peer 3c1770123dc14bba80e00db5f182b585 for tablet ffe66a5cfb564964951c619e51e7c47c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 12 times. W0703 01:43:26.639822 31326 consensus_peers.cc:433] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 3c1770123dc14bba80e00db5f182b585 (127.5.240.197:44326): Couldn't send request to peer 3c1770123dc14bba80e00db5f182b585 for tablet 1f01bfdde47b4d37a0f77d199bc7b4d2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 13 times. I0703 01:43:26.772061 32767 raft_consensus.cc:1026] T f9f1c40150da4144acb68186f9aad763 P 768d8de4b2ba40f180b18194c28be589 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.663->[1.664-1.786] Dedup: 1.671->[1.672-1.786] I0703 01:43:26.797495 2333 tablet_bootstrap.cc:427] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 3c1770123dc14bba80e00db5f182b585: Bootstrap replaying log segment 1/1 (213.2K/214.0K this segment, stats: ops{read=757 overwritten=0 applied=757 ignored=0} inserts{seen=1442 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0) I0703 01:43:26.814416 2333 tablet_bootstrap.cc:427] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 3c1770123dc14bba80e00db5f182b585: Bootstrap replayed 1/1 log segments. Stats: ops{read=760 overwritten=0 applied=760 ignored=0} inserts{seen=1445 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:43:26.814859 2333 tablet_bootstrap.cc:427] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 3c1770123dc14bba80e00db5f182b585: Bootstrap complete. I0703 01:43:26.815121 2333 ts_tablet_manager.cc:1045] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 3c1770123dc14bba80e00db5f182b585: Time spent bootstrapping tablet: real 5.958s user 0.842s sys 0.076s I0703 01:43:26.831291 2333 raft_consensus.cc:325] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 3c1770123dc14bba80e00db5f182b585 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 746 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } peers { permanent_uuid: "3c1770123dc14bba80e00db5f182b585" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 44326 } } I0703 01:43:26.831550 2333 raft_consensus.cc:628] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 3c1770123dc14bba80e00db5f182b585 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c1770123dc14bba80e00db5f182b585, State: Initialized, Role: FOLLOWER I0703 01:43:26.831778 2333 consensus_queue.cc:227] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 3c1770123dc14bba80e00db5f182b585 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 760, Last appended: 1.760, Last appended by leader: 760, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 746 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } peers { permanent_uuid: "3c1770123dc14bba80e00db5f182b585" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 44326 } } I0703 01:43:26.854792 2333 ts_tablet_manager.cc:1073] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 3c1770123dc14bba80e00db5f182b585: Time spent starting tablet: real 0.038s user 0.001s sys 0.003s I0703 01:43:26.855576 31415 tablet_copy_service.cc:325] P 2f3fdb7c089640c0b936a574bc64707e: Request end of tablet copy session 3c1770123dc14bba80e00db5f182b585-1f01bfdde47b4d37a0f77d199bc7b4d2 received from {username='jenkins'} at 127.5.240.197:38739 I0703 01:43:26.855667 31415 tablet_copy_service.cc:413] P 2f3fdb7c089640c0b936a574bc64707e: ending tablet copy session 3c1770123dc14bba80e00db5f182b585-1f01bfdde47b4d37a0f77d199bc7b4d2 on tablet 1f01bfdde47b4d37a0f77d199bc7b4d2 with peer 3c1770123dc14bba80e00db5f182b585 I0703 01:43:27.004894 2648 raft_consensus.cc:457] T a200f8a763f14d54a1e84082dbafce8f P 768d8de4b2ba40f180b18194c28be589 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1288ca0989104bbea36e22f38c374218) I0703 01:43:27.005072 2648 raft_consensus.cc:2730] T a200f8a763f14d54a1e84082dbafce8f P 768d8de4b2ba40f180b18194c28be589: Snoozing failure detection for 2.100s (starting election) I0703 01:43:27.005177 2648 raft_consensus.cc:479] T a200f8a763f14d54a1e84082dbafce8f P 768d8de4b2ba40f180b18194c28be589 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 517 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } I0703 01:43:27.006122 2648 leader_election.cc:243] T a200f8a763f14d54a1e84082dbafce8f P 768d8de4b2ba40f180b18194c28be589 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521) I0703 01:43:27.006443 2648 leader_election.cc:243] T a200f8a763f14d54a1e84082dbafce8f P 768d8de4b2ba40f180b18194c28be589 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer d9dc68ace903455d809cb1f715fe36be (127.5.240.196:56543) I0703 01:43:27.008524 31272 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "a200f8a763f14d54a1e84082dbafce8f" candidate_uuid: "768d8de4b2ba40f180b18194c28be589" candidate_term: 2 candidate_status { last_received { term: 1 index: 613 } } ignore_live_leader: false dest_uuid: "1288ca0989104bbea36e22f38c374218" is_pre_election: true I0703 01:43:27.008721 31272 raft_consensus.cc:2730] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218: Snoozing failure detection for 1.589s (vote granted) I0703 01:43:27.009166 31272 raft_consensus.cc:2243] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768d8de4b2ba40f180b18194c28be589 in term 1. I0703 01:43:27.010314 32697 leader_election.cc:379] T a200f8a763f14d54a1e84082dbafce8f P 768d8de4b2ba40f180b18194c28be589 [CANDIDATE]: Term 2 pre-election: Vote granted by peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521) I0703 01:43:27.010419 32697 leader_election.cc:270] T a200f8a763f14d54a1e84082dbafce8f P 768d8de4b2ba40f180b18194c28be589 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. I0703 01:43:27.010798 2659 raft_consensus.cc:457] T a200f8a763f14d54a1e84082dbafce8f P d9dc68ace903455d809cb1f715fe36be [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1288ca0989104bbea36e22f38c374218) I0703 01:43:27.010890 2659 raft_consensus.cc:2730] T a200f8a763f14d54a1e84082dbafce8f P d9dc68ace903455d809cb1f715fe36be: Snoozing failure detection for 2.012s (starting election) I0703 01:43:27.010969 2659 raft_consensus.cc:479] T a200f8a763f14d54a1e84082dbafce8f P d9dc68ace903455d809cb1f715fe36be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 517 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } I0703 01:43:27.011306 2648 raft_consensus.cc:2730] T a200f8a763f14d54a1e84082dbafce8f P 768d8de4b2ba40f180b18194c28be589: Snoozing failure detection for 2.207s (election complete) I0703 01:43:27.011412 2648 raft_consensus.cc:2537] T a200f8a763f14d54a1e84082dbafce8f P 768d8de4b2ba40f180b18194c28be589 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0703 01:43:27.011464 2659 leader_election.cc:243] T a200f8a763f14d54a1e84082dbafce8f P d9dc68ace903455d809cb1f715fe36be [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521) I0703 01:43:27.011510 2648 raft_consensus.cc:457] T a200f8a763f14d54a1e84082dbafce8f P 768d8de4b2ba40f180b18194c28be589 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1288ca0989104bbea36e22f38c374218) I0703 01:43:27.011596 2648 raft_consensus.cc:2730] T a200f8a763f14d54a1e84082dbafce8f P 768d8de4b2ba40f180b18194c28be589: Snoozing failure detection for 1.633s (starting election) I0703 01:43:27.011626 2659 leader_election.cc:243] T a200f8a763f14d54a1e84082dbafce8f P d9dc68ace903455d809cb1f715fe36be [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 768d8de4b2ba40f180b18194c28be589 (127.5.240.198:42654) I0703 01:43:27.011708 2648 raft_consensus.cc:2782] T a200f8a763f14d54a1e84082dbafce8f P 768d8de4b2ba40f180b18194c28be589 [term 1 FOLLOWER]: Advancing to term 2 I0703 01:43:27.015503 31272 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "a200f8a763f14d54a1e84082dbafce8f" candidate_uuid: "d9dc68ace903455d809cb1f715fe36be" candidate_term: 2 candidate_status { last_received { term: 1 index: 613 } } ignore_live_leader: false dest_uuid: "1288ca0989104bbea36e22f38c374218" is_pre_election: true I0703 01:43:27.015699 31272 raft_consensus.cc:2730] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218: Snoozing failure detection for 2.193s (vote granted) I0703 01:43:27.015791 31272 raft_consensus.cc:2243] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9dc68ace903455d809cb1f715fe36be in term 1. I0703 01:43:27.015830 2648 raft_consensus.cc:479] T a200f8a763f14d54a1e84082dbafce8f P 768d8de4b2ba40f180b18194c28be589 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 517 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } I0703 01:43:27.016351 32433 leader_election.cc:379] T a200f8a763f14d54a1e84082dbafce8f P d9dc68ace903455d809cb1f715fe36be [CANDIDATE]: Term 2 pre-election: Vote granted by peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521) I0703 01:43:27.016419 32433 leader_election.cc:270] T a200f8a763f14d54a1e84082dbafce8f P d9dc68ace903455d809cb1f715fe36be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. I0703 01:43:27.016638 2648 leader_election.cc:243] T a200f8a763f14d54a1e84082dbafce8f P 768d8de4b2ba40f180b18194c28be589 [CANDIDATE]: Term 2 election: Requesting vote from peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521) I0703 01:43:27.016646 2659 raft_consensus.cc:2730] T a200f8a763f14d54a1e84082dbafce8f P d9dc68ace903455d809cb1f715fe36be: Snoozing failure detection for 2.159s (election complete) I0703 01:43:27.016716 2659 raft_consensus.cc:2537] T a200f8a763f14d54a1e84082dbafce8f P d9dc68ace903455d809cb1f715fe36be [term 1 FOLLOWER]: Leader pre-election won for term 2 I0703 01:43:27.016772 2659 raft_consensus.cc:457] T a200f8a763f14d54a1e84082dbafce8f P d9dc68ace903455d809cb1f715fe36be [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1288ca0989104bbea36e22f38c374218) I0703 01:43:27.016837 2659 raft_consensus.cc:2730] T a200f8a763f14d54a1e84082dbafce8f P d9dc68ace903455d809cb1f715fe36be: Snoozing failure detection for 1.941s (starting election) I0703 01:43:27.016896 2659 raft_consensus.cc:2782] T a200f8a763f14d54a1e84082dbafce8f P d9dc68ace903455d809cb1f715fe36be [term 1 FOLLOWER]: Advancing to term 2 I0703 01:43:27.017536 2648 leader_election.cc:243] T a200f8a763f14d54a1e84082dbafce8f P 768d8de4b2ba40f180b18194c28be589 [CANDIDATE]: Term 2 election: Requesting vote from peer d9dc68ace903455d809cb1f715fe36be (127.5.240.196:56543) I0703 01:43:27.017882 31272 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "a200f8a763f14d54a1e84082dbafce8f" candidate_uuid: "768d8de4b2ba40f180b18194c28be589" candidate_term: 2 candidate_status { last_received { term: 1 index: 613 } } ignore_live_leader: false dest_uuid: "1288ca0989104bbea36e22f38c374218" I0703 01:43:27.018043 31272 raft_consensus.cc:2782] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218 [term 1 FOLLOWER]: Advancing to term 2 I0703 01:43:27.018136 31272 raft_consensus.cc:2730] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218: Snoozing failure detection for 1.849s (vote granted) I0703 01:43:27.018911 2659 raft_consensus.cc:479] T a200f8a763f14d54a1e84082dbafce8f P d9dc68ace903455d809cb1f715fe36be [term 2 FOLLOWER]: Starting leader election with config: opid_index: 517 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } I0703 01:43:27.019358 2659 leader_election.cc:243] T a200f8a763f14d54a1e84082dbafce8f P d9dc68ace903455d809cb1f715fe36be [CANDIDATE]: Term 2 election: Requesting vote from peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521) I0703 01:43:27.019552 2659 leader_election.cc:243] T a200f8a763f14d54a1e84082dbafce8f P d9dc68ace903455d809cb1f715fe36be [CANDIDATE]: Term 2 election: Requesting vote from peer 768d8de4b2ba40f180b18194c28be589 (127.5.240.198:42654) I0703 01:43:27.020287 31268 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "a200f8a763f14d54a1e84082dbafce8f" candidate_uuid: "d9dc68ace903455d809cb1f715fe36be" candidate_term: 2 candidate_status { last_received { term: 1 index: 613 } } ignore_live_leader: false dest_uuid: "1288ca0989104bbea36e22f38c374218" I0703 01:43:27.020720 31272 raft_consensus.cc:2243] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768d8de4b2ba40f180b18194c28be589 in term 2. I0703 01:43:27.021342 32697 leader_election.cc:379] T a200f8a763f14d54a1e84082dbafce8f P 768d8de4b2ba40f180b18194c28be589 [CANDIDATE]: Term 2 election: Vote granted by peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521) I0703 01:43:27.021390 31268 raft_consensus.cc:2218] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d9dc68ace903455d809cb1f715fe36be for term 2 because replica is already servicing an update from a current leader or another vote. I0703 01:43:27.021400 32697 leader_election.cc:270] T a200f8a763f14d54a1e84082dbafce8f P 768d8de4b2ba40f180b18194c28be589 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0703 01:43:27.021571 2648 raft_consensus.cc:2730] T a200f8a763f14d54a1e84082dbafce8f P 768d8de4b2ba40f180b18194c28be589: Snoozing failure detection for 2.159s (election complete) I0703 01:43:27.021697 2648 raft_consensus.cc:2537] T a200f8a763f14d54a1e84082dbafce8f P 768d8de4b2ba40f180b18194c28be589 [term 2 FOLLOWER]: Leader election won for term 2 I0703 01:43:27.022019 32433 leader_election.cc:393] T a200f8a763f14d54a1e84082dbafce8f P d9dc68ace903455d809cb1f715fe36be [CANDIDATE]: Term 2 election: Vote denied by peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521). Message: Service unavailable: T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d9dc68ace903455d809cb1f715fe36be for term 2 because replica is already servicing an update from a current leader or another vote. I0703 01:43:27.062528 32502 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "a200f8a763f14d54a1e84082dbafce8f" candidate_uuid: "768d8de4b2ba40f180b18194c28be589" candidate_term: 2 candidate_status { last_received { term: 1 index: 613 } } ignore_live_leader: false dest_uuid: "d9dc68ace903455d809cb1f715fe36be" is_pre_election: true I0703 01:43:27.064178 32502 raft_consensus.cc:2173] T a200f8a763f14d54a1e84082dbafce8f P d9dc68ace903455d809cb1f715fe36be [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768d8de4b2ba40f180b18194c28be589 in current term 2: Already voted for candidate d9dc68ace903455d809cb1f715fe36be in this term. I0703 01:43:27.064779 32503 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "a200f8a763f14d54a1e84082dbafce8f" candidate_uuid: "768d8de4b2ba40f180b18194c28be589" candidate_term: 2 candidate_status { last_received { term: 1 index: 613 } } ignore_live_leader: false dest_uuid: "d9dc68ace903455d809cb1f715fe36be" I0703 01:43:27.064867 32698 leader_election.cc:393] T a200f8a763f14d54a1e84082dbafce8f P 768d8de4b2ba40f180b18194c28be589 [CANDIDATE]: Term 2 pre-election: Vote denied by peer d9dc68ace903455d809cb1f715fe36be (127.5.240.196:56543). Message: Invalid argument: T a200f8a763f14d54a1e84082dbafce8f P d9dc68ace903455d809cb1f715fe36be [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 768d8de4b2ba40f180b18194c28be589 in current term 2: Already voted for candidate d9dc68ace903455d809cb1f715fe36be in this term. I0703 01:43:27.064939 32503 raft_consensus.cc:2173] T a200f8a763f14d54a1e84082dbafce8f P d9dc68ace903455d809cb1f715fe36be [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 768d8de4b2ba40f180b18194c28be589 in current term 2: Already voted for candidate d9dc68ace903455d809cb1f715fe36be in this term. I0703 01:43:27.065703 32698 leader_election.cc:393] T a200f8a763f14d54a1e84082dbafce8f P 768d8de4b2ba40f180b18194c28be589 [CANDIDATE]: Term 2 election: Vote denied by peer d9dc68ace903455d809cb1f715fe36be (127.5.240.196:56543). Message: Invalid argument: T a200f8a763f14d54a1e84082dbafce8f P d9dc68ace903455d809cb1f715fe36be [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 768d8de4b2ba40f180b18194c28be589 in current term 2: Already voted for candidate d9dc68ace903455d809cb1f715fe36be in this term. I0703 01:43:27.069974 2648 raft_consensus.cc:593] T a200f8a763f14d54a1e84082dbafce8f P 768d8de4b2ba40f180b18194c28be589 [term 2 LEADER]: Becoming Leader. State: Replica: 768d8de4b2ba40f180b18194c28be589, State: Running, Role: LEADER I0703 01:43:27.072894 2648 consensus_queue.cc:204] T a200f8a763f14d54a1e84082dbafce8f P 768d8de4b2ba40f180b18194c28be589 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 613, Committed index: 613, Last appended: 1.613, Last appended by leader: 613, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 517 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } I0703 01:43:27.082953 300 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "a200f8a763f14d54a1e84082dbafce8f" candidate_uuid: "d9dc68ace903455d809cb1f715fe36be" candidate_term: 2 candidate_status { last_received { term: 1 index: 613 } } ignore_live_leader: false dest_uuid: "768d8de4b2ba40f180b18194c28be589" I0703 01:43:27.082953 32767 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "a200f8a763f14d54a1e84082dbafce8f" candidate_uuid: "d9dc68ace903455d809cb1f715fe36be" candidate_term: 2 candidate_status { last_received { term: 1 index: 613 } } ignore_live_leader: false dest_uuid: "768d8de4b2ba40f180b18194c28be589" is_pre_election: true I0703 01:43:27.084111 32767 raft_consensus.cc:2218] T a200f8a763f14d54a1e84082dbafce8f P 768d8de4b2ba40f180b18194c28be589 [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate d9dc68ace903455d809cb1f715fe36be for term 2 because replica is already servicing an update from a current leader or another vote. I0703 01:43:27.084481 300 raft_consensus.cc:2204] T a200f8a763f14d54a1e84082dbafce8f P 768d8de4b2ba40f180b18194c28be589 [term 2 LEADER]: Leader election vote request: Denying vote to candidate d9dc68ace903455d809cb1f715fe36be for term 2 because replica is either leader or believes a valid leader to be alive. I0703 01:43:27.085502 32432 leader_election.cc:393] T a200f8a763f14d54a1e84082dbafce8f P d9dc68ace903455d809cb1f715fe36be [CANDIDATE]: Term 2 pre-election: Vote denied by peer 768d8de4b2ba40f180b18194c28be589 (127.5.240.198:42654). Message: Service unavailable: T a200f8a763f14d54a1e84082dbafce8f P 768d8de4b2ba40f180b18194c28be589 [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate d9dc68ace903455d809cb1f715fe36be for term 2 because replica is already servicing an update from a current leader or another vote. I0703 01:43:27.085891 32432 leader_election.cc:393] T a200f8a763f14d54a1e84082dbafce8f P d9dc68ace903455d809cb1f715fe36be [CANDIDATE]: Term 2 election: Vote denied by peer 768d8de4b2ba40f180b18194c28be589 (127.5.240.198:42654). Message: Invalid argument: T a200f8a763f14d54a1e84082dbafce8f P 768d8de4b2ba40f180b18194c28be589 [term 2 LEADER]: Leader election vote request: Denying vote to candidate d9dc68ace903455d809cb1f715fe36be for term 2 because replica is either leader or believes a valid leader to be alive. I0703 01:43:27.085978 32432 leader_election.cc:270] T a200f8a763f14d54a1e84082dbafce8f P d9dc68ace903455d809cb1f715fe36be [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. I0703 01:43:27.086272 2659 raft_consensus.cc:2730] T a200f8a763f14d54a1e84082dbafce8f P d9dc68ace903455d809cb1f715fe36be: Snoozing failure detection for 1.563s (election complete) I0703 01:43:27.086344 2659 raft_consensus.cc:2485] T a200f8a763f14d54a1e84082dbafce8f P d9dc68ace903455d809cb1f715fe36be [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority I0703 01:43:27.093286 31149 catalog_manager.cc:3798] T a200f8a763f14d54a1e84082dbafce8f P 768d8de4b2ba40f180b18194c28be589 reported cstate change: term changed from 1 to 2, leader changed from 1288ca0989104bbea36e22f38c374218 (127.5.240.193) to 768d8de4b2ba40f180b18194c28be589 (127.5.240.198). New cstate: current_term: 2 leader_uuid: "768d8de4b2ba40f180b18194c28be589" committed_config { opid_index: 517 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } } I0703 01:43:27.103868 31272 raft_consensus.cc:1081] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218 [term 2 FOLLOWER]: Refusing update from remote peer 768d8de4b2ba40f180b18194c28be589: Log matching property violated. Preceding OpId in replica: term: 1 index: 613. Preceding OpId from leader: term: 2 index: 615. (index mismatch) I0703 01:43:27.104087 32503 raft_consensus.cc:1081] T a200f8a763f14d54a1e84082dbafce8f P d9dc68ace903455d809cb1f715fe36be [term 2 FOLLOWER]: Refusing update from remote peer 768d8de4b2ba40f180b18194c28be589: Log matching property violated. Preceding OpId in replica: term: 1 index: 613. Preceding OpId from leader: term: 2 index: 615. (index mismatch) I0703 01:43:27.104602 2648 consensus_queue.cc:962] T a200f8a763f14d54a1e84082dbafce8f P 768d8de4b2ba40f180b18194c28be589 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 614, Last known committed idx: 613, Time since last communication: 0.000s I0703 01:43:27.105052 2648 consensus_queue.cc:962] T a200f8a763f14d54a1e84082dbafce8f P 768d8de4b2ba40f180b18194c28be589 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 614, Last known committed idx: 613, Time since last communication: 0.000s W0703 01:43:27.174684 32432 consensus_peers.cc:433] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be -> Peer 3c1770123dc14bba80e00db5f182b585 (127.5.240.197:44326): Couldn't send request to peer 3c1770123dc14bba80e00db5f182b585 for tablet ffe66a5cfb564964951c619e51e7c47c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 13 times. I0703 01:43:27.194885 32638 raft_consensus.cc:1026] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 3c1770123dc14bba80e00db5f182b585 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.745->[1.746-1.898] Dedup: 1.760->[1.761-1.898] I0703 01:43:27.309118 2364 tablet_bootstrap.cc:427] T ffe66a5cfb564964951c619e51e7c47c P 3c1770123dc14bba80e00db5f182b585: Bootstrap replayed 1/1 log segments. Stats: ops{read=721 overwritten=0 applied=721 ignored=0} inserts{seen=1370 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:43:27.309715 2364 tablet_bootstrap.cc:427] T ffe66a5cfb564964951c619e51e7c47c P 3c1770123dc14bba80e00db5f182b585: Bootstrap complete. I0703 01:43:27.310353 2364 ts_tablet_manager.cc:1045] T ffe66a5cfb564964951c619e51e7c47c P 3c1770123dc14bba80e00db5f182b585: Time spent bootstrapping tablet: real 5.724s user 0.815s sys 0.064s I0703 01:43:27.311357 2364 raft_consensus.cc:325] T ffe66a5cfb564964951c619e51e7c47c P 3c1770123dc14bba80e00db5f182b585 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 704 OBSOLETE_local: false peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } peers { permanent_uuid: "3c1770123dc14bba80e00db5f182b585" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 44326 } } I0703 01:43:27.312166 2364 raft_consensus.cc:628] T ffe66a5cfb564964951c619e51e7c47c P 3c1770123dc14bba80e00db5f182b585 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c1770123dc14bba80e00db5f182b585, State: Initialized, Role: FOLLOWER I0703 01:43:27.312510 2364 consensus_queue.cc:227] T ffe66a5cfb564964951c619e51e7c47c P 3c1770123dc14bba80e00db5f182b585 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 721, Last appended: 2.721, Last appended by leader: 721, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 704 OBSOLETE_local: false peers { permanent_uuid: "de93b58c2d67488dae12285e1806ebac" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 59321 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } peers { permanent_uuid: "3c1770123dc14bba80e00db5f182b585" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 44326 } } I0703 01:43:27.313310 2364 ts_tablet_manager.cc:1073] T ffe66a5cfb564964951c619e51e7c47c P 3c1770123dc14bba80e00db5f182b585: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s I0703 01:43:27.317576 32525 tablet_copy_service.cc:325] P d9dc68ace903455d809cb1f715fe36be: Request end of tablet copy session 3c1770123dc14bba80e00db5f182b585-ffe66a5cfb564964951c619e51e7c47c received from {username='jenkins'} at 127.5.240.197:60550 I0703 01:43:27.317716 32525 tablet_copy_service.cc:413] P d9dc68ace903455d809cb1f715fe36be: ending tablet copy session 3c1770123dc14bba80e00db5f182b585-ffe66a5cfb564964951c619e51e7c47c on tablet ffe66a5cfb564964951c619e51e7c47c with peer 3c1770123dc14bba80e00db5f182b585 W0703 01:43:27.560667 31328 outbound_call.cc:287] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.240.199:57714, user_credentials={real_user=jenkins}} blocked reactor thread for 46928.2us I0703 01:43:27.741354 31393 tablet_service.cc:1140] Received LeaderStepDown RPC: tablet_id: "f9f1c40150da4144acb68186f9aad763" dest_uuid: "2f3fdb7c089640c0b936a574bc64707e" from {username='jenkins'} at 127.5.240.194:59412 I0703 01:43:27.741484 31393 raft_consensus.cc:552] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e [term 1 LEADER]: Received request to step down I0703 01:43:27.741536 31393 raft_consensus.cc:628] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2f3fdb7c089640c0b936a574bc64707e, State: Running, Role: LEADER I0703 01:43:27.741677 31393 consensus_queue.cc:227] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 805, Committed index: 805, Last appended: 1.805, Last appended by leader: 805, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 664 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } I0703 01:43:27.741864 31393 consensus_peers.cc:455] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 768d8de4b2ba40f180b18194c28be589 (127.5.240.198:42654): Closing peer: 768d8de4b2ba40f180b18194c28be589 I0703 01:43:27.741927 31393 consensus_peers.cc:455] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521): Closing peer: 1288ca0989104bbea36e22f38c374218 I0703 01:43:27.742027 31393 raft_consensus.cc:2730] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e: Snoozing failure detection for 3.000s (explicit stepdown request) I0703 01:43:27.767599 32638 raft_consensus.cc:1026] T ffe66a5cfb564964951c619e51e7c47c P 3c1770123dc14bba80e00db5f182b585 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.703->[2.704-2.887] Dedup: 2.721->[2.722-2.887] I0703 01:43:28.881961 31393 tablet_service.cc:1140] Received LeaderStepDown RPC: tablet_id: "1f01bfdde47b4d37a0f77d199bc7b4d2" dest_uuid: "2f3fdb7c089640c0b936a574bc64707e" from {username='jenkins'} at 127.5.240.194:59436 I0703 01:43:28.882093 31393 raft_consensus.cc:552] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e [term 1 LEADER]: Received request to step down I0703 01:43:28.882128 31393 raft_consensus.cc:628] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2f3fdb7c089640c0b936a574bc64707e, State: Running, Role: LEADER I0703 01:43:28.882282 31393 consensus_queue.cc:227] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 926, Committed index: 926, Last appended: 1.927, Last appended by leader: 927, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 746 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } peers { permanent_uuid: "3c1770123dc14bba80e00db5f182b585" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 44326 } } I0703 01:43:28.882462 31393 consensus_peers.cc:455] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 3c1770123dc14bba80e00db5f182b585 (127.5.240.197:44326): Closing peer: 3c1770123dc14bba80e00db5f182b585 I0703 01:43:28.882496 31393 consensus_peers.cc:455] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e -> Peer 768d8de4b2ba40f180b18194c28be589 (127.5.240.198:42654): Closing peer: 768d8de4b2ba40f180b18194c28be589 I0703 01:43:28.882525 31393 raft_consensus.cc:2730] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e: Snoozing failure detection for 3.000s (explicit stepdown request) I0703 01:43:29.236279 2396 raft_consensus.cc:457] T f9f1c40150da4144acb68186f9aad763 P 1288ca0989104bbea36e22f38c374218 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2f3fdb7c089640c0b936a574bc64707e) I0703 01:43:29.236382 2396 raft_consensus.cc:2730] T f9f1c40150da4144acb68186f9aad763 P 1288ca0989104bbea36e22f38c374218: Snoozing failure detection for 1.605s (starting election) I0703 01:43:29.236505 2396 raft_consensus.cc:479] T f9f1c40150da4144acb68186f9aad763 P 1288ca0989104bbea36e22f38c374218 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 664 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } I0703 01:43:29.237043 2396 leader_election.cc:243] T f9f1c40150da4144acb68186f9aad763 P 1288ca0989104bbea36e22f38c374218 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194:39827) I0703 01:43:29.237223 2396 leader_election.cc:243] T f9f1c40150da4144acb68186f9aad763 P 1288ca0989104bbea36e22f38c374218 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 768d8de4b2ba40f180b18194c28be589 (127.5.240.198:42654) I0703 01:43:29.237730 31393 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "f9f1c40150da4144acb68186f9aad763" candidate_uuid: "1288ca0989104bbea36e22f38c374218" candidate_term: 2 candidate_status { last_received { term: 1 index: 802 } } ignore_live_leader: false dest_uuid: "2f3fdb7c089640c0b936a574bc64707e" is_pre_election: true I0703 01:43:29.237968 31393 raft_consensus.cc:2190] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1288ca0989104bbea36e22f38c374218 for term 2 because replica has last-logged OpId of term: 1 index: 805, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 802. I0703 01:43:29.238272 32765 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "f9f1c40150da4144acb68186f9aad763" candidate_uuid: "1288ca0989104bbea36e22f38c374218" candidate_term: 2 candidate_status { last_received { term: 1 index: 802 } } ignore_live_leader: false dest_uuid: "768d8de4b2ba40f180b18194c28be589" is_pre_election: true I0703 01:43:29.238539 32765 raft_consensus.cc:2204] T f9f1c40150da4144acb68186f9aad763 P 768d8de4b2ba40f180b18194c28be589 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1288ca0989104bbea36e22f38c374218 for term 2 because replica is either leader or believes a valid leader to be alive. I0703 01:43:29.239085 31203 leader_election.cc:393] T f9f1c40150da4144acb68186f9aad763 P 1288ca0989104bbea36e22f38c374218 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 768d8de4b2ba40f180b18194c28be589 (127.5.240.198:42654). Message: Invalid argument: T f9f1c40150da4144acb68186f9aad763 P 768d8de4b2ba40f180b18194c28be589 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1288ca0989104bbea36e22f38c374218 for term 2 because replica is either leader or believes a valid leader to be alive. I0703 01:43:29.239269 31203 leader_election.cc:393] T f9f1c40150da4144acb68186f9aad763 P 1288ca0989104bbea36e22f38c374218 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194:39827). Message: Invalid argument: T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1288ca0989104bbea36e22f38c374218 for term 2 because replica has last-logged OpId of term: 1 index: 805, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 802. I0703 01:43:29.239326 31203 leader_election.cc:270] T f9f1c40150da4144acb68186f9aad763 P 1288ca0989104bbea36e22f38c374218 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0703 01:43:29.239534 2396 raft_consensus.cc:2730] T f9f1c40150da4144acb68186f9aad763 P 1288ca0989104bbea36e22f38c374218: Snoozing failure detection for 2.033s (election complete) I0703 01:43:29.239600 2396 raft_consensus.cc:2485] T f9f1c40150da4144acb68186f9aad763 P 1288ca0989104bbea36e22f38c374218 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0703 01:43:29.241164 2830 raft_consensus.cc:457] T f9f1c40150da4144acb68186f9aad763 P 768d8de4b2ba40f180b18194c28be589 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2f3fdb7c089640c0b936a574bc64707e) I0703 01:43:29.241401 2830 raft_consensus.cc:2730] T f9f1c40150da4144acb68186f9aad763 P 768d8de4b2ba40f180b18194c28be589: Snoozing failure detection for 2.039s (starting election) I0703 01:43:29.241864 2830 raft_consensus.cc:479] T f9f1c40150da4144acb68186f9aad763 P 768d8de4b2ba40f180b18194c28be589 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 664 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } I0703 01:43:29.242574 2830 leader_election.cc:243] T f9f1c40150da4144acb68186f9aad763 P 768d8de4b2ba40f180b18194c28be589 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194:39827) I0703 01:43:29.242761 2830 leader_election.cc:243] T f9f1c40150da4144acb68186f9aad763 P 768d8de4b2ba40f180b18194c28be589 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521) I0703 01:43:29.243741 31271 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "f9f1c40150da4144acb68186f9aad763" candidate_uuid: "768d8de4b2ba40f180b18194c28be589" candidate_term: 2 candidate_status { last_received { term: 1 index: 805 } } ignore_live_leader: false dest_uuid: "1288ca0989104bbea36e22f38c374218" is_pre_election: true I0703 01:43:29.243875 31271 raft_consensus.cc:2730] T f9f1c40150da4144acb68186f9aad763 P 1288ca0989104bbea36e22f38c374218: Snoozing failure detection for 1.757s (vote granted) I0703 01:43:29.243929 31271 raft_consensus.cc:2243] T f9f1c40150da4144acb68186f9aad763 P 1288ca0989104bbea36e22f38c374218 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768d8de4b2ba40f180b18194c28be589 in term 1. I0703 01:43:29.244457 32697 leader_election.cc:379] T f9f1c40150da4144acb68186f9aad763 P 768d8de4b2ba40f180b18194c28be589 [CANDIDATE]: Term 2 pre-election: Vote granted by peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521) I0703 01:43:29.244513 32697 leader_election.cc:270] T f9f1c40150da4144acb68186f9aad763 P 768d8de4b2ba40f180b18194c28be589 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. I0703 01:43:29.244750 2830 raft_consensus.cc:2730] T f9f1c40150da4144acb68186f9aad763 P 768d8de4b2ba40f180b18194c28be589: Snoozing failure detection for 1.555s (election complete) I0703 01:43:29.244823 2830 raft_consensus.cc:2537] T f9f1c40150da4144acb68186f9aad763 P 768d8de4b2ba40f180b18194c28be589 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0703 01:43:29.244874 2830 raft_consensus.cc:457] T f9f1c40150da4144acb68186f9aad763 P 768d8de4b2ba40f180b18194c28be589 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2f3fdb7c089640c0b936a574bc64707e) I0703 01:43:29.244932 2830 raft_consensus.cc:2730] T f9f1c40150da4144acb68186f9aad763 P 768d8de4b2ba40f180b18194c28be589: Snoozing failure detection for 1.545s (starting election) I0703 01:43:29.244987 2830 raft_consensus.cc:2782] T f9f1c40150da4144acb68186f9aad763 P 768d8de4b2ba40f180b18194c28be589 [term 1 FOLLOWER]: Advancing to term 2 I0703 01:43:29.245079 31393 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "f9f1c40150da4144acb68186f9aad763" candidate_uuid: "768d8de4b2ba40f180b18194c28be589" candidate_term: 2 candidate_status { last_received { term: 1 index: 805 } } ignore_live_leader: false dest_uuid: "2f3fdb7c089640c0b936a574bc64707e" is_pre_election: true I0703 01:43:29.245213 31393 raft_consensus.cc:2730] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e: Snoozing failure detection for 2.181s (vote granted) I0703 01:43:29.245273 31393 raft_consensus.cc:2243] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768d8de4b2ba40f180b18194c28be589 in term 1. I0703 01:43:29.247171 32696 leader_election.cc:379] T f9f1c40150da4144acb68186f9aad763 P 768d8de4b2ba40f180b18194c28be589 [CANDIDATE]: Term 2 pre-election: Vote granted by peer 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194:39827) I0703 01:43:29.247459 2830 raft_consensus.cc:479] T f9f1c40150da4144acb68186f9aad763 P 768d8de4b2ba40f180b18194c28be589 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 664 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } I0703 01:43:29.249650 2830 leader_election.cc:243] T f9f1c40150da4144acb68186f9aad763 P 768d8de4b2ba40f180b18194c28be589 [CANDIDATE]: Term 2 election: Requesting vote from peer 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194:39827) I0703 01:43:29.250149 2830 leader_election.cc:243] T f9f1c40150da4144acb68186f9aad763 P 768d8de4b2ba40f180b18194c28be589 [CANDIDATE]: Term 2 election: Requesting vote from peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521) I0703 01:43:29.250725 31393 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "f9f1c40150da4144acb68186f9aad763" candidate_uuid: "768d8de4b2ba40f180b18194c28be589" candidate_term: 2 candidate_status { last_received { term: 1 index: 805 } } ignore_live_leader: false dest_uuid: "2f3fdb7c089640c0b936a574bc64707e" I0703 01:43:29.250849 31393 raft_consensus.cc:2782] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e [term 1 FOLLOWER]: Advancing to term 2 I0703 01:43:29.250916 31393 raft_consensus.cc:2730] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e: Snoozing failure detection for 2.190s (vote granted) I0703 01:43:29.250886 31271 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "f9f1c40150da4144acb68186f9aad763" candidate_uuid: "768d8de4b2ba40f180b18194c28be589" candidate_term: 2 candidate_status { last_received { term: 1 index: 805 } } ignore_live_leader: false dest_uuid: "1288ca0989104bbea36e22f38c374218" I0703 01:43:29.251009 31271 raft_consensus.cc:2782] T f9f1c40150da4144acb68186f9aad763 P 1288ca0989104bbea36e22f38c374218 [term 1 FOLLOWER]: Advancing to term 2 I0703 01:43:29.251078 31271 raft_consensus.cc:2730] T f9f1c40150da4144acb68186f9aad763 P 1288ca0989104bbea36e22f38c374218: Snoozing failure detection for 2.961s (vote granted) I0703 01:43:29.252820 31393 raft_consensus.cc:2243] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768d8de4b2ba40f180b18194c28be589 in term 2. I0703 01:43:29.253083 31271 raft_consensus.cc:2243] T f9f1c40150da4144acb68186f9aad763 P 1288ca0989104bbea36e22f38c374218 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768d8de4b2ba40f180b18194c28be589 in term 2. I0703 01:43:29.253609 32696 leader_election.cc:379] T f9f1c40150da4144acb68186f9aad763 P 768d8de4b2ba40f180b18194c28be589 [CANDIDATE]: Term 2 election: Vote granted by peer 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194:39827) I0703 01:43:29.253679 32696 leader_election.cc:270] T f9f1c40150da4144acb68186f9aad763 P 768d8de4b2ba40f180b18194c28be589 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0703 01:43:29.253876 2709 raft_consensus.cc:2730] T f9f1c40150da4144acb68186f9aad763 P 768d8de4b2ba40f180b18194c28be589: Snoozing failure detection for 2.065s (election complete) I0703 01:43:29.253954 2709 raft_consensus.cc:2537] T f9f1c40150da4144acb68186f9aad763 P 768d8de4b2ba40f180b18194c28be589 [term 2 FOLLOWER]: Leader election won for term 2 I0703 01:43:29.254096 2709 raft_consensus.cc:593] T f9f1c40150da4144acb68186f9aad763 P 768d8de4b2ba40f180b18194c28be589 [term 2 LEADER]: Becoming Leader. State: Replica: 768d8de4b2ba40f180b18194c28be589, State: Running, Role: LEADER I0703 01:43:29.254382 32697 leader_election.cc:379] T f9f1c40150da4144acb68186f9aad763 P 768d8de4b2ba40f180b18194c28be589 [CANDIDATE]: Term 2 election: Vote granted by peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521) I0703 01:43:29.254449 2709 consensus_queue.cc:204] T f9f1c40150da4144acb68186f9aad763 P 768d8de4b2ba40f180b18194c28be589 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 805, Committed index: 805, Last appended: 1.805, Last appended by leader: 805, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 664 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } I0703 01:43:29.264153 31147 catalog_manager.cc:3798] T f9f1c40150da4144acb68186f9aad763 P 768d8de4b2ba40f180b18194c28be589 reported cstate change: term changed from 1 to 2, leader changed from 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194) to 768d8de4b2ba40f180b18194c28be589 (127.5.240.198). New cstate: current_term: 2 leader_uuid: "768d8de4b2ba40f180b18194c28be589" committed_config { opid_index: 664 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } } I0703 01:43:29.408604 31272 raft_consensus.cc:1081] T f9f1c40150da4144acb68186f9aad763 P 1288ca0989104bbea36e22f38c374218 [term 2 FOLLOWER]: Refusing update from remote peer 768d8de4b2ba40f180b18194c28be589: Log matching property violated. Preceding OpId in replica: term: 1 index: 802. Preceding OpId from leader: term: 2 index: 807. (index mismatch) I0703 01:43:29.408771 31393 raft_consensus.cc:1081] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e [term 2 FOLLOWER]: Refusing update from remote peer 768d8de4b2ba40f180b18194c28be589: Log matching property violated. Preceding OpId in replica: term: 1 index: 805. Preceding OpId from leader: term: 2 index: 807. (index mismatch) I0703 01:43:29.409559 2830 consensus_queue.cc:962] T f9f1c40150da4144acb68186f9aad763 P 768d8de4b2ba40f180b18194c28be589 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 806, Last known committed idx: 802, Time since last communication: 0.000s I0703 01:43:29.412065 2709 consensus_queue.cc:962] T f9f1c40150da4144acb68186f9aad763 P 768d8de4b2ba40f180b18194c28be589 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 806, Last known committed idx: 805, Time since last communication: 0.000s I0703 01:43:29.892316 300 consensus_peers.cc:455] T a200f8a763f14d54a1e84082dbafce8f P 768d8de4b2ba40f180b18194c28be589 -> Peer d9dc68ace903455d809cb1f715fe36be (127.5.240.196:56543): Closing peer: d9dc68ace903455d809cb1f715fe36be I0703 01:43:29.892410 300 consensus_peers.cc:455] T a200f8a763f14d54a1e84082dbafce8f P 768d8de4b2ba40f180b18194c28be589 -> Peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521): Closing peer: 1288ca0989104bbea36e22f38c374218 I0703 01:43:29.892753 300 consensus_queue.cc:204] T a200f8a763f14d54a1e84082dbafce8f P 768d8de4b2ba40f180b18194c28be589 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 681, Committed index: 681, Last appended: 2.681, Last appended by leader: 613, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 682 OBSOLETE_local: false peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } I0703 01:43:29.895462 32502 raft_consensus.cc:1081] T a200f8a763f14d54a1e84082dbafce8f P d9dc68ace903455d809cb1f715fe36be [term 2 FOLLOWER]: Refusing update from remote peer 768d8de4b2ba40f180b18194c28be589: Log matching property violated. Preceding OpId in replica: term: 2 index: 681. Preceding OpId from leader: term: 2 index: 682. (index mismatch) I0703 01:43:29.897241 2830 consensus_queue.cc:962] T a200f8a763f14d54a1e84082dbafce8f P 768d8de4b2ba40f180b18194c28be589 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 682, Last known committed idx: 681, Time since last communication: 0.000s I0703 01:43:29.901715 2917 raft_consensus.cc:2684] T a200f8a763f14d54a1e84082dbafce8f P 768d8de4b2ba40f180b18194c28be589 [term 2 LEADER]: Committing config change with OpId 2.682: config changed from index 517 to 682, VOTER 1288ca0989104bbea36e22f38c374218 (127.5.240.193) evicted. New config: { opid_index: 682 OBSOLETE_local: false peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } } I0703 01:43:29.903656 32502 raft_consensus.cc:2684] T a200f8a763f14d54a1e84082dbafce8f P d9dc68ace903455d809cb1f715fe36be [term 2 FOLLOWER]: Committing config change with OpId 2.682: config changed from index 517 to 682, VOTER 1288ca0989104bbea36e22f38c374218 (127.5.240.193) evicted. New config: { opid_index: 682 OBSOLETE_local: false peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } } I0703 01:43:29.906975 31147 catalog_manager.cc:3798] T a200f8a763f14d54a1e84082dbafce8f P 768d8de4b2ba40f180b18194c28be589 reported cstate change: config changed from index 517 to 682, VOTER 1288ca0989104bbea36e22f38c374218 (127.5.240.193) evicted. New cstate: current_term: 2 leader_uuid: "768d8de4b2ba40f180b18194c28be589" committed_config { opid_index: 682 OBSOLETE_local: false peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } } I0703 01:43:29.947371 31147 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet a200f8a763f14d54a1e84082dbafce8f on 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521) (TS 1288ca0989104bbea36e22f38c374218 not found in new config with opid_index 682) I0703 01:43:29.948283 31250 tablet_service.cc:800] Processing DeleteTablet for tablet a200f8a763f14d54a1e84082dbafce8f with delete_type TABLET_DATA_TOMBSTONED (TS 1288ca0989104bbea36e22f38c374218 not found in new config with opid_index 682) from {username='jenkins'} at 127.5.240.193:33318 I0703 01:43:29.950363 2924 tablet_replica.cc:261] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218: stopping tablet replica I0703 01:43:29.950929 2924 maintenance_manager.cc:237] P 1288ca0989104bbea36e22f38c374218: Unregistered op CompactRowSetsOp(a200f8a763f14d54a1e84082dbafce8f) I0703 01:43:29.951030 2924 maintenance_manager.cc:237] P 1288ca0989104bbea36e22f38c374218: Unregistered op MinorDeltaCompactionOp(a200f8a763f14d54a1e84082dbafce8f) I0703 01:43:29.951117 2924 maintenance_manager.cc:237] P 1288ca0989104bbea36e22f38c374218: Unregistered op MajorDeltaCompactionOp(a200f8a763f14d54a1e84082dbafce8f) I0703 01:43:29.951203 2924 maintenance_manager.cc:237] P 1288ca0989104bbea36e22f38c374218: Unregistered op UndoDeltaBlockGCOp(a200f8a763f14d54a1e84082dbafce8f) I0703 01:43:29.951287 2924 maintenance_manager.cc:237] P 1288ca0989104bbea36e22f38c374218: Unregistered op FlushMRSOp(a200f8a763f14d54a1e84082dbafce8f) I0703 01:43:29.951352 2924 maintenance_manager.cc:237] P 1288ca0989104bbea36e22f38c374218: Unregistered op FlushDeltaMemStoresOp(a200f8a763f14d54a1e84082dbafce8f) I0703 01:43:29.951416 2924 maintenance_manager.cc:237] P 1288ca0989104bbea36e22f38c374218: Unregistered op LogGCOp(a200f8a763f14d54a1e84082dbafce8f) I0703 01:43:29.951483 2924 raft_consensus.cc:2040] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218 [term 2 FOLLOWER]: Raft consensus shutting down. I0703 01:43:29.952474 2924 raft_consensus.cc:2067] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218 [term 2 FOLLOWER]: Raft consensus is shut down! I0703 01:43:29.953474 2924 ts_tablet_manager.cc:1369] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:43:29.970604 2924 ts_tablet_manager.cc:1382] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.681 I0703 01:43:29.970754 2924 log.cc:986] T a200f8a763f14d54a1e84082dbafce8f P 1288ca0989104bbea36e22f38c374218: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal/wals/a200f8a763f14d54a1e84082dbafce8f I0703 01:43:29.971853 31134 catalog_manager.cc:3103] TS 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521): tablet a200f8a763f14d54a1e84082dbafce8f (table rebalance_test_table_2 [id=b1de6184513b4ad5b539fd284d2c95d9]) successfully deleted I0703 01:43:30.383424 2648 raft_consensus.cc:457] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 768d8de4b2ba40f180b18194c28be589 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2f3fdb7c089640c0b936a574bc64707e) I0703 01:43:30.383579 2648 raft_consensus.cc:2730] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 768d8de4b2ba40f180b18194c28be589: Snoozing failure detection for 1.887s (starting election) I0703 01:43:30.383687 2648 raft_consensus.cc:479] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 768d8de4b2ba40f180b18194c28be589 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 746 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } peers { permanent_uuid: "3c1770123dc14bba80e00db5f182b585" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 44326 } } I0703 01:43:30.384244 2648 leader_election.cc:243] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 768d8de4b2ba40f180b18194c28be589 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194:39827) I0703 01:43:30.384384 2648 leader_election.cc:243] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 768d8de4b2ba40f180b18194c28be589 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 3c1770123dc14bba80e00db5f182b585 (127.5.240.197:44326) I0703 01:43:30.404779 31394 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "1f01bfdde47b4d37a0f77d199bc7b4d2" candidate_uuid: "768d8de4b2ba40f180b18194c28be589" candidate_term: 2 candidate_status { last_received { term: 1 index: 927 } } ignore_live_leader: false dest_uuid: "2f3fdb7c089640c0b936a574bc64707e" is_pre_election: true I0703 01:43:30.404960 31394 raft_consensus.cc:2730] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e: Snoozing failure detection for 1.564s (vote granted) I0703 01:43:30.405047 31394 raft_consensus.cc:2243] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768d8de4b2ba40f180b18194c28be589 in term 1. I0703 01:43:30.407953 32696 leader_election.cc:379] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 768d8de4b2ba40f180b18194c28be589 [CANDIDATE]: Term 2 pre-election: Vote granted by peer 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194:39827) I0703 01:43:30.408097 32696 leader_election.cc:270] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 768d8de4b2ba40f180b18194c28be589 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. I0703 01:43:30.408393 2709 raft_consensus.cc:2730] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 768d8de4b2ba40f180b18194c28be589: Snoozing failure detection for 1.956s (election complete) I0703 01:43:30.408500 2709 raft_consensus.cc:2537] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 768d8de4b2ba40f180b18194c28be589 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0703 01:43:30.408581 2709 raft_consensus.cc:457] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 768d8de4b2ba40f180b18194c28be589 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2f3fdb7c089640c0b936a574bc64707e) I0703 01:43:30.408663 2709 raft_consensus.cc:2730] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 768d8de4b2ba40f180b18194c28be589: Snoozing failure detection for 2.170s (starting election) I0703 01:43:30.408716 2709 raft_consensus.cc:2782] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 768d8de4b2ba40f180b18194c28be589 [term 1 FOLLOWER]: Advancing to term 2 I0703 01:43:30.410722 2709 raft_consensus.cc:479] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 768d8de4b2ba40f180b18194c28be589 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 746 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } peers { permanent_uuid: "3c1770123dc14bba80e00db5f182b585" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 44326 } } I0703 01:43:30.411192 2709 leader_election.cc:243] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 768d8de4b2ba40f180b18194c28be589 [CANDIDATE]: Term 2 election: Requesting vote from peer 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194:39827) I0703 01:43:30.411329 2709 leader_election.cc:243] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 768d8de4b2ba40f180b18194c28be589 [CANDIDATE]: Term 2 election: Requesting vote from peer 3c1770123dc14bba80e00db5f182b585 (127.5.240.197:44326) I0703 01:43:30.412818 31393 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "1f01bfdde47b4d37a0f77d199bc7b4d2" candidate_uuid: "768d8de4b2ba40f180b18194c28be589" candidate_term: 2 candidate_status { last_received { term: 1 index: 927 } } ignore_live_leader: false dest_uuid: "2f3fdb7c089640c0b936a574bc64707e" I0703 01:43:30.412987 31393 raft_consensus.cc:2782] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e [term 1 FOLLOWER]: Advancing to term 2 I0703 01:43:30.413092 31393 raft_consensus.cc:2730] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e: Snoozing failure detection for 1.963s (vote granted) I0703 01:43:30.414999 31393 raft_consensus.cc:2243] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768d8de4b2ba40f180b18194c28be589 in term 2. I0703 01:43:30.415791 32696 leader_election.cc:379] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 768d8de4b2ba40f180b18194c28be589 [CANDIDATE]: Term 2 election: Vote granted by peer 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194:39827) I0703 01:43:30.415920 32696 leader_election.cc:270] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 768d8de4b2ba40f180b18194c28be589 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0703 01:43:30.416157 2917 raft_consensus.cc:2730] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 768d8de4b2ba40f180b18194c28be589: Snoozing failure detection for 1.871s (election complete) I0703 01:43:30.416285 2917 raft_consensus.cc:2537] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 768d8de4b2ba40f180b18194c28be589 [term 2 FOLLOWER]: Leader election won for term 2 I0703 01:43:30.416448 2917 raft_consensus.cc:593] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 768d8de4b2ba40f180b18194c28be589 [term 2 LEADER]: Becoming Leader. State: Replica: 768d8de4b2ba40f180b18194c28be589, State: Running, Role: LEADER I0703 01:43:30.416698 2917 consensus_queue.cc:204] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 768d8de4b2ba40f180b18194c28be589 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 926, Committed index: 926, Last appended: 1.927, Last appended by leader: 927, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 746 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } peers { permanent_uuid: "3c1770123dc14bba80e00db5f182b585" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 44326 } } I0703 01:43:30.418223 2955 raft_consensus.cc:457] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 3c1770123dc14bba80e00db5f182b585 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2f3fdb7c089640c0b936a574bc64707e) I0703 01:43:30.422642 31147 catalog_manager.cc:3798] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 768d8de4b2ba40f180b18194c28be589 reported cstate change: term changed from 1 to 2, leader changed from 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194) to 768d8de4b2ba40f180b18194c28be589 (127.5.240.198). New cstate: current_term: 2 leader_uuid: "768d8de4b2ba40f180b18194c28be589" committed_config { opid_index: 746 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } peers { permanent_uuid: "3c1770123dc14bba80e00db5f182b585" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 44326 } } } I0703 01:43:30.427043 2955 raft_consensus.cc:2730] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 3c1770123dc14bba80e00db5f182b585: Snoozing failure detection for 2.180s (starting election) I0703 01:43:30.428946 2955 raft_consensus.cc:479] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 3c1770123dc14bba80e00db5f182b585 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 746 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } peers { permanent_uuid: "3c1770123dc14bba80e00db5f182b585" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 44326 } } I0703 01:43:30.430248 2955 leader_election.cc:243] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 3c1770123dc14bba80e00db5f182b585 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194:39827) I0703 01:43:30.432947 2955 leader_election.cc:243] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 3c1770123dc14bba80e00db5f182b585 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 768d8de4b2ba40f180b18194c28be589 (127.5.240.198:42654) I0703 01:43:30.434718 31393 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "1f01bfdde47b4d37a0f77d199bc7b4d2" candidate_uuid: "3c1770123dc14bba80e00db5f182b585" candidate_term: 2 candidate_status { last_received { term: 1 index: 927 } } ignore_live_leader: false dest_uuid: "2f3fdb7c089640c0b936a574bc64707e" is_pre_election: true I0703 01:43:30.434883 31393 raft_consensus.cc:2173] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c1770123dc14bba80e00db5f182b585 in current term 2: Already voted for candidate 768d8de4b2ba40f180b18194c28be589 in this term. I0703 01:43:30.447006 32563 leader_election.cc:393] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 3c1770123dc14bba80e00db5f182b585 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194:39827). Message: Invalid argument: T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c1770123dc14bba80e00db5f182b585 in current term 2: Already voted for candidate 768d8de4b2ba40f180b18194c28be589 in this term. I0703 01:43:30.450990 32636 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "1f01bfdde47b4d37a0f77d199bc7b4d2" candidate_uuid: "768d8de4b2ba40f180b18194c28be589" candidate_term: 2 candidate_status { last_received { term: 1 index: 927 } } ignore_live_leader: false dest_uuid: "3c1770123dc14bba80e00db5f182b585" is_pre_election: true I0703 01:43:30.451244 32636 raft_consensus.cc:2730] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 3c1770123dc14bba80e00db5f182b585: Snoozing failure detection for 1.962s (vote granted) I0703 01:43:30.451375 32636 raft_consensus.cc:2243] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 3c1770123dc14bba80e00db5f182b585 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768d8de4b2ba40f180b18194c28be589 in term 1. I0703 01:43:30.451458 32638 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "1f01bfdde47b4d37a0f77d199bc7b4d2" candidate_uuid: "768d8de4b2ba40f180b18194c28be589" candidate_term: 2 candidate_status { last_received { term: 1 index: 927 } } ignore_live_leader: false dest_uuid: "3c1770123dc14bba80e00db5f182b585" I0703 01:43:30.451674 32638 raft_consensus.cc:2782] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 3c1770123dc14bba80e00db5f182b585 [term 1 FOLLOWER]: Advancing to term 2 I0703 01:43:30.451798 32638 raft_consensus.cc:2730] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 3c1770123dc14bba80e00db5f182b585: Snoozing failure detection for 2.062s (vote granted) I0703 01:43:30.453987 32638 raft_consensus.cc:2243] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 3c1770123dc14bba80e00db5f182b585 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768d8de4b2ba40f180b18194c28be589 in term 2. I0703 01:43:30.456693 32696 leader_election.cc:379] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 768d8de4b2ba40f180b18194c28be589 [CANDIDATE]: Term 2 pre-election: Vote granted by peer 3c1770123dc14bba80e00db5f182b585 (127.5.240.197:44326) I0703 01:43:30.457231 32696 leader_election.cc:379] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 768d8de4b2ba40f180b18194c28be589 [CANDIDATE]: Term 2 election: Vote granted by peer 3c1770123dc14bba80e00db5f182b585 (127.5.240.197:44326) I0703 01:43:30.478997 300 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "1f01bfdde47b4d37a0f77d199bc7b4d2" candidate_uuid: "3c1770123dc14bba80e00db5f182b585" candidate_term: 2 candidate_status { last_received { term: 1 index: 927 } } ignore_live_leader: false dest_uuid: "768d8de4b2ba40f180b18194c28be589" is_pre_election: true I0703 01:43:30.479199 300 raft_consensus.cc:2204] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 768d8de4b2ba40f180b18194c28be589 [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 3c1770123dc14bba80e00db5f182b585 for term 2 because replica is either leader or believes a valid leader to be alive. I0703 01:43:30.482614 32563 leader_election.cc:393] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 3c1770123dc14bba80e00db5f182b585 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 768d8de4b2ba40f180b18194c28be589 (127.5.240.198:42654). Message: Invalid argument: T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 768d8de4b2ba40f180b18194c28be589 [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 3c1770123dc14bba80e00db5f182b585 for term 2 because replica is either leader or believes a valid leader to be alive. I0703 01:43:30.485011 32563 leader_election.cc:270] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 3c1770123dc14bba80e00db5f182b585 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0703 01:43:30.485328 2955 raft_consensus.cc:2730] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 3c1770123dc14bba80e00db5f182b585: Snoozing failure detection for 1.938s (election complete) I0703 01:43:30.485396 2955 raft_consensus.cc:2485] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 3c1770123dc14bba80e00db5f182b585 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0703 01:43:30.833272 32636 raft_consensus.cc:1081] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 3c1770123dc14bba80e00db5f182b585 [term 2 FOLLOWER]: Refusing update from remote peer 768d8de4b2ba40f180b18194c28be589: Log matching property violated. Preceding OpId in replica: term: 1 index: 927. Preceding OpId from leader: term: 2 index: 928. (index mismatch) I0703 01:43:30.836930 2709 consensus_queue.cc:962] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 768d8de4b2ba40f180b18194c28be589 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c1770123dc14bba80e00db5f182b585" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 44326 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 928, Last known committed idx: 926, Time since last communication: 0.000s I0703 01:43:30.853070 31394 raft_consensus.cc:1081] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e [term 2 FOLLOWER]: Refusing update from remote peer 768d8de4b2ba40f180b18194c28be589: Log matching property violated. Preceding OpId in replica: term: 1 index: 927. Preceding OpId from leader: term: 2 index: 928. (index mismatch) I0703 01:43:30.855139 2648 consensus_queue.cc:962] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 768d8de4b2ba40f180b18194c28be589 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 928, Last known committed idx: 926, Time since last communication: 0.000s I0703 01:43:30.858311 2903 rpcz_store.cc:265] Call kudu.tserver.TabletServerService.Write from 127.5.240.194:58921 (ReqId={client: 917d67f52cfc466888988d31b04b3654, seq_no=2715, attempt_no=0}) took 1979ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":1065,"num_ops":1,"prepare.queue_time_us":173,"prepare.run_cpu_time_us":435,"prepare.run_wall_time_us":435,"raft.queue_time_us":266,"raft.run_cpu_time_us":349,"raft.run_wall_time_us":347,"replication_time_us":1976743}]]} I0703 01:43:30.958958 32502 consensus_peers.cc:455] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be -> Peer 3c1770123dc14bba80e00db5f182b585 (127.5.240.197:44326): Closing peer: 3c1770123dc14bba80e00db5f182b585 I0703 01:43:30.959038 32502 consensus_peers.cc:455] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be -> Peer de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321): Closing peer: de93b58c2d67488dae12285e1806ebac I0703 01:43:30.959231 32502 consensus_queue.cc:204] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 982, Committed index: 982, Last appended: 2.982, Last appended by leader: 459, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 983 OBSOLETE_local: false peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } peers { permanent_uuid: "3c1770123dc14bba80e00db5f182b585" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 44326 } } I0703 01:43:30.960690 32636 raft_consensus.cc:1081] T ffe66a5cfb564964951c619e51e7c47c P 3c1770123dc14bba80e00db5f182b585 [term 2 FOLLOWER]: Refusing update from remote peer d9dc68ace903455d809cb1f715fe36be: Log matching property violated. Preceding OpId in replica: term: 2 index: 982. Preceding OpId from leader: term: 2 index: 983. (index mismatch) I0703 01:43:30.962244 2808 consensus_queue.cc:962] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c1770123dc14bba80e00db5f182b585" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 44326 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 983, Last known committed idx: 982, Time since last communication: 0.000s I0703 01:43:30.967505 2808 raft_consensus.cc:2684] T ffe66a5cfb564964951c619e51e7c47c P d9dc68ace903455d809cb1f715fe36be [term 2 LEADER]: Committing config change with OpId 2.983: config changed from index 704 to 983, VOTER de93b58c2d67488dae12285e1806ebac (127.5.240.195) evicted. New config: { opid_index: 983 OBSOLETE_local: false peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } peers { permanent_uuid: "3c1770123dc14bba80e00db5f182b585" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 44326 } } } I0703 01:43:30.971482 32636 raft_consensus.cc:2684] T ffe66a5cfb564964951c619e51e7c47c P 3c1770123dc14bba80e00db5f182b585 [term 2 FOLLOWER]: Committing config change with OpId 2.983: config changed from index 704 to 983, VOTER de93b58c2d67488dae12285e1806ebac (127.5.240.195) evicted. New config: { opid_index: 983 OBSOLETE_local: false peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } peers { permanent_uuid: "3c1770123dc14bba80e00db5f182b585" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 44326 } } } I0703 01:43:30.976301 31147 catalog_manager.cc:3798] T ffe66a5cfb564964951c619e51e7c47c P 3c1770123dc14bba80e00db5f182b585 reported cstate change: config changed from index 704 to 983, VOTER de93b58c2d67488dae12285e1806ebac (127.5.240.195) evicted. New cstate: current_term: 2 leader_uuid: "d9dc68ace903455d809cb1f715fe36be" committed_config { opid_index: 983 OBSOLETE_local: false peers { permanent_uuid: "d9dc68ace903455d809cb1f715fe36be" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 56543 } } peers { permanent_uuid: "3c1770123dc14bba80e00db5f182b585" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 44326 } } } I0703 01:43:31.041373 31147 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet ffe66a5cfb564964951c619e51e7c47c on de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321) (TS de93b58c2d67488dae12285e1806ebac not found in new config with opid_index 983) I0703 01:43:31.041889 31497 tablet_service.cc:800] Processing DeleteTablet for tablet ffe66a5cfb564964951c619e51e7c47c with delete_type TABLET_DATA_TOMBSTONED (TS de93b58c2d67488dae12285e1806ebac not found in new config with opid_index 983) from {username='jenkins'} at 127.5.240.195:46850 I0703 01:43:31.050936 3009 tablet_replica.cc:261] T ffe66a5cfb564964951c619e51e7c47c P de93b58c2d67488dae12285e1806ebac: stopping tablet replica I0703 01:43:31.051043 3009 maintenance_manager.cc:237] P de93b58c2d67488dae12285e1806ebac: Unregistered op CompactRowSetsOp(ffe66a5cfb564964951c619e51e7c47c) I0703 01:43:31.051136 3009 maintenance_manager.cc:237] P de93b58c2d67488dae12285e1806ebac: Unregistered op MinorDeltaCompactionOp(ffe66a5cfb564964951c619e51e7c47c) I0703 01:43:31.051316 3009 maintenance_manager.cc:237] P de93b58c2d67488dae12285e1806ebac: Unregistered op MajorDeltaCompactionOp(ffe66a5cfb564964951c619e51e7c47c) I0703 01:43:31.051393 3009 maintenance_manager.cc:237] P de93b58c2d67488dae12285e1806ebac: Unregistered op UndoDeltaBlockGCOp(ffe66a5cfb564964951c619e51e7c47c) I0703 01:43:31.051475 3009 maintenance_manager.cc:237] P de93b58c2d67488dae12285e1806ebac: Unregistered op FlushMRSOp(ffe66a5cfb564964951c619e51e7c47c) I0703 01:43:31.051542 3009 maintenance_manager.cc:237] P de93b58c2d67488dae12285e1806ebac: Unregistered op FlushDeltaMemStoresOp(ffe66a5cfb564964951c619e51e7c47c) I0703 01:43:31.052332 3009 maintenance_manager.cc:237] P de93b58c2d67488dae12285e1806ebac: Unregistered op LogGCOp(ffe66a5cfb564964951c619e51e7c47c) I0703 01:43:31.052407 3009 raft_consensus.cc:2040] T ffe66a5cfb564964951c619e51e7c47c P de93b58c2d67488dae12285e1806ebac [term 2 FOLLOWER]: Raft consensus shutting down. I0703 01:43:31.052623 3009 raft_consensus.cc:2067] T ffe66a5cfb564964951c619e51e7c47c P de93b58c2d67488dae12285e1806ebac [term 2 FOLLOWER]: Raft consensus is shut down! I0703 01:43:31.060281 3009 ts_tablet_manager.cc:1369] T ffe66a5cfb564964951c619e51e7c47c P de93b58c2d67488dae12285e1806ebac: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:43:31.094849 3009 ts_tablet_manager.cc:1382] T ffe66a5cfb564964951c619e51e7c47c P de93b58c2d67488dae12285e1806ebac: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.982 I0703 01:43:31.094946 3009 log.cc:986] T ffe66a5cfb564964951c619e51e7c47c P de93b58c2d67488dae12285e1806ebac: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal/wals/ffe66a5cfb564964951c619e51e7c47c I0703 01:43:31.096067 31136 catalog_manager.cc:3103] TS de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321): tablet ffe66a5cfb564964951c619e51e7c47c (table rebalance_test_table_4 [id=b0493a1d5ef04606a6d4de013749c53b]) successfully deleted I0703 01:43:32.043329 300 consensus_peers.cc:455] T f9f1c40150da4144acb68186f9aad763 P 768d8de4b2ba40f180b18194c28be589 -> Peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521): Closing peer: 1288ca0989104bbea36e22f38c374218 I0703 01:43:32.043403 300 consensus_peers.cc:455] T f9f1c40150da4144acb68186f9aad763 P 768d8de4b2ba40f180b18194c28be589 -> Peer 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194:39827): Closing peer: 2f3fdb7c089640c0b936a574bc64707e I0703 01:43:32.043712 300 consensus_queue.cc:204] T f9f1c40150da4144acb68186f9aad763 P 768d8de4b2ba40f180b18194c28be589 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 899, Committed index: 899, Last appended: 2.899, Last appended by leader: 805, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 900 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } I0703 01:43:32.050000 31269 raft_consensus.cc:1081] T f9f1c40150da4144acb68186f9aad763 P 1288ca0989104bbea36e22f38c374218 [term 2 FOLLOWER]: Refusing update from remote peer 768d8de4b2ba40f180b18194c28be589: Log matching property violated. Preceding OpId in replica: term: 2 index: 899. Preceding OpId from leader: term: 2 index: 900. (index mismatch) I0703 01:43:32.050844 2917 consensus_queue.cc:962] T f9f1c40150da4144acb68186f9aad763 P 768d8de4b2ba40f180b18194c28be589 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 900, Last known committed idx: 899, Time since last communication: 0.000s I0703 01:43:32.056020 2917 raft_consensus.cc:2684] T f9f1c40150da4144acb68186f9aad763 P 768d8de4b2ba40f180b18194c28be589 [term 2 LEADER]: Committing config change with OpId 2.900: config changed from index 664 to 900, VOTER 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194) evicted. New config: { opid_index: 900 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } } I0703 01:43:32.059252 31271 raft_consensus.cc:2684] T f9f1c40150da4144acb68186f9aad763 P 1288ca0989104bbea36e22f38c374218 [term 2 FOLLOWER]: Committing config change with OpId 2.900: config changed from index 664 to 900, VOTER 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194) evicted. New config: { opid_index: 900 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } } I0703 01:43:32.063717 31152 catalog_manager.cc:3798] T f9f1c40150da4144acb68186f9aad763 P 1288ca0989104bbea36e22f38c374218 reported cstate change: config changed from index 664 to 900, VOTER 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194) evicted. New cstate: current_term: 2 leader_uuid: "768d8de4b2ba40f180b18194c28be589" committed_config { opid_index: 900 OBSOLETE_local: false peers { permanent_uuid: "1288ca0989104bbea36e22f38c374218" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 33521 } } peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } } I0703 01:43:32.106361 31152 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet f9f1c40150da4144acb68186f9aad763 on 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194:39827) (TS 2f3fdb7c089640c0b936a574bc64707e not found in new config with opid_index 900) I0703 01:43:32.109489 31375 tablet_service.cc:800] Processing DeleteTablet for tablet f9f1c40150da4144acb68186f9aad763 with delete_type TABLET_DATA_TOMBSTONED (TS 2f3fdb7c089640c0b936a574bc64707e not found in new config with opid_index 900) from {username='jenkins'} at 127.5.240.194:58888 I0703 01:43:32.115742 3079 tablet_replica.cc:261] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e: stopping tablet replica I0703 01:43:32.115886 3079 maintenance_manager.cc:237] P 2f3fdb7c089640c0b936a574bc64707e: Unregistered op CompactRowSetsOp(f9f1c40150da4144acb68186f9aad763) I0703 01:43:32.116044 3079 maintenance_manager.cc:237] P 2f3fdb7c089640c0b936a574bc64707e: Unregistered op MinorDeltaCompactionOp(f9f1c40150da4144acb68186f9aad763) I0703 01:43:32.116184 3079 maintenance_manager.cc:237] P 2f3fdb7c089640c0b936a574bc64707e: Unregistered op MajorDeltaCompactionOp(f9f1c40150da4144acb68186f9aad763) I0703 01:43:32.116310 3079 maintenance_manager.cc:237] P 2f3fdb7c089640c0b936a574bc64707e: Unregistered op UndoDeltaBlockGCOp(f9f1c40150da4144acb68186f9aad763) I0703 01:43:32.117064 3079 maintenance_manager.cc:237] P 2f3fdb7c089640c0b936a574bc64707e: Unregistered op FlushMRSOp(f9f1c40150da4144acb68186f9aad763) I0703 01:43:32.117175 3079 maintenance_manager.cc:237] P 2f3fdb7c089640c0b936a574bc64707e: Unregistered op FlushDeltaMemStoresOp(f9f1c40150da4144acb68186f9aad763) I0703 01:43:32.117255 3079 maintenance_manager.cc:237] P 2f3fdb7c089640c0b936a574bc64707e: Unregistered op LogGCOp(f9f1c40150da4144acb68186f9aad763) I0703 01:43:32.117738 3079 raft_consensus.cc:2040] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e [term 2 FOLLOWER]: Raft consensus shutting down. I0703 01:43:32.119006 3079 raft_consensus.cc:2067] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e [term 2 FOLLOWER]: Raft consensus is shut down! I0703 01:43:32.120213 3079 ts_tablet_manager.cc:1369] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:43:32.144580 3079 ts_tablet_manager.cc:1382] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.899 I0703 01:43:32.145174 3079 log.cc:986] T f9f1c40150da4144acb68186f9aad763 P 2f3fdb7c089640c0b936a574bc64707e: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal/wals/f9f1c40150da4144acb68186f9aad763 I0703 01:43:32.159730 31133 catalog_manager.cc:3103] TS 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194:39827): tablet f9f1c40150da4144acb68186f9aad763 (table rebalance_test_table_3 [id=b06785934be6432b91c9f50ee8c104d2]) successfully deleted I0703 01:43:33.173630 300 consensus_peers.cc:455] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 768d8de4b2ba40f180b18194c28be589 -> Peer 3c1770123dc14bba80e00db5f182b585 (127.5.240.197:44326): Closing peer: 3c1770123dc14bba80e00db5f182b585 I0703 01:43:33.173712 300 consensus_peers.cc:455] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 768d8de4b2ba40f180b18194c28be589 -> Peer 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194:39827): Closing peer: 2f3fdb7c089640c0b936a574bc64707e I0703 01:43:33.173969 300 consensus_queue.cc:204] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 768d8de4b2ba40f180b18194c28be589 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 984, Committed index: 984, Last appended: 2.984, Last appended by leader: 927, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 985 OBSOLETE_local: false peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } peers { permanent_uuid: "3c1770123dc14bba80e00db5f182b585" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 44326 } } I0703 01:43:33.175774 32638 raft_consensus.cc:1081] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 3c1770123dc14bba80e00db5f182b585 [term 2 FOLLOWER]: Refusing update from remote peer 768d8de4b2ba40f180b18194c28be589: Log matching property violated. Preceding OpId in replica: term: 2 index: 984. Preceding OpId from leader: term: 2 index: 985. (index mismatch) I0703 01:43:33.176486 3092 consensus_queue.cc:962] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 768d8de4b2ba40f180b18194c28be589 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c1770123dc14bba80e00db5f182b585" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 44326 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 985, Last known committed idx: 984, Time since last communication: 0.000s I0703 01:43:33.183708 2648 raft_consensus.cc:2684] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 768d8de4b2ba40f180b18194c28be589 [term 2 LEADER]: Committing config change with OpId 2.985: config changed from index 746 to 985, VOTER 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194) evicted. New config: { opid_index: 985 OBSOLETE_local: false peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } peers { permanent_uuid: "3c1770123dc14bba80e00db5f182b585" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 44326 } } } I0703 01:43:33.187595 32636 raft_consensus.cc:2684] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 3c1770123dc14bba80e00db5f182b585 [term 2 FOLLOWER]: Committing config change with OpId 2.985: config changed from index 746 to 985, VOTER 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194) evicted. New config: { opid_index: 985 OBSOLETE_local: false peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } peers { permanent_uuid: "3c1770123dc14bba80e00db5f182b585" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 44326 } } } I0703 01:43:33.190109 31147 catalog_manager.cc:3798] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 768d8de4b2ba40f180b18194c28be589 reported cstate change: config changed from index 746 to 985, VOTER 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194) evicted. New cstate: current_term: 2 leader_uuid: "768d8de4b2ba40f180b18194c28be589" committed_config { opid_index: 985 OBSOLETE_local: false peers { permanent_uuid: "768d8de4b2ba40f180b18194c28be589" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 42654 } } peers { permanent_uuid: "3c1770123dc14bba80e00db5f182b585" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 44326 } } } I0703 01:43:33.335294 31147 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 1f01bfdde47b4d37a0f77d199bc7b4d2 on 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194:39827) (TS 2f3fdb7c089640c0b936a574bc64707e not found in new config with opid_index 985) I0703 01:43:33.336467 31375 tablet_service.cc:800] Processing DeleteTablet for tablet 1f01bfdde47b4d37a0f77d199bc7b4d2 with delete_type TABLET_DATA_TOMBSTONED (TS 2f3fdb7c089640c0b936a574bc64707e not found in new config with opid_index 985) from {username='jenkins'} at 127.5.240.194:58888 I0703 01:43:33.361204 3151 tablet_replica.cc:261] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e: stopping tablet replica I0703 01:43:33.361316 3151 maintenance_manager.cc:237] P 2f3fdb7c089640c0b936a574bc64707e: Unregistered op CompactRowSetsOp(1f01bfdde47b4d37a0f77d199bc7b4d2) I0703 01:43:33.361409 3151 maintenance_manager.cc:237] P 2f3fdb7c089640c0b936a574bc64707e: Unregistered op MinorDeltaCompactionOp(1f01bfdde47b4d37a0f77d199bc7b4d2) I0703 01:43:33.361488 3151 maintenance_manager.cc:237] P 2f3fdb7c089640c0b936a574bc64707e: Unregistered op MajorDeltaCompactionOp(1f01bfdde47b4d37a0f77d199bc7b4d2) I0703 01:43:33.361635 3151 maintenance_manager.cc:237] P 2f3fdb7c089640c0b936a574bc64707e: Unregistered op UndoDeltaBlockGCOp(1f01bfdde47b4d37a0f77d199bc7b4d2) I0703 01:43:33.361759 3151 maintenance_manager.cc:237] P 2f3fdb7c089640c0b936a574bc64707e: Unregistered op FlushMRSOp(1f01bfdde47b4d37a0f77d199bc7b4d2) I0703 01:43:33.361838 3151 maintenance_manager.cc:237] P 2f3fdb7c089640c0b936a574bc64707e: Unregistered op FlushDeltaMemStoresOp(1f01bfdde47b4d37a0f77d199bc7b4d2) I0703 01:43:33.361917 3151 maintenance_manager.cc:237] P 2f3fdb7c089640c0b936a574bc64707e: Unregistered op LogGCOp(1f01bfdde47b4d37a0f77d199bc7b4d2) I0703 01:43:33.361994 3151 raft_consensus.cc:2040] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e [term 2 FOLLOWER]: Raft consensus shutting down. I0703 01:43:33.367394 3151 raft_consensus.cc:2067] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e [term 2 FOLLOWER]: Raft consensus is shut down! I0703 01:43:33.369318 3151 ts_tablet_manager.cc:1369] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:43:33.388736 3151 ts_tablet_manager.cc:1382] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.984 I0703 01:43:33.388831 3151 log.cc:986] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 2f3fdb7c089640c0b936a574bc64707e: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal/wals/1f01bfdde47b4d37a0f77d199bc7b4d2 I0703 01:43:33.389961 31133 catalog_manager.cc:3103] TS 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194:39827): tablet 1f01bfdde47b4d37a0f77d199bc7b4d2 (table rebalance_test_table_1 [id=7d7c285baa3a4e26a62b27d088cedcc9]) successfully deleted Master Summary UUID | Address | Status ----------------------------------+---------------------+--------- 61b07253093846a589ff6e2547007d7d | 127.5.240.254:36104 | HEALTHY Flag | Value | Tags | Master ------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- allow_unsafe_replication_factor | true | unsafe | all 1 server(s) checked ipki_ca_key_size | 1024 | experimental | all 1 server(s) checked ipki_server_key_size | 1024 | experimental | all 1 server(s) checked never_fsync | true | unsafe,advanced | all 1 server(s) checked raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked rpc_reuseport | true | experimental | all 1 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked server_dump_info_format | pb | hidden | all 1 server(s) checked server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked Tablet Server Summary UUID | Address | Status ----------------------------------+---------------------+--------- 1288ca0989104bbea36e22f38c374218 | 127.5.240.193:33521 | HEALTHY 2f3fdb7c089640c0b936a574bc64707e | 127.5.240.194:39827 | HEALTHY 3c1770123dc14bba80e00db5f182b585 | 127.5.240.197:44326 | HEALTHY 56eaab17f812433a8f7880f0744e1744 | 127.5.240.199:57714 | HEALTHY 768d8de4b2ba40f180b18194c28be589 | 127.5.240.198:42654 | HEALTHY d9dc68ace903455d809cb1f715fe36be | 127.5.240.196:56543 | HEALTHY de93b58c2d67488dae12285e1806ebac | 127.5.240.195:59321 | HEALTHY Flag | Value | Tags | Tablet Server ------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+------------------------- ipki_server_key_size | 1024 | experimental | all 7 server(s) checked local_ip_for_outbound_sockets | 127.5.240.193 | experimental | 127.5.240.193:33521 local_ip_for_outbound_sockets | 127.5.240.194 | experimental | 127.5.240.194:39827 local_ip_for_outbound_sockets | 127.5.240.195 | experimental | 127.5.240.195:59321 local_ip_for_outbound_sockets | 127.5.240.196 | experimental | 127.5.240.196:56543 local_ip_for_outbound_sockets | 127.5.240.197 | experimental | 127.5.240.197:44326 local_ip_for_outbound_sockets | 127.5.240.198 | experimental | 127.5.240.198:42654 local_ip_for_outbound_sockets | 127.5.240.199 | experimental | 127.5.240.199:57714 never_fsync | true | unsafe,advanced | all 7 server(s) checked raft_prepare_replacement_before_eviction | false | experimental,advanced | all 7 server(s) checked rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked server_dump_info_format | pb | hidden | all 7 server(s) checked server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.5.240.193:33521 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.240.194:39827 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.240.195:59321 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.5.240.196:56543 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.240.197:44326 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.240.198:42654 server_dump_info_path | /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_2.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.5.240.199:57714 Version Summary Version | Servers ----------------+------------------------- 1.8.0-SNAPSHOT | all 8 server(s) checked Summary by table Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable ------------------------+----+---------+---------------+---------+------------+------------------+------------- TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0 rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0 rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0 rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0 rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0 rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0 | Total Count ----------------+------------- Masters | 1 Tablet Servers | 7 Tables | 6 Tablets | 16 Replicas | 32 ================== Warnings: ================== Some masters have unsafe, experimental, or hidden flags set Some tablet servers have unsafe, experimental, or hidden flags set OK I0703 01:43:38.938633 6083 log_verifier.cc:121] Checking tablet 19c33e54ee9e476796de7de42ba495b6 I0703 01:43:39.421896 6083 log_verifier.cc:172] Verified matching terms for 1086 ops in tablet 19c33e54ee9e476796de7de42ba495b6 I0703 01:43:39.422785 6083 log_verifier.cc:121] Checking tablet 1f01bfdde47b4d37a0f77d199bc7b4d2 I0703 01:43:39.928290 6083 log_verifier.cc:172] Verified matching terms for 1174 ops in tablet 1f01bfdde47b4d37a0f77d199bc7b4d2 I0703 01:43:39.929244 6083 log_verifier.cc:121] Checking tablet 3d7a6e87780a49949cf7e024817cf3a3 I0703 01:43:40.390302 6083 log_verifier.cc:172] Verified matching terms for 1106 ops in tablet 3d7a6e87780a49949cf7e024817cf3a3 I0703 01:43:40.391214 6083 log_verifier.cc:121] Checking tablet 44f06af6e8ef46d78419aea845bb38a7 I0703 01:43:40.851866 6083 log_verifier.cc:172] Verified matching terms for 1087 ops in tablet 44f06af6e8ef46d78419aea845bb38a7 I0703 01:43:40.852767 6083 log_verifier.cc:121] Checking tablet 454bab377db147a1a2f8241ecc2241f3 I0703 01:43:40.865064 6083 log_verifier.cc:172] Verified matching terms for 4 ops in tablet 454bab377db147a1a2f8241ecc2241f3 I0703 01:43:40.865106 6083 log_verifier.cc:121] Checking tablet 479104ce31404ee4ad2488d9f46509e4 I0703 01:43:41.242507 6083 log_verifier.cc:172] Verified matching terms for 915 ops in tablet 479104ce31404ee4ad2488d9f46509e4 I0703 01:43:41.243273 6083 log_verifier.cc:121] Checking tablet 517dc72fa73f42f0aae102212fbc2d36 I0703 01:43:41.625648 6083 log_verifier.cc:172] Verified matching terms for 887 ops in tablet 517dc72fa73f42f0aae102212fbc2d36 I0703 01:43:41.626374 6083 log_verifier.cc:121] Checking tablet 5eaa9c06d73c4e23b51fae2688eaf26c I0703 01:43:42.078187 6083 log_verifier.cc:172] Verified matching terms for 1131 ops in tablet 5eaa9c06d73c4e23b51fae2688eaf26c I0703 01:43:42.079107 6083 log_verifier.cc:121] Checking tablet 73518ef45438415da1300ef5004ea7d5 I0703 01:43:42.533965 6083 log_verifier.cc:172] Verified matching terms for 1097 ops in tablet 73518ef45438415da1300ef5004ea7d5 I0703 01:43:42.534848 6083 log_verifier.cc:121] Checking tablet a200f8a763f14d54a1e84082dbafce8f I0703 01:43:42.904747 6083 log_verifier.cc:172] Verified matching terms for 900 ops in tablet a200f8a763f14d54a1e84082dbafce8f I0703 01:43:42.905479 6083 log_verifier.cc:121] Checking tablet b2c8b58f935a4d00b5cf3e2cedec4d81 I0703 01:43:43.350455 6083 log_verifier.cc:172] Verified matching terms for 1081 ops in tablet b2c8b58f935a4d00b5cf3e2cedec4d81 I0703 01:43:43.351315 6083 log_verifier.cc:121] Checking tablet bbe64aa4b0e04f468b035048629b969b I0703 01:43:43.794940 6083 log_verifier.cc:172] Verified matching terms for 1098 ops in tablet bbe64aa4b0e04f468b035048629b969b I0703 01:43:43.795825 6083 log_verifier.cc:121] Checking tablet e0af316d40d143a8b387933151adf0e4 I0703 01:43:44.263092 6083 log_verifier.cc:172] Verified matching terms for 1145 ops in tablet e0af316d40d143a8b387933151adf0e4 I0703 01:43:44.264021 6083 log_verifier.cc:121] Checking tablet ee15a8cd712a45d6ac7468bc550d7635 I0703 01:43:44.731353 6083 log_verifier.cc:172] Verified matching terms for 1124 ops in tablet ee15a8cd712a45d6ac7468bc550d7635 I0703 01:43:44.732328 6083 log_verifier.cc:121] Checking tablet f9f1c40150da4144acb68186f9aad763 I0703 01:43:45.161929 6083 log_verifier.cc:172] Verified matching terms for 1056 ops in tablet f9f1c40150da4144acb68186f9aad763 I0703 01:43:45.162781 6083 log_verifier.cc:121] Checking tablet ffe66a5cfb564964951c619e51e7c47c I0703 01:43:45.646697 6083 log_verifier.cc:172] Verified matching terms for 1173 ops in tablet ffe66a5cfb564964951c619e51e7c47c I0703 01:43:45.658892 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 31186 I0703 01:43:45.661097 31312 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:43:45.981773 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 31186 W0703 01:43:46.023663 31327 connection.cc:512] client connection to 127.5.240.193:33521 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:33521): Connection reset by peer (error 104) W0703 01:43:46.023663 31134 connection.cc:512] server connection from 127.5.240.193:48823 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:48823): Connection reset by peer (error 104) W0703 01:43:46.024061 31328 connection.cc:512] server connection from 127.5.240.193:50037 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:50037): Connection reset by peer (error 104) W0703 01:43:46.024060 31449 connection.cc:512] client connection to 127.5.240.193:33521 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:33521): Connection reset by peer (error 104) W0703 01:43:46.024227 32696 connection.cc:512] server connection from 127.5.240.193:60763 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:60763): Connection reset by peer (error 104) W0703 01:43:46.024227 32697 connection.cc:512] client connection to 127.5.240.193:33521 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:33521): Connection reset by peer (error 104) W0703 01:43:46.024399 32697 connection.cc:148] Shutting down client connection to 127.5.240.193:33521 with pending inbound data (10/66 bytes received, last active 0 ns ago, status=Network error: failed to read from TLS socket (remote: 127.5.240.193:33521): Connection reset by peer (error 104)) W0703 01:43:46.024420 359 connection.cc:512] client connection to 127.5.240.193:33521 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:33521): Connection reset by peer (error 104) W0703 01:43:46.024513 31449 connection.cc:512] server connection from 127.5.240.193:53426 recv error: Network error: failed to read from TLS socket (remote: 0.0.0.0:0): Connection reset by peer (error 104) W0703 01:43:46.024601 32697 consensus_peers.cc:433] T f9f1c40150da4144acb68186f9aad763 P 768d8de4b2ba40f180b18194c28be589 -> Peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521): Couldn't send request to peer 1288ca0989104bbea36e22f38c374218 for tablet f9f1c40150da4144acb68186f9aad763. Status: Network error: failed to read from TLS socket (remote: 127.5.240.193:33521): Connection reset by peer (error 104). Retrying in the next heartbeat period. Already tried 1 times. W0703 01:43:46.024756 32432 connection.cc:512] server connection from 127.5.240.193:46492 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:46492): Connection reset by peer (error 104) W0703 01:43:46.024818 31449 connection.cc:185] Error closing socket: Network error: TlsSocket::Close: Broken pipe W0703 01:43:46.024852 358 connection.cc:512] server connection from 127.5.240.193:45995 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:45995): Connection reset by peer (error 104) W0703 01:43:46.025064 32433 connection.cc:512] client connection to 127.5.240.193:33521 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.193:33521): Connection reset by peer (error 104) W0703 01:43:46.025176 32433 consensus_peers.cc:433] T 19c33e54ee9e476796de7de42ba495b6 P d9dc68ace903455d809cb1f715fe36be -> Peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521): Couldn't send request to peer 1288ca0989104bbea36e22f38c374218 for tablet 19c33e54ee9e476796de7de42ba495b6. Status: Network error: failed to read from TLS socket (remote: 127.5.240.193:33521): Connection reset by peer (error 104). Retrying in the next heartbeat period. Already tried 1 times. I0703 01:43:46.025279 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 31319 I0703 01:43:46.025877 31435 generic_service.cc:169] Checking for leaks (request via RPC) W0703 01:43:46.244966 32433 consensus_peers.cc:433] T 19c33e54ee9e476796de7de42ba495b6 P d9dc68ace903455d809cb1f715fe36be -> Peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521): Couldn't send request to peer 1288ca0989104bbea36e22f38c374218 for tablet 19c33e54ee9e476796de7de42ba495b6. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.193:33521: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. I0703 01:43:46.323700 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 31319 W0703 01:43:46.365330 31135 connection.cc:512] server connection from 127.5.240.194:53448 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:53448): Connection reset by peer (error 104) W0703 01:43:46.365370 31448 connection.cc:512] client connection to 127.5.240.194:39827 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:39827): Connection reset by peer (error 104) W0703 01:43:46.365453 32696 connection.cc:512] client connection to 127.5.240.194:39827 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:39827): Connection reset by peer (error 104) W0703 01:43:46.365469 32432 connection.cc:512] client connection to 127.5.240.194:39827 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:39827): Connection reset by peer (error 104) W0703 01:43:46.365741 31448 connection.cc:512] server connection from 127.5.240.194:43677 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:43677): Connection reset by peer (error 104) W0703 01:43:46.365949 32563 connection.cc:512] client connection to 127.5.240.194:39827 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:39827): Connection reset by peer (error 104) W0703 01:43:46.366047 32565 connection.cc:512] server connection from 127.5.240.194:42817 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:42817): Connection reset by peer (error 104) W0703 01:43:46.366046 361 connection.cc:512] server connection from 127.5.240.194:39644 recv error: Network error: failed to read from TLS socket (remote: 0.0.0.0:0): Connection reset by peer (error 104) W0703 01:43:46.366046 358 connection.cc:512] client connection to 127.5.240.194:39827 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:39827): Connection reset by peer (error 104) W0703 01:43:46.366361 361 connection.cc:185] Error closing socket: Network error: TlsSocket::Close: Broken pipe W0703 01:43:46.366487 32699 connection.cc:512] server connection from 127.5.240.194:48735 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:48735): Connection reset by peer (error 104) W0703 01:43:46.366716 32434 connection.cc:512] server connection from 127.5.240.194:44944 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.194:44944): Connection reset by peer (error 104) I0703 01:43:46.366775 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 31441 I0703 01:43:46.367491 31557 generic_service.cc:169] Checking for leaks (request via RPC) W0703 01:43:46.502357 32697 consensus_peers.cc:433] T f9f1c40150da4144acb68186f9aad763 P 768d8de4b2ba40f180b18194c28be589 -> Peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521): Couldn't send request to peer 1288ca0989104bbea36e22f38c374218 for tablet f9f1c40150da4144acb68186f9aad763. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.193:33521: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. I0703 01:43:46.662457 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 31441 W0703 01:43:46.704754 31135 connection.cc:512] server connection from 127.5.240.195:41435 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.195:41435): Connection reset by peer (error 104) W0703 01:43:46.705178 359 connection.cc:512] server connection from 127.5.240.195:42756 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.195:42756): Connection reset by peer (error 104) W0703 01:43:46.705173 361 connection.cc:512] client connection to 127.5.240.195:59321 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.195:59321): Connection reset by peer (error 104) W0703 01:43:46.705219 32566 connection.cc:512] client connection to 127.5.240.195:59321 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.195:59321): Connection reset by peer (error 104) W0703 01:43:46.705219 32435 connection.cc:512] client connection to 127.5.240.195:59321 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.195:59321): Connection reset by peer (error 104) W0703 01:43:46.705669 32699 connection.cc:512] client connection to 127.5.240.195:59321 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.195:59321): Connection reset by peer (error 104) W0703 01:43:46.705767 32434 connection.cc:512] server connection from 127.5.240.195:47111 recv error: Network error: failed to read from TLS socket (remote: 0.0.0.0:0): Connection reset by peer (error 104) W0703 01:43:46.706030 32434 connection.cc:185] Error closing socket: Network error: TlsSocket::Close: Broken pipe W0703 01:43:46.706076 32698 connection.cc:512] server connection from 127.5.240.195:39799 recv error: Network error: failed to read from TLS socket (remote: 0.0.0.0:0): Connection reset by peer (error 104) W0703 01:43:46.706066 32564 connection.cc:512] server connection from 127.5.240.195:40750 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.195:40750): Connection reset by peer (error 104) W0703 01:43:46.706254 32698 connection.cc:185] Error closing socket: Network error: TlsSocket::Close: Broken pipe I0703 01:43:46.706276 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 32417 I0703 01:43:46.710636 32546 generic_service.cc:169] Checking for leaks (request via RPC) W0703 01:43:46.908563 32697 consensus_peers.cc:433] T f9f1c40150da4144acb68186f9aad763 P 768d8de4b2ba40f180b18194c28be589 -> Peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521): Couldn't send request to peer 1288ca0989104bbea36e22f38c374218 for tablet f9f1c40150da4144acb68186f9aad763. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.193:33521: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 3 times. W0703 01:43:46.948112 361 consensus_peers.cc:433] T bbe64aa4b0e04f468b035048629b969b P 56eaab17f812433a8f7880f0744e1744 -> Peer de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321): Couldn't send request to peer de93b58c2d67488dae12285e1806ebac for tablet bbe64aa4b0e04f468b035048629b969b. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.195:59321: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. I0703 01:43:46.951987 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 32417 W0703 01:43:46.988747 32565 connection.cc:512] client connection to 127.5.240.196:56543 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.196:56543): Connection reset by peer (error 104) W0703 01:43:46.988747 31136 connection.cc:512] server connection from 127.5.240.196:46282 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.196:46282): Connection reset by peer (error 104) W0703 01:43:46.988814 361 connection.cc:512] server connection from 127.5.240.196:34718 recv error: Network error: failed to read from TLS socket (remote: 0.0.0.0:0): Connection reset by peer (error 104) W0703 01:43:46.988997 361 connection.cc:185] Error closing socket: Network error: TlsSocket::Close: Broken pipe W0703 01:43:46.989130 360 connection.cc:512] client connection to 127.5.240.196:56543 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.196:56543): Connection reset by peer (error 104) W0703 01:43:46.989197 32696 connection.cc:512] server connection from 127.5.240.196:33749 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.196:33749): Connection reset by peer (error 104) W0703 01:43:46.989194 32564 connection.cc:512] server connection from 127.5.240.196:33323 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.196:33323): Connection reset by peer (error 104) W0703 01:43:46.989605 32698 connection.cc:512] client connection to 127.5.240.196:56543 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.196:56543): Connection reset by peer (error 104) I0703 01:43:46.989655 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 32553 I0703 01:43:46.993425 32679 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:43:47.191484 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 32553 W0703 01:43:47.221001 31136 connection.cc:512] server connection from 127.5.240.197:59728 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.197:59728): Connection reset by peer (error 104) W0703 01:43:47.221058 32699 connection.cc:512] server connection from 127.5.240.197:46298 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.197:46298): Connection reset by peer (error 104) I0703 01:43:47.221128 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 32686 W0703 01:43:47.221211 32696 connection.cc:512] client connection to 127.5.240.197:44326 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.197:44326): Connection reset by peer (error 104) I0703 01:43:47.225313 341 generic_service.cc:169] Checking for leaks (request via RPC) W0703 01:43:47.458145 32697 consensus_peers.cc:433] T f9f1c40150da4144acb68186f9aad763 P 768d8de4b2ba40f180b18194c28be589 -> Peer 1288ca0989104bbea36e22f38c374218 (127.5.240.193:33521): Couldn't send request to peer 1288ca0989104bbea36e22f38c374218 for tablet f9f1c40150da4144acb68186f9aad763. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.193:33521: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 4 times. W0703 01:43:47.458883 32696 consensus_peers.cc:433] T 1f01bfdde47b4d37a0f77d199bc7b4d2 P 768d8de4b2ba40f180b18194c28be589 -> Peer 3c1770123dc14bba80e00db5f182b585 (127.5.240.197:44326): Couldn't send request to peer 3c1770123dc14bba80e00db5f182b585 for tablet 1f01bfdde47b4d37a0f77d199bc7b4d2. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.197:44326: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. W0703 01:43:47.458890 32698 consensus_peers.cc:433] T a200f8a763f14d54a1e84082dbafce8f P 768d8de4b2ba40f180b18194c28be589 -> Peer d9dc68ace903455d809cb1f715fe36be (127.5.240.196:56543): Couldn't send request to peer d9dc68ace903455d809cb1f715fe36be for tablet a200f8a763f14d54a1e84082dbafce8f. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.196:56543: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 1 times. I0703 01:43:47.459237 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 32686 I0703 01:43:47.466341 3416 raft_consensus.cc:457] T 3d7a6e87780a49949cf7e024817cf3a3 P 56eaab17f812433a8f7880f0744e1744 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2f3fdb7c089640c0b936a574bc64707e) I0703 01:43:47.466418 3416 raft_consensus.cc:2730] T 3d7a6e87780a49949cf7e024817cf3a3 P 56eaab17f812433a8f7880f0744e1744: Snoozing failure detection for 1.965s (starting election) I0703 01:43:47.466501 3416 raft_consensus.cc:479] T 3d7a6e87780a49949cf7e024817cf3a3 P 56eaab17f812433a8f7880f0744e1744 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 438 OBSOLETE_local: false peers { permanent_uuid: "2f3fdb7c089640c0b936a574bc64707e" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 39827 } } peers { permanent_uuid: "56eaab17f812433a8f7880f0744e1744" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 57714 } } I0703 01:43:47.466886 3416 leader_election.cc:243] T 3d7a6e87780a49949cf7e024817cf3a3 P 56eaab17f812433a8f7880f0744e1744 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194:39827) W0703 01:43:47.467828 358 leader_election.cc:299] T 3d7a6e87780a49949cf7e024817cf3a3 P 56eaab17f812433a8f7880f0744e1744 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2f3fdb7c089640c0b936a574bc64707e (127.5.240.194:39827): Network error: Client connection negotiation failed: client connection to 127.5.240.194:39827: connect: Connection refused (error 111) I0703 01:43:47.467908 358 leader_election.cc:270] T 3d7a6e87780a49949cf7e024817cf3a3 P 56eaab17f812433a8f7880f0744e1744 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0703 01:43:47.468113 3416 raft_consensus.cc:2730] T 3d7a6e87780a49949cf7e024817cf3a3 P 56eaab17f812433a8f7880f0744e1744: Snoozing failure detection for 1.543s (election complete) I0703 01:43:47.468199 3416 raft_consensus.cc:2485] T 3d7a6e87780a49949cf7e024817cf3a3 P 56eaab17f812433a8f7880f0744e1744 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority W0703 01:43:47.477357 361 consensus_peers.cc:433] T bbe64aa4b0e04f468b035048629b969b P 56eaab17f812433a8f7880f0744e1744 -> Peer de93b58c2d67488dae12285e1806ebac (127.5.240.195:59321): Couldn't send request to peer de93b58c2d67488dae12285e1806ebac for tablet bbe64aa4b0e04f468b035048629b969b. Status: Network error: Client connection negotiation failed: client connection to 127.5.240.195:59321: connect: Connection refused (error 111). Retrying in the next heartbeat period. Already tried 2 times. W0703 01:43:47.495573 31135 connection.cc:512] server connection from 127.5.240.198:42191 recv error: Network error: failed to read from TLS socket (remote: 0.0.0.0:0): Connection reset by peer (error 104) I0703 01:43:47.495724 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 348 W0703 01:43:47.495839 31135 connection.cc:185] Error closing socket: Network error: TlsSocket::Close: Broken pipe I0703 01:43:47.499635 475 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:43:47.703444 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 348 I0703 01:43:47.736446 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master pid 31126 W0703 01:43:47.736449 31133 connection.cc:512] server connection from 127.5.240.199:38158 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.199:38158): Connection reset by peer (error 104) I0703 01:43:47.740339 31182 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:43:47.854329 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master with pid 31126 [ OK ] RebalanceParamTest.Rebalance/2 (69404 ms) [ RUN ] RebalanceParamTest.Rebalance/3 I0703 01:43:47.924424 6083 test_util.cc:208] Using random seed: 1342714303 I0703 01:43:47.925891 6083 ts_itest-base.cc:113] Starting cluster with: I0703 01:43:47.925914 6083 ts_itest-base.cc:114] -------------- I0703 01:43:47.925928 6083 ts_itest-base.cc:115] 7 tablet servers I0703 01:43:47.925962 6083 ts_itest-base.cc:116] 2 replicas per TS I0703 01:43:47.925983 6083 ts_itest-base.cc:117] -------------- I0703 01:43:47.927072 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:32866 --webserver_interface=127.5.240.254 --webserver_port=0 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --allow_unsafe_replication_factor --raft_prepare_replacement_before_eviction=true --tserver_unresponsive_timeout_ms=3000 --rpc_reuseport=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:43:48.115607 3427 flags.cc:406] Enabled unsafe flag: --allow_unsafe_replication_factor=true W0703 01:43:48.116623 3427 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:43:48.116705 3427 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:43:48.118928 3427 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0703 01:43:48.119005 3427 flags.cc:406] Enabled experimental flag: --ipki_ca_key_size=1024 W0703 01:43:48.119053 3427 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:43:48.119097 3427 flags.cc:406] Enabled experimental flag: --tsk_num_rsa_bits=512 W0703 01:43:48.119143 3427 flags.cc:406] Enabled experimental flag: --rpc_reuseport=true I0703 01:43:48.127771 3427 master_main.cc:73] Master server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal --allow_unsafe_replication_factor=true --ipki_ca_key_size=1024 --tserver_unresponsive_timeout_ms=3000 --ipki_server_key_size=1024 --tsk_num_rsa_bits=512 --rpc_bind_addresses=127.5.240.254:32866 --rpc_reuseport=true --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb --webserver_interface=127.5.240.254 --webserver_port=0 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/master-0/logs --logbuflevel=-1 --logtostderr=true Master server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:43:48.133344 3427 master_main.cc:80] Initializing master server... I0703 01:43:48.134047 3427 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 267708us I0703 01:43:48.135098 3427 fs_manager.cc:261] Metadata directory not provided I0703 01:43:48.135162 3427 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:43:48.135470 3427 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:43:48.135558 3427 server_base.cc:433] Attempting to create new FS layout instead I0703 01:43:48.139842 3427 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/instance: uuid: "c27ecf301629438ca9a6c9698194e12b" format_stamp: "Formatted at 2018-07-03 08:43:48 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:43:48.140753 3427 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal/instance: uuid: "c27ecf301629438ca9a6c9698194e12b" format_stamp: "Formatted at 2018-07-03 08:43:48 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:43:48.148766 3427 fs_manager.cc:501] Time spent creating directory manager: real 0.008s user 0.003s sys 0.004s I0703 01:43:48.149962 3427 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:43:48.150347 3427 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:43:48.153969 3427 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s I0703 01:43:48.154028 3427 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal uuid: "c27ecf301629438ca9a6c9698194e12b" format_stamp: "Formatted at 2018-07-03 08:43:48 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:43:48.154140 3427 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/master-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:43:48.234433 3427 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:43:48.235378 3427 master_main.cc:83] Starting Master server... I0703 01:43:48.272804 3427 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.254:32866 I0703 01:43:48.273272 3427 webserver.cc:175] Starting webserver on 127.5.240.254:0 I0703 01:43:48.273320 3427 webserver.cc:186] Document root disabled I0703 01:43:48.274560 3427 webserver.cc:313] Webserver started. Bound to: http://127.5.240.254:43113/ I0703 01:43:48.275143 3427 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/master-0/data/info.pb I0703 01:43:48.277155 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-master as pid 3427 I0703 01:43:48.277796 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data --block_manager=log --rpc_bind_addresses=127.5.240.193:0 --local_ip_for_outbound_sockets=127.5.240.193 --webserver_interface=127.5.240.193 --webserver_port=0 --tserver_master_addrs=127.5.240.254:32866 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0703 01:43:48.278844 3486 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:43:48.290184 3486 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P c27ecf301629438ca9a6c9698194e12b: Bootstrap starting. I0703 01:43:48.292208 3486 tablet_bootstrap.cc:578] T 00000000000000000000000000000000 P c27ecf301629438ca9a6c9698194e12b: No blocks or log segments found. Creating new log. I0703 01:43:48.292951 3486 log.cc:525] T 00000000000000000000000000000000 P c27ecf301629438ca9a6c9698194e12b: Log is configured to *not* fsync() on all Append() calls I0703 01:43:48.294698 3486 tablet_bootstrap.cc:427] T 00000000000000000000000000000000 P c27ecf301629438ca9a6c9698194e12b: No bootstrap required, opened a new log I0703 01:43:48.300956 3486 raft_consensus.cc:325] T 00000000000000000000000000000000 P c27ecf301629438ca9a6c9698194e12b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c27ecf301629438ca9a6c9698194e12b" member_type: VOTER } I0703 01:43:48.301142 3486 raft_consensus.cc:351] T 00000000000000000000000000000000 P c27ecf301629438ca9a6c9698194e12b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:43:48.301201 3486 raft_consensus.cc:628] T 00000000000000000000000000000000 P c27ecf301629438ca9a6c9698194e12b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c27ecf301629438ca9a6c9698194e12b, State: Initialized, Role: FOLLOWER I0703 01:43:48.301687 3486 consensus_queue.cc:227] T 00000000000000000000000000000000 P c27ecf301629438ca9a6c9698194e12b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c27ecf301629438ca9a6c9698194e12b" member_type: VOTER } I0703 01:43:48.301837 3486 raft_consensus.cc:365] T 00000000000000000000000000000000 P c27ecf301629438ca9a6c9698194e12b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately I0703 01:43:48.301920 3486 raft_consensus.cc:457] T 00000000000000000000000000000000 P c27ecf301629438ca9a6c9698194e12b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration) I0703 01:43:48.302008 3486 raft_consensus.cc:2730] T 00000000000000000000000000000000 P c27ecf301629438ca9a6c9698194e12b: Snoozing failure detection for 2.212s (starting election) I0703 01:43:48.302098 3486 raft_consensus.cc:2782] T 00000000000000000000000000000000 P c27ecf301629438ca9a6c9698194e12b [term 0 FOLLOWER]: Advancing to term 1 I0703 01:43:48.303478 3486 raft_consensus.cc:479] T 00000000000000000000000000000000 P c27ecf301629438ca9a6c9698194e12b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c27ecf301629438ca9a6c9698194e12b" member_type: VOTER } I0703 01:43:48.303863 3486 leader_election.cc:270] T 00000000000000000000000000000000 P c27ecf301629438ca9a6c9698194e12b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:43:48.305346 3489 raft_consensus.cc:2730] T 00000000000000000000000000000000 P c27ecf301629438ca9a6c9698194e12b: Snoozing failure detection for 2.054s (election complete) I0703 01:43:48.305842 3489 raft_consensus.cc:2537] T 00000000000000000000000000000000 P c27ecf301629438ca9a6c9698194e12b [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:43:48.307224 3489 raft_consensus.cc:593] T 00000000000000000000000000000000 P c27ecf301629438ca9a6c9698194e12b [term 1 LEADER]: Becoming Leader. State: Replica: c27ecf301629438ca9a6c9698194e12b, State: Running, Role: LEADER I0703 01:43:48.307631 3489 consensus_queue.cc:204] T 00000000000000000000000000000000 P c27ecf301629438ca9a6c9698194e12b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c27ecf301629438ca9a6c9698194e12b" member_type: VOTER } I0703 01:43:48.310215 3490 sys_catalog.cc:337] T 00000000000000000000000000000000 P c27ecf301629438ca9a6c9698194e12b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c27ecf301629438ca9a6c9698194e12b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c27ecf301629438ca9a6c9698194e12b" member_type: VOTER } } I0703 01:43:48.310847 3490 sys_catalog.cc:340] T 00000000000000000000000000000000 P c27ecf301629438ca9a6c9698194e12b [sys.catalog]: This master's current role is: LEADER I0703 01:43:48.310317 3486 sys_catalog.cc:337] T 00000000000000000000000000000000 P c27ecf301629438ca9a6c9698194e12b [sys.catalog]: SysCatalogTable state changed. Reason: Started TabletReplica. Latest consensus state: current_term: 1 leader_uuid: "c27ecf301629438ca9a6c9698194e12b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c27ecf301629438ca9a6c9698194e12b" member_type: VOTER } } I0703 01:43:48.311293 3486 sys_catalog.cc:340] T 00000000000000000000000000000000 P c27ecf301629438ca9a6c9698194e12b [sys.catalog]: This master's current role is: LEADER I0703 01:43:48.310685 3491 sys_catalog.cc:337] T 00000000000000000000000000000000 P c27ecf301629438ca9a6c9698194e12b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c27ecf301629438ca9a6c9698194e12b. Latest consensus state: current_term: 1 leader_uuid: "c27ecf301629438ca9a6c9698194e12b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c27ecf301629438ca9a6c9698194e12b" member_type: VOTER } } I0703 01:43:48.311936 3491 sys_catalog.cc:340] T 00000000000000000000000000000000 P c27ecf301629438ca9a6c9698194e12b [sys.catalog]: This master's current role is: LEADER I0703 01:43:48.313343 3486 sys_catalog.cc:424] T 00000000000000000000000000000000 P c27ecf301629438ca9a6c9698194e12b [sys.catalog]: configured and running, proceeding with master startup. I0703 01:43:48.314791 3427 master_main.cc:86] Master server successfully started. W0703 01:43:48.320056 3494 catalog_manager.cc:1064] T 00000000000000000000000000000000 P c27ecf301629438ca9a6c9698194e12b: acquiring CA information for follower catalog manager: Not found: root CA entry not found W0703 01:43:48.320199 3494 catalog_manager.cc:563] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry I0703 01:43:48.324066 3493 catalog_manager.cc:1006] Loading table and tablet metadata into memory... I0703 01:43:48.327322 3493 catalog_manager.cc:1017] Initializing Kudu internal certificate authority... I0703 01:43:48.369261 3493 catalog_manager.cc:904] Generated new certificate authority record I0703 01:43:48.371301 3493 catalog_manager.cc:1026] Loading token signing keys... I0703 01:43:48.377862 3493 catalog_manager.cc:4102] T 00000000000000000000000000000000 P c27ecf301629438ca9a6c9698194e12b: Generated new TSK 0 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:43:48.449868 3487 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:43:48.450881 3487 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:43:48.452901 3487 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0703 01:43:48.452965 3487 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:43:48.453071 3487 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.193 I0703 01:43:48.460444 3487 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.193:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb --webserver_interface=127.5.240.193 --webserver_port=0 --tserver_master_addrs=127.5.240.254:32866 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.193 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:43:48.467070 3487 tablet_server_main.cc:85] Initializing tablet server... I0703 01:43:48.468607 3487 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 267708us I0703 01:43:48.469167 3487 fs_manager.cc:261] Metadata directory not provided I0703 01:43:48.469233 3487 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:43:48.469847 3487 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:43:48.469916 3487 server_base.cc:433] Attempting to create new FS layout instead I0703 01:43:48.473182 3487 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/instance: uuid: "55223945fe3d4dd8ac03ba9dab346f54" format_stamp: "Formatted at 2018-07-03 08:43:48 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:43:48.473678 3487 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal/instance: uuid: "55223945fe3d4dd8ac03ba9dab346f54" format_stamp: "Formatted at 2018-07-03 08:43:48 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:43:48.481827 3487 fs_manager.cc:501] Time spent creating directory manager: real 0.008s user 0.003s sys 0.004s I0703 01:43:48.483032 3487 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:43:48.483402 3487 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:43:48.487844 3487 fs_manager.cc:417] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s I0703 01:43:48.487921 3487 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal uuid: "55223945fe3d4dd8ac03ba9dab346f54" format_stamp: "Formatted at 2018-07-03 08:43:48 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:43:48.488049 3487 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:43:48.538092 3487 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:43:48.540220 3487 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:43:48.540627 3487 tablet_server_main.cc:90] Starting tablet server... I0703 01:43:48.655318 3487 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.193:54919 I0703 01:43:48.655808 3487 webserver.cc:175] Starting webserver on 127.5.240.193:0 I0703 01:43:48.655854 3487 webserver.cc:186] Document root disabled I0703 01:43:48.656661 3487 webserver.cc:313] Webserver started. Bound to: http://127.5.240.193:43005/ I0703 01:43:48.657187 3487 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/data/info.pb I0703 01:43:48.659790 3487 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:43:48.661561 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 3487 I0703 01:43:48.662128 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data --block_manager=log --rpc_bind_addresses=127.5.240.194:0 --local_ip_for_outbound_sockets=127.5.240.194 --webserver_interface=127.5.240.194 --webserver_port=0 --tserver_master_addrs=127.5.240.254:32866 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0703 01:43:48.678661 3616 heartbeater.cc:346] Connected to a master server at 127.5.240.254:32866 I0703 01:43:48.678872 3616 heartbeater.cc:426] Registering TS with master... I0703 01:43:48.679419 3616 heartbeater.cc:475] Master 127.5.240.254:32866 requested a full tablet report, sending... I0703 01:43:48.682929 3453 ts_manager.cc:98] Registered new tserver with Master: 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193:54919) I0703 01:43:48.685433 3453 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.193:47589 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:43:48.829978 3620 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:43:48.830988 3620 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:43:48.832969 3620 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0703 01:43:48.833034 3620 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:43:48.833150 3620 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.194 I0703 01:43:48.840389 3620 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.194:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb --webserver_interface=127.5.240.194 --webserver_port=0 --tserver_master_addrs=127.5.240.254:32866 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.194 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:43:48.846920 3620 tablet_server_main.cc:85] Initializing tablet server... I0703 01:43:48.848410 3620 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 267708us I0703 01:43:48.848980 3620 fs_manager.cc:261] Metadata directory not provided I0703 01:43:48.849050 3620 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:43:48.849277 3620 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:43:48.849345 3620 server_base.cc:433] Attempting to create new FS layout instead I0703 01:43:48.852560 3620 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/instance: uuid: "57bea6dc182c41a78304417a3cbfc4a7" format_stamp: "Formatted at 2018-07-03 08:43:48 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:43:48.853116 3620 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal/instance: uuid: "57bea6dc182c41a78304417a3cbfc4a7" format_stamp: "Formatted at 2018-07-03 08:43:48 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:43:48.860447 3620 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.003s sys 0.003s I0703 01:43:48.861668 3620 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:43:48.862066 3620 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:43:48.865656 3620 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s I0703 01:43:48.865707 3620 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal uuid: "57bea6dc182c41a78304417a3cbfc4a7" format_stamp: "Formatted at 2018-07-03 08:43:48 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:43:48.865836 3620 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:43:48.893975 3620 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:43:48.895866 3620 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:43:48.896229 3620 tablet_server_main.cc:90] Starting tablet server... I0703 01:43:48.981416 3620 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.194:59746 I0703 01:43:48.981881 3620 webserver.cc:175] Starting webserver on 127.5.240.194:0 I0703 01:43:48.981925 3620 webserver.cc:186] Document root disabled I0703 01:43:48.983093 3620 webserver.cc:313] Webserver started. Bound to: http://127.5.240.194:57372/ I0703 01:43:48.983650 3620 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/data/info.pb I0703 01:43:48.985421 3620 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:43:48.993795 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 3620 I0703 01:43:48.994436 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data --block_manager=log --rpc_bind_addresses=127.5.240.195:0 --local_ip_for_outbound_sockets=127.5.240.195 --webserver_interface=127.5.240.195 --webserver_port=0 --tserver_master_addrs=127.5.240.254:32866 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0703 01:43:48.997560 3739 heartbeater.cc:346] Connected to a master server at 127.5.240.254:32866 I0703 01:43:48.997740 3739 heartbeater.cc:426] Registering TS with master... I0703 01:43:48.998205 3739 heartbeater.cc:475] Master 127.5.240.254:32866 requested a full tablet report, sending... I0703 01:43:48.999251 3453 ts_manager.cc:98] Registered new tserver with Master: 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746) I0703 01:43:49.000099 3453 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.194:33875 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:43:49.161389 3742 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:43:49.162384 3742 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:43:49.164360 3742 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0703 01:43:49.164419 3742 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:43:49.164507 3742 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.195 I0703 01:43:49.171833 3742 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.195:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb --webserver_interface=127.5.240.195 --webserver_port=0 --tserver_master_addrs=127.5.240.254:32866 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.195 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:43:49.178509 3742 tablet_server_main.cc:85] Initializing tablet server... I0703 01:43:49.180027 3742 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 268208us I0703 01:43:49.180562 3742 fs_manager.cc:261] Metadata directory not provided I0703 01:43:49.180645 3742 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:43:49.181229 3742 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:43:49.181291 3742 server_base.cc:433] Attempting to create new FS layout instead I0703 01:43:49.184550 3742 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/instance: uuid: "8c1e8943f5654f71b030ec7c0b733742" format_stamp: "Formatted at 2018-07-03 08:43:49 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:43:49.185096 3742 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal/instance: uuid: "8c1e8943f5654f71b030ec7c0b733742" format_stamp: "Formatted at 2018-07-03 08:43:49 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:43:49.192762 3742 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.003s sys 0.004s I0703 01:43:49.193946 3742 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:43:49.194317 3742 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:43:49.197548 3742 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0703 01:43:49.197597 3742 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal uuid: "8c1e8943f5654f71b030ec7c0b733742" format_stamp: "Formatted at 2018-07-03 08:43:49 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:43:49.197734 3742 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:43:49.236479 3742 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:43:49.238566 3742 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:43:49.238979 3742 tablet_server_main.cc:90] Starting tablet server... I0703 01:43:49.323670 3742 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.195:51064 I0703 01:43:49.324121 3742 webserver.cc:175] Starting webserver on 127.5.240.195:0 I0703 01:43:49.324160 3742 webserver.cc:186] Document root disabled I0703 01:43:49.324895 3742 webserver.cc:313] Webserver started. Bound to: http://127.5.240.195:38335/ I0703 01:43:49.325458 3742 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/data/info.pb I0703 01:43:49.327450 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 3742 I0703 01:43:49.328193 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.5.240.196:0 --local_ip_for_outbound_sockets=127.5.240.196 --webserver_interface=127.5.240.196 --webserver_port=0 --tserver_master_addrs=127.5.240.254:32866 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0703 01:43:49.336401 3742 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:43:49.348279 3861 heartbeater.cc:346] Connected to a master server at 127.5.240.254:32866 I0703 01:43:49.348426 3861 heartbeater.cc:426] Registering TS with master... I0703 01:43:49.348857 3861 heartbeater.cc:475] Master 127.5.240.254:32866 requested a full tablet report, sending... I0703 01:43:49.349895 3453 ts_manager.cc:98] Registered new tserver with Master: 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064) I0703 01:43:49.350759 3453 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.195:48897 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:43:49.498827 3864 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:43:49.499819 3864 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:43:49.501827 3864 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0703 01:43:49.501876 3864 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:43:49.501961 3864 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.196 I0703 01:43:49.509351 3864 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.196:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb --webserver_interface=127.5.240.196 --webserver_port=0 --tserver_master_addrs=127.5.240.254:32866 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.196 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:43:49.515902 3864 tablet_server_main.cc:85] Initializing tablet server... I0703 01:43:49.517395 3864 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 268208us I0703 01:43:49.517947 3864 fs_manager.cc:261] Metadata directory not provided I0703 01:43:49.518010 3864 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:43:49.518592 3864 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:43:49.518671 3864 server_base.cc:433] Attempting to create new FS layout instead I0703 01:43:49.521884 3864 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/instance: uuid: "213596374e4142dca6ecfc701ba1c566" format_stamp: "Formatted at 2018-07-03 08:43:49 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:43:49.522387 3864 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal/instance: uuid: "213596374e4142dca6ecfc701ba1c566" format_stamp: "Formatted at 2018-07-03 08:43:49 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:43:49.530117 3864 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s I0703 01:43:49.531297 3864 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:43:49.531683 3864 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:43:49.534750 3864 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s I0703 01:43:49.534795 3864 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal uuid: "213596374e4142dca6ecfc701ba1c566" format_stamp: "Formatted at 2018-07-03 08:43:49 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:43:49.534904 3864 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:43:49.567898 3864 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:43:49.569733 3864 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:43:49.570051 3864 tablet_server_main.cc:90] Starting tablet server... I0703 01:43:49.656946 3864 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.196:51288 I0703 01:43:49.657338 3864 webserver.cc:175] Starting webserver on 127.5.240.196:0 I0703 01:43:49.657377 3864 webserver.cc:186] Document root disabled I0703 01:43:49.658076 3864 webserver.cc:313] Webserver started. Bound to: http://127.5.240.196:38993/ I0703 01:43:49.658612 3864 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb I0703 01:43:49.660975 3864 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:43:49.664206 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 3864 I0703 01:43:49.665011 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data --block_manager=log --rpc_bind_addresses=127.5.240.197:0 --local_ip_for_outbound_sockets=127.5.240.197 --webserver_interface=127.5.240.197 --webserver_port=0 --tserver_master_addrs=127.5.240.254:32866 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0703 01:43:49.674165 4008 heartbeater.cc:346] Connected to a master server at 127.5.240.254:32866 I0703 01:43:49.674345 4008 heartbeater.cc:426] Registering TS with master... I0703 01:43:49.674772 4008 heartbeater.cc:475] Master 127.5.240.254:32866 requested a full tablet report, sending... I0703 01:43:49.675770 3453 ts_manager.cc:98] Registered new tserver with Master: 213596374e4142dca6ecfc701ba1c566 (127.5.240.196:51288) I0703 01:43:49.676645 3453 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.196:42932 I0703 01:43:49.688081 3616 heartbeater.cc:467] Master 127.5.240.254:32866 was elected leader, sending a full tablet report... WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:43:49.835789 4011 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:43:49.836781 4011 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:43:49.838831 4011 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0703 01:43:49.838893 4011 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:43:49.838995 4011 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.197 I0703 01:43:49.845969 4011 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.197:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb --webserver_interface=127.5.240.197 --webserver_port=0 --tserver_master_addrs=127.5.240.254:32866 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.197 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:43:49.852566 4011 tablet_server_main.cc:85] Initializing tablet server... I0703 01:43:49.854089 4011 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 268208us I0703 01:43:49.854666 4011 fs_manager.cc:261] Metadata directory not provided I0703 01:43:49.854733 4011 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:43:49.855334 4011 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:43:49.855401 4011 server_base.cc:433] Attempting to create new FS layout instead I0703 01:43:49.858661 4011 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/instance: uuid: "aa8b9c58b466436faa5086e85b5224a7" format_stamp: "Formatted at 2018-07-03 08:43:49 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:43:49.859182 4011 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal/instance: uuid: "aa8b9c58b466436faa5086e85b5224a7" format_stamp: "Formatted at 2018-07-03 08:43:49 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:43:49.866550 4011 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.004s sys 0.002s I0703 01:43:49.867753 4011 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:43:49.868115 4011 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:43:49.871660 4011 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s I0703 01:43:49.871711 4011 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal uuid: "aa8b9c58b466436faa5086e85b5224a7" format_stamp: "Formatted at 2018-07-03 08:43:49 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:43:49.871834 4011 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:43:49.918869 4011 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:43:49.920724 4011 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:43:49.921047 4011 tablet_server_main.cc:90] Starting tablet server... I0703 01:43:50.002295 3739 heartbeater.cc:467] Master 127.5.240.254:32866 was elected leader, sending a full tablet report... I0703 01:43:50.007730 4011 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.197:51329 I0703 01:43:50.008141 4011 webserver.cc:175] Starting webserver on 127.5.240.197:0 I0703 01:43:50.008185 4011 webserver.cc:186] Document root disabled I0703 01:43:50.008900 4011 webserver.cc:313] Webserver started. Bound to: http://127.5.240.197:36989/ I0703 01:43:50.009408 4011 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb I0703 01:43:50.010658 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 4011 I0703 01:43:50.011389 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data --block_manager=log --rpc_bind_addresses=127.5.240.198:0 --local_ip_for_outbound_sockets=127.5.240.198 --webserver_interface=127.5.240.198 --webserver_port=0 --tserver_master_addrs=127.5.240.254:32866 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0703 01:43:50.024863 4130 heartbeater.cc:346] Connected to a master server at 127.5.240.254:32866 I0703 01:43:50.025028 4130 heartbeater.cc:426] Registering TS with master... I0703 01:43:50.025534 4130 heartbeater.cc:475] Master 127.5.240.254:32866 requested a full tablet report, sending... I0703 01:43:50.026334 4011 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:43:50.026906 3453 ts_manager.cc:98] Registered new tserver with Master: aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197:51329) I0703 01:43:50.028106 3453 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.197:39911 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:43:50.184247 4133 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:43:50.185238 4133 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:43:50.187198 4133 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0703 01:43:50.187247 4133 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:43:50.187331 4133 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.198 I0703 01:43:50.194674 4133 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.198:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data/info.pb --webserver_interface=127.5.240.198 --webserver_port=0 --tserver_master_addrs=127.5.240.254:32866 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.198 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:43:50.201246 4133 tablet_server_main.cc:85] Initializing tablet server... I0703 01:43:50.202735 4133 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 268708us I0703 01:43:50.203274 4133 fs_manager.cc:261] Metadata directory not provided I0703 01:43:50.203332 4133 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:43:50.203536 4133 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:43:50.203608 4133 server_base.cc:433] Attempting to create new FS layout instead I0703 01:43:50.206849 4133 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data/instance: uuid: "6932f1cd716a48fe8d6c5573acfc48bf" format_stamp: "Formatted at 2018-07-03 08:43:50 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:43:50.207355 4133 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/wal/instance: uuid: "6932f1cd716a48fe8d6c5573acfc48bf" format_stamp: "Formatted at 2018-07-03 08:43:50 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:43:50.214612 4133 fs_manager.cc:501] Time spent creating directory manager: real 0.007s user 0.004s sys 0.003s I0703 01:43:50.215801 4133 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:43:50.216153 4133 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:43:50.219719 4133 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s I0703 01:43:50.219764 4133 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/wal uuid: "6932f1cd716a48fe8d6c5573acfc48bf" format_stamp: "Formatted at 2018-07-03 08:43:50 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:43:50.219885 4133 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:43:50.280100 4133 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:43:50.281989 4133 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:43:50.282312 4133 tablet_server_main.cc:90] Starting tablet server... I0703 01:43:50.352677 3861 heartbeater.cc:467] Master 127.5.240.254:32866 was elected leader, sending a full tablet report... I0703 01:43:50.368263 4133 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.198:51033 I0703 01:43:50.368729 4133 webserver.cc:175] Starting webserver on 127.5.240.198:0 I0703 01:43:50.368768 4133 webserver.cc:186] Document root disabled I0703 01:43:50.369442 4133 webserver.cc:313] Webserver started. Bound to: http://127.5.240.198:39793/ I0703 01:43:50.369974 4133 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data/info.pb I0703 01:43:50.371237 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 4133 I0703 01:43:50.371978 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data --block_manager=log --rpc_bind_addresses=127.5.240.199:0 --local_ip_for_outbound_sockets=127.5.240.199 --webserver_interface=127.5.240.199 --webserver_port=0 --tserver_master_addrs=127.5.240.254:32866 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0703 01:43:50.378887 4133 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:43:50.387390 4253 heartbeater.cc:346] Connected to a master server at 127.5.240.254:32866 I0703 01:43:50.387560 4253 heartbeater.cc:426] Registering TS with master... I0703 01:43:50.387993 4253 heartbeater.cc:475] Master 127.5.240.254:32866 requested a full tablet report, sending... I0703 01:43:50.389003 3453 ts_manager.cc:98] Registered new tserver with Master: 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198:51033) I0703 01:43:50.389863 3453 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.198:41360 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:43:50.541414 4256 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:43:50.542408 4256 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:43:50.544469 4256 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0703 01:43:50.544519 4256 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:43:50.544625 4256 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.199 I0703 01:43:50.551558 4256 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.199:0 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data/info.pb --webserver_interface=127.5.240.199 --webserver_port=0 --tserver_master_addrs=127.5.240.254:32866 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.199 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:43:50.558284 4256 tablet_server_main.cc:85] Initializing tablet server... I0703 01:43:50.559804 4256 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 268708us I0703 01:43:50.560353 4256 fs_manager.cc:261] Metadata directory not provided I0703 01:43:50.560410 4256 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:43:50.561033 4256 server_base.cc:432] Could not load existing FS layout: Not found: could not find a healthy instance file I0703 01:43:50.561096 4256 server_base.cc:433] Attempting to create new FS layout instead I0703 01:43:50.564347 4256 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data/instance: uuid: "1157c91cc6f54a68a1d99017ac8444aa" format_stamp: "Formatted at 2018-07-03 08:43:50 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:43:50.564890 4256 fs_manager.cc:601] Generated new instance metadata in path /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/wal/instance: uuid: "1157c91cc6f54a68a1d99017ac8444aa" format_stamp: "Formatted at 2018-07-03 08:43:50 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:43:50.572711 4256 fs_manager.cc:501] Time spent creating directory manager: real 0.008s user 0.003s sys 0.004s I0703 01:43:50.573928 4256 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:43:50.574306 4256 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:43:50.577878 4256 fs_manager.cc:417] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s I0703 01:43:50.577922 4256 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/wal uuid: "1157c91cc6f54a68a1d99017ac8444aa" format_stamp: "Formatted at 2018-07-03 08:43:50 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:43:50.578042 4256 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:43:50.612323 4256 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:43:50.614172 4256 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:43:50.614475 4256 tablet_server_main.cc:90] Starting tablet server... I0703 01:43:50.678500 4008 heartbeater.cc:467] Master 127.5.240.254:32866 was elected leader, sending a full tablet report... I0703 01:43:50.701159 4256 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.199:55377 I0703 01:43:50.701563 4256 webserver.cc:175] Starting webserver on 127.5.240.199:0 I0703 01:43:50.701601 4256 webserver.cc:186] Document root disabled I0703 01:43:50.702286 4256 webserver.cc:313] Webserver started. Bound to: http://127.5.240.199:38435/ I0703 01:43:50.702826 4256 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data/info.pb I0703 01:43:50.705163 4256 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:43:50.706778 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 4256 I0703 01:43:50.717025 4378 heartbeater.cc:346] Connected to a master server at 127.5.240.254:32866 I0703 01:43:50.717164 4378 heartbeater.cc:426] Registering TS with master... I0703 01:43:50.717562 4378 heartbeater.cc:475] Master 127.5.240.254:32866 requested a full tablet report, sending... I0703 01:43:50.718657 3453 ts_manager.cc:98] Registered new tserver with Master: 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377) I0703 01:43:50.719619 3453 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.199:44240 I0703 01:43:50.721207 6083 external_mini_cluster.cc:490] 7 TS(s) registered with all masters I0703 01:43:50.731644 3453 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.5.240.254:58445: name: "TestTable" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 2 split_rows_range_bounds { } partition_schema { range_schema { columns { name: "key" } } } I0703 01:43:50.757166 3798 tablet_service.cc:759] Processing CreateTablet for tablet dc02a796568c4677a2eebfacae32fc38 (table=TestTable [id=35e213a216c74c5b815f1702a16a7c2e]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:43:50.757864 3798 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet dc02a796568c4677a2eebfacae32fc38. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:43:50.757740 4312 tablet_service.cc:759] Processing CreateTablet for tablet dc02a796568c4677a2eebfacae32fc38 (table=TestTable [id=35e213a216c74c5b815f1702a16a7c2e]), partition=RANGE (key) PARTITION UNBOUNDED I0703 01:43:50.758289 4312 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet dc02a796568c4677a2eebfacae32fc38. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:43:50.767407 4312 ts_tablet_manager.cc:1173] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa: Registered tablet (data state: TABLET_DATA_READY) I0703 01:43:50.767866 3798 ts_tablet_manager.cc:1173] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742: Registered tablet (data state: TABLET_DATA_READY) I0703 01:43:50.768612 4397 ts_tablet_manager.cc:1028] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742: Bootstrapping tablet I0703 01:43:50.768818 4396 ts_tablet_manager.cc:1028] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa: Bootstrapping tablet I0703 01:43:50.769109 4397 tablet_bootstrap.cc:427] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742: Bootstrap starting. I0703 01:43:50.769518 4396 tablet_bootstrap.cc:427] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa: Bootstrap starting. I0703 01:43:50.771225 4397 tablet_bootstrap.cc:578] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742: No blocks or log segments found. Creating new log. I0703 01:43:50.771975 4397 log.cc:525] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742: Log is configured to *not* fsync() on all Append() calls I0703 01:43:50.772387 4396 tablet_bootstrap.cc:578] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa: No blocks or log segments found. Creating new log. I0703 01:43:50.773092 4396 log.cc:525] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa: Log is configured to *not* fsync() on all Append() calls I0703 01:43:50.774446 4397 tablet_bootstrap.cc:427] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742: No bootstrap required, opened a new log I0703 01:43:50.774462 4396 tablet_bootstrap.cc:427] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa: No bootstrap required, opened a new log I0703 01:43:50.774634 4397 ts_tablet_manager.cc:1045] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.002s I0703 01:43:50.774636 4396 ts_tablet_manager.cc:1045] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s I0703 01:43:50.780261 4396 raft_consensus.cc:325] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } } peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } I0703 01:43:50.780519 4396 raft_consensus.cc:351] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:43:50.780347 4397 raft_consensus.cc:325] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } } peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } I0703 01:43:50.780619 4397 raft_consensus.cc:351] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:43:50.780616 4396 raft_consensus.cc:628] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1157c91cc6f54a68a1d99017ac8444aa, State: Initialized, Role: FOLLOWER I0703 01:43:50.780679 4397 raft_consensus.cc:628] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c1e8943f5654f71b030ec7c0b733742, State: Initialized, Role: FOLLOWER I0703 01:43:50.781087 4396 consensus_queue.cc:227] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } } peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } I0703 01:43:50.781062 4397 consensus_queue.cc:227] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } } peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } I0703 01:43:50.782455 4378 heartbeater.cc:467] Master 127.5.240.254:32866 was elected leader, sending a full tablet report... I0703 01:43:50.782899 4396 ts_tablet_manager.cc:1073] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa: Time spent starting tablet: real 0.008s user 0.007s sys 0.001s I0703 01:43:50.783594 4397 ts_tablet_manager.cc:1073] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742: Time spent starting tablet: real 0.009s user 0.005s sys 0.003s W0703 01:43:50.837525 3862 tablet.cc:1678] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742: Can't schedule compaction. Clean time has not been advanced past its initial value. W0703 01:43:50.955351 4379 tablet.cc:1678] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:43:51.029961 4130 heartbeater.cc:467] Master 127.5.240.254:32866 was elected leader, sending a full tablet report... I0703 01:43:51.163575 4400 raft_consensus.cc:457] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0703 01:43:51.163691 4400 raft_consensus.cc:2730] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa: Snoozing failure detection for 2.006s (starting election) I0703 01:43:51.163820 4400 raft_consensus.cc:479] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } } peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } I0703 01:43:51.164913 4400 leader_election.cc:243] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064) I0703 01:43:51.169966 3818 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "dc02a796568c4677a2eebfacae32fc38" candidate_uuid: "1157c91cc6f54a68a1d99017ac8444aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c1e8943f5654f71b030ec7c0b733742" is_pre_election: true I0703 01:43:51.170627 3818 raft_consensus.cc:2730] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742: Snoozing failure detection for 2.148s (vote granted) I0703 01:43:51.170724 3818 raft_consensus.cc:2243] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1157c91cc6f54a68a1d99017ac8444aa in term 0. I0703 01:43:51.171281 4266 leader_election.cc:379] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa [CANDIDATE]: Term 1 pre-election: Vote granted by peer 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064) I0703 01:43:51.171345 4266 leader_election.cc:270] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0703 01:43:51.171630 4400 raft_consensus.cc:2730] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa: Snoozing failure detection for 1.857s (election complete) I0703 01:43:51.171738 4400 raft_consensus.cc:2537] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa [term 0 FOLLOWER]: Leader pre-election won for term 1 I0703 01:43:51.171798 4400 raft_consensus.cc:457] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0703 01:43:51.171838 4400 raft_consensus.cc:2730] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa: Snoozing failure detection for 1.721s (starting election) I0703 01:43:51.171885 4400 raft_consensus.cc:2782] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa [term 0 FOLLOWER]: Advancing to term 1 I0703 01:43:51.173537 4400 raft_consensus.cc:479] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } } peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } I0703 01:43:51.173800 4400 leader_election.cc:243] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa [CANDIDATE]: Term 1 election: Requesting vote from peer 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064) I0703 01:43:51.174362 3818 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "dc02a796568c4677a2eebfacae32fc38" candidate_uuid: "1157c91cc6f54a68a1d99017ac8444aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c1e8943f5654f71b030ec7c0b733742" I0703 01:43:51.174474 3818 raft_consensus.cc:2782] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:43:51.174561 3818 raft_consensus.cc:2730] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742: Snoozing failure detection for 1.850s (vote granted) I0703 01:43:51.176285 3818 raft_consensus.cc:2243] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1157c91cc6f54a68a1d99017ac8444aa in term 1. I0703 01:43:51.176672 4266 leader_election.cc:379] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa [CANDIDATE]: Term 1 election: Vote granted by peer 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064) I0703 01:43:51.176722 4266 leader_election.cc:270] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:43:51.176887 4400 raft_consensus.cc:2730] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa: Snoozing failure detection for 1.634s (election complete) I0703 01:43:51.176949 4400 raft_consensus.cc:2537] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:43:51.177291 4400 raft_consensus.cc:593] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa [term 1 LEADER]: Becoming Leader. State: Replica: 1157c91cc6f54a68a1d99017ac8444aa, State: Running, Role: LEADER I0703 01:43:51.177614 4400 consensus_queue.cc:204] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } } peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } I0703 01:43:51.181457 3453 catalog_manager.cc:3798] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa reported cstate change: term changed from 0 to 1, leader changed from to 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199). New cstate: current_term: 1 leader_uuid: "1157c91cc6f54a68a1d99017ac8444aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } health_report { overall_health: UNKNOWN } } } I0703 01:43:51.190150 6083 external_mini_cluster.cc:490] 7 TS(s) registered with all masters I0703 01:43:51.191215 6083 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 55223945fe3d4dd8ac03ba9dab346f54 to finish bootstrapping I0703 01:43:51.197923 6083 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 57bea6dc182c41a78304417a3cbfc4a7 to finish bootstrapping I0703 01:43:51.204385 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 8c1e8943f5654f71b030ec7c0b733742 to finish bootstrapping I0703 01:43:51.209774 6083 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 213596374e4142dca6ecfc701ba1c566 to finish bootstrapping I0703 01:43:51.216148 6083 ts_itest-base.cc:233] Waiting for 0 tablets on tserver aa8b9c58b466436faa5086e85b5224a7 to finish bootstrapping I0703 01:43:51.222586 6083 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 6932f1cd716a48fe8d6c5573acfc48bf to finish bootstrapping I0703 01:43:51.228503 6083 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 1157c91cc6f54a68a1d99017ac8444aa to finish bootstrapping I0703 01:43:51.238215 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 3864 I0703 01:43:51.238987 4005 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:43:51.276393 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 3864 W0703 01:43:51.287077 3437 connection.cc:512] server connection from 127.5.240.196:42932 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.196:42932): Connection reset by peer (error 104) I0703 01:43:51.287245 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 4011 I0703 01:43:51.287972 4127 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:43:51.325598 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 4011 W0703 01:43:51.333307 3437 connection.cc:512] server connection from 127.5.240.197:39911 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.197:39911): Connection reset by peer (error 104) I0703 01:43:51.333397 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 4133 I0703 01:43:51.334133 4250 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:43:51.371657 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 4133 W0703 01:43:51.379703 3435 connection.cc:512] server connection from 127.5.240.198:41360 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.198:41360): Connection reset by peer (error 104) I0703 01:43:51.379771 6083 external_mini_cluster.cc:1034] Attempting to check leaks for /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver pid 4256 I0703 01:43:51.380473 4375 generic_service.cc:169] Checking for leaks (request via RPC) I0703 01:43:51.423616 6083 external_mini_cluster.cc:993] Killing /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver with pid 4256 W0703 01:43:51.432082 3752 connection.cc:512] server connection from 127.5.240.199:57047 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.199:57047): Connection reset by peer (error 104) W0703 01:43:51.432199 3434 connection.cc:512] server connection from 127.5.240.199:44240 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.199:44240): Connection reset by peer (error 104) I0703 01:43:53.027549 4413 raft_consensus.cc:457] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0703 01:43:53.027645 4413 raft_consensus.cc:2730] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742: Snoozing failure detection for 2.079s (starting election) I0703 01:43:53.027721 4413 raft_consensus.cc:479] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } } peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } I0703 01:43:53.028355 4413 leader_election.cc:243] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377) W0703 01:43:53.029474 3749 leader_election.cc:299] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377): Network error: Client connection negotiation failed: client connection to 127.5.240.199:55377: connect: Connection refused (error 111) I0703 01:43:53.029579 3749 leader_election.cc:270] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0703 01:43:53.029848 4413 raft_consensus.cc:2730] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742: Snoozing failure detection for 2.196s (election complete) I0703 01:43:53.029952 4413 raft_consensus.cc:2485] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0703 01:43:55.183274 3453 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.5.240.254:58445: name: "rebalance_test_table_0" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 2 split_rows_range_bounds { } partition_schema { hash_bucket_schemas { columns { name: "key" } num_buckets: 3 seed: 0 } } I0703 01:43:55.200500 3675 tablet_service.cc:759] Processing CreateTablet for tablet eada0e8b8608424594cabd35cf9891e3 (table=rebalance_test_table_0 [id=5bc38732cb8746a7ad1c2b9fede5e7d9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I0703 01:43:55.200611 3551 tablet_service.cc:759] Processing CreateTablet for tablet 049bd2ef267f47fca634858420ac35b9 (table=rebalance_test_table_0 [id=5bc38732cb8746a7ad1c2b9fede5e7d9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I0703 01:43:55.200932 3552 tablet_service.cc:759] Processing CreateTablet for tablet eada0e8b8608424594cabd35cf9891e3 (table=rebalance_test_table_0 [id=5bc38732cb8746a7ad1c2b9fede5e7d9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I0703 01:43:55.201007 3676 tablet_service.cc:759] Processing CreateTablet for tablet 75f9ea8ba93f442aa064d4623d2c36a1 (table=rebalance_test_table_0 [id=5bc38732cb8746a7ad1c2b9fede5e7d9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I0703 01:43:55.200631 3674 tablet_service.cc:759] Processing CreateTablet for tablet 049bd2ef267f47fca634858420ac35b9 (table=rebalance_test_table_0 [id=5bc38732cb8746a7ad1c2b9fede5e7d9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I0703 01:43:55.201455 3552 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet eada0e8b8608424594cabd35cf9891e3. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:43:55.201583 3675 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet eada0e8b8608424594cabd35cf9891e3. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:43:55.200732 3553 tablet_service.cc:759] Processing CreateTablet for tablet 75f9ea8ba93f442aa064d4623d2c36a1 (table=rebalance_test_table_0 [id=5bc38732cb8746a7ad1c2b9fede5e7d9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I0703 01:43:55.202175 3551 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 049bd2ef267f47fca634858420ac35b9. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:43:55.202472 3674 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 049bd2ef267f47fca634858420ac35b9. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:43:55.202854 3676 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 75f9ea8ba93f442aa064d4623d2c36a1. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:43:55.202952 3553 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 75f9ea8ba93f442aa064d4623d2c36a1. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:43:55.221082 3675 ts_tablet_manager.cc:1173] T eada0e8b8608424594cabd35cf9891e3 P 57bea6dc182c41a78304417a3cbfc4a7: Registered tablet (data state: TABLET_DATA_READY) I0703 01:43:55.221175 3551 ts_tablet_manager.cc:1173] T 049bd2ef267f47fca634858420ac35b9 P 55223945fe3d4dd8ac03ba9dab346f54: Registered tablet (data state: TABLET_DATA_READY) I0703 01:43:55.221289 3676 ts_tablet_manager.cc:1173] T 75f9ea8ba93f442aa064d4623d2c36a1 P 57bea6dc182c41a78304417a3cbfc4a7: Registered tablet (data state: TABLET_DATA_READY) I0703 01:43:55.222617 4424 ts_tablet_manager.cc:1028] T 049bd2ef267f47fca634858420ac35b9 P 55223945fe3d4dd8ac03ba9dab346f54: Bootstrapping tablet I0703 01:43:55.222921 3674 ts_tablet_manager.cc:1173] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7: Registered tablet (data state: TABLET_DATA_READY) I0703 01:43:55.223058 4424 tablet_bootstrap.cc:427] T 049bd2ef267f47fca634858420ac35b9 P 55223945fe3d4dd8ac03ba9dab346f54: Bootstrap starting. I0703 01:43:55.223959 3553 ts_tablet_manager.cc:1173] T 75f9ea8ba93f442aa064d4623d2c36a1 P 55223945fe3d4dd8ac03ba9dab346f54: Registered tablet (data state: TABLET_DATA_READY) I0703 01:43:55.224422 4423 ts_tablet_manager.cc:1028] T eada0e8b8608424594cabd35cf9891e3 P 57bea6dc182c41a78304417a3cbfc4a7: Bootstrapping tablet I0703 01:43:55.224661 4423 tablet_bootstrap.cc:427] T eada0e8b8608424594cabd35cf9891e3 P 57bea6dc182c41a78304417a3cbfc4a7: Bootstrap starting. I0703 01:43:55.225950 3552 ts_tablet_manager.cc:1173] T eada0e8b8608424594cabd35cf9891e3 P 55223945fe3d4dd8ac03ba9dab346f54: Registered tablet (data state: TABLET_DATA_READY) I0703 01:43:55.226861 4424 tablet_bootstrap.cc:578] T 049bd2ef267f47fca634858420ac35b9 P 55223945fe3d4dd8ac03ba9dab346f54: No blocks or log segments found. Creating new log. I0703 01:43:55.227108 4423 tablet_bootstrap.cc:578] T eada0e8b8608424594cabd35cf9891e3 P 57bea6dc182c41a78304417a3cbfc4a7: No blocks or log segments found. Creating new log. I0703 01:43:55.227154 4425 raft_consensus.cc:457] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0703 01:43:55.227241 4425 raft_consensus.cc:2730] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742: Snoozing failure detection for 3.274s (starting election) I0703 01:43:55.227340 4425 raft_consensus.cc:479] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } } peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } I0703 01:43:55.227784 4425 leader_election.cc:243] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377) I0703 01:43:55.227879 4423 log.cc:525] T eada0e8b8608424594cabd35cf9891e3 P 57bea6dc182c41a78304417a3cbfc4a7: Log is configured to *not* fsync() on all Append() calls I0703 01:43:55.227886 4424 log.cc:525] T 049bd2ef267f47fca634858420ac35b9 P 55223945fe3d4dd8ac03ba9dab346f54: Log is configured to *not* fsync() on all Append() calls W0703 01:43:55.229854 3749 leader_election.cc:299] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377): Network error: Client connection negotiation failed: client connection to 127.5.240.199:55377: connect: Connection refused (error 111) I0703 01:43:55.229918 3749 leader_election.cc:270] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0703 01:43:55.229929 4423 tablet_bootstrap.cc:427] T eada0e8b8608424594cabd35cf9891e3 P 57bea6dc182c41a78304417a3cbfc4a7: No bootstrap required, opened a new log I0703 01:43:55.230113 4425 raft_consensus.cc:2730] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742: Snoozing failure detection for 1.604s (election complete) I0703 01:43:55.230141 4423 ts_tablet_manager.cc:1045] T eada0e8b8608424594cabd35cf9891e3 P 57bea6dc182c41a78304417a3cbfc4a7: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s I0703 01:43:55.230201 4425 raft_consensus.cc:2485] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0703 01:43:55.233151 4424 tablet_bootstrap.cc:427] T 049bd2ef267f47fca634858420ac35b9 P 55223945fe3d4dd8ac03ba9dab346f54: No bootstrap required, opened a new log I0703 01:43:55.233353 4424 ts_tablet_manager.cc:1045] T 049bd2ef267f47fca634858420ac35b9 P 55223945fe3d4dd8ac03ba9dab346f54: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s I0703 01:43:55.236747 4423 raft_consensus.cc:325] T eada0e8b8608424594cabd35cf9891e3 P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } I0703 01:43:55.237396 4423 raft_consensus.cc:351] T eada0e8b8608424594cabd35cf9891e3 P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:43:55.237493 4423 raft_consensus.cc:628] T eada0e8b8608424594cabd35cf9891e3 P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57bea6dc182c41a78304417a3cbfc4a7, State: Initialized, Role: FOLLOWER I0703 01:43:55.238004 4423 consensus_queue.cc:227] T eada0e8b8608424594cabd35cf9891e3 P 57bea6dc182c41a78304417a3cbfc4a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } W0703 01:43:55.239475 3740 tablet.cc:1678] T eada0e8b8608424594cabd35cf9891e3 P 57bea6dc182c41a78304417a3cbfc4a7: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:43:55.239859 4423 ts_tablet_manager.cc:1073] T eada0e8b8608424594cabd35cf9891e3 P 57bea6dc182c41a78304417a3cbfc4a7: Time spent starting tablet: real 0.009s user 0.006s sys 0.002s I0703 01:43:55.240080 4423 ts_tablet_manager.cc:1028] T 75f9ea8ba93f442aa064d4623d2c36a1 P 57bea6dc182c41a78304417a3cbfc4a7: Bootstrapping tablet I0703 01:43:55.240224 4423 tablet_bootstrap.cc:427] T 75f9ea8ba93f442aa064d4623d2c36a1 P 57bea6dc182c41a78304417a3cbfc4a7: Bootstrap starting. I0703 01:43:55.241456 4423 tablet_bootstrap.cc:578] T 75f9ea8ba93f442aa064d4623d2c36a1 P 57bea6dc182c41a78304417a3cbfc4a7: No blocks or log segments found. Creating new log. I0703 01:43:55.243110 4424 raft_consensus.cc:325] T 049bd2ef267f47fca634858420ac35b9 P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } I0703 01:43:55.243460 4424 raft_consensus.cc:351] T 049bd2ef267f47fca634858420ac35b9 P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:43:55.243542 4424 raft_consensus.cc:628] T 049bd2ef267f47fca634858420ac35b9 P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55223945fe3d4dd8ac03ba9dab346f54, State: Initialized, Role: FOLLOWER I0703 01:43:55.244002 4424 consensus_queue.cc:227] T 049bd2ef267f47fca634858420ac35b9 P 55223945fe3d4dd8ac03ba9dab346f54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } I0703 01:43:55.244709 4423 tablet_bootstrap.cc:427] T 75f9ea8ba93f442aa064d4623d2c36a1 P 57bea6dc182c41a78304417a3cbfc4a7: No bootstrap required, opened a new log I0703 01:43:55.244829 4423 ts_tablet_manager.cc:1045] T 75f9ea8ba93f442aa064d4623d2c36a1 P 57bea6dc182c41a78304417a3cbfc4a7: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s I0703 01:43:55.245501 4424 ts_tablet_manager.cc:1073] T 049bd2ef267f47fca634858420ac35b9 P 55223945fe3d4dd8ac03ba9dab346f54: Time spent starting tablet: real 0.012s user 0.008s sys 0.002s I0703 01:43:55.245710 4424 ts_tablet_manager.cc:1028] T 75f9ea8ba93f442aa064d4623d2c36a1 P 55223945fe3d4dd8ac03ba9dab346f54: Bootstrapping tablet I0703 01:43:55.245846 4424 tablet_bootstrap.cc:427] T 75f9ea8ba93f442aa064d4623d2c36a1 P 55223945fe3d4dd8ac03ba9dab346f54: Bootstrap starting. I0703 01:43:55.246110 4423 raft_consensus.cc:325] T 75f9ea8ba93f442aa064d4623d2c36a1 P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } I0703 01:43:55.246290 4423 raft_consensus.cc:351] T 75f9ea8ba93f442aa064d4623d2c36a1 P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:43:55.246366 4423 raft_consensus.cc:628] T 75f9ea8ba93f442aa064d4623d2c36a1 P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57bea6dc182c41a78304417a3cbfc4a7, State: Initialized, Role: FOLLOWER I0703 01:43:55.246563 4423 consensus_queue.cc:227] T 75f9ea8ba93f442aa064d4623d2c36a1 P 57bea6dc182c41a78304417a3cbfc4a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } I0703 01:43:55.247370 4423 ts_tablet_manager.cc:1073] T 75f9ea8ba93f442aa064d4623d2c36a1 P 57bea6dc182c41a78304417a3cbfc4a7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s I0703 01:43:55.247370 4424 tablet_bootstrap.cc:578] T 75f9ea8ba93f442aa064d4623d2c36a1 P 55223945fe3d4dd8ac03ba9dab346f54: No blocks or log segments found. Creating new log. I0703 01:43:55.247524 4423 ts_tablet_manager.cc:1028] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7: Bootstrapping tablet I0703 01:43:55.247715 4423 tablet_bootstrap.cc:427] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7: Bootstrap starting. I0703 01:43:55.249402 4423 tablet_bootstrap.cc:578] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7: No blocks or log segments found. Creating new log. I0703 01:43:55.250020 4424 tablet_bootstrap.cc:427] T 75f9ea8ba93f442aa064d4623d2c36a1 P 55223945fe3d4dd8ac03ba9dab346f54: No bootstrap required, opened a new log I0703 01:43:55.250144 4424 ts_tablet_manager.cc:1045] T 75f9ea8ba93f442aa064d4623d2c36a1 P 55223945fe3d4dd8ac03ba9dab346f54: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s I0703 01:43:55.250859 4424 raft_consensus.cc:325] T 75f9ea8ba93f442aa064d4623d2c36a1 P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } I0703 01:43:55.251045 4424 raft_consensus.cc:351] T 75f9ea8ba93f442aa064d4623d2c36a1 P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:43:55.251114 4424 raft_consensus.cc:628] T 75f9ea8ba93f442aa064d4623d2c36a1 P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55223945fe3d4dd8ac03ba9dab346f54, State: Initialized, Role: FOLLOWER I0703 01:43:55.251302 4424 consensus_queue.cc:227] T 75f9ea8ba93f442aa064d4623d2c36a1 P 55223945fe3d4dd8ac03ba9dab346f54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } I0703 01:43:55.251894 4424 ts_tablet_manager.cc:1073] T 75f9ea8ba93f442aa064d4623d2c36a1 P 55223945fe3d4dd8ac03ba9dab346f54: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s I0703 01:43:55.252059 4424 ts_tablet_manager.cc:1028] T eada0e8b8608424594cabd35cf9891e3 P 55223945fe3d4dd8ac03ba9dab346f54: Bootstrapping tablet I0703 01:43:55.252171 4424 tablet_bootstrap.cc:427] T eada0e8b8608424594cabd35cf9891e3 P 55223945fe3d4dd8ac03ba9dab346f54: Bootstrap starting. I0703 01:43:55.253115 4424 tablet_bootstrap.cc:578] T eada0e8b8608424594cabd35cf9891e3 P 55223945fe3d4dd8ac03ba9dab346f54: No blocks or log segments found. Creating new log. I0703 01:43:55.253274 4423 tablet_bootstrap.cc:427] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7: No bootstrap required, opened a new log I0703 01:43:55.253401 4423 ts_tablet_manager.cc:1045] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s I0703 01:43:55.254062 4423 raft_consensus.cc:325] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } I0703 01:43:55.254180 4423 raft_consensus.cc:351] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:43:55.254235 4423 raft_consensus.cc:628] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57bea6dc182c41a78304417a3cbfc4a7, State: Initialized, Role: FOLLOWER I0703 01:43:55.254390 4423 consensus_queue.cc:227] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } I0703 01:43:55.254814 4424 tablet_bootstrap.cc:427] T eada0e8b8608424594cabd35cf9891e3 P 55223945fe3d4dd8ac03ba9dab346f54: No bootstrap required, opened a new log I0703 01:43:55.254930 4424 ts_tablet_manager.cc:1045] T eada0e8b8608424594cabd35cf9891e3 P 55223945fe3d4dd8ac03ba9dab346f54: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s I0703 01:43:55.255061 4423 ts_tablet_manager.cc:1073] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s I0703 01:43:55.255479 4424 raft_consensus.cc:325] T eada0e8b8608424594cabd35cf9891e3 P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } I0703 01:43:55.255599 4424 raft_consensus.cc:351] T eada0e8b8608424594cabd35cf9891e3 P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:43:55.255671 4424 raft_consensus.cc:628] T eada0e8b8608424594cabd35cf9891e3 P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55223945fe3d4dd8ac03ba9dab346f54, State: Initialized, Role: FOLLOWER I0703 01:43:55.255832 4424 consensus_queue.cc:227] T eada0e8b8608424594cabd35cf9891e3 P 55223945fe3d4dd8ac03ba9dab346f54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } I0703 01:43:55.256289 4424 ts_tablet_manager.cc:1073] T eada0e8b8608424594cabd35cf9891e3 P 55223945fe3d4dd8ac03ba9dab346f54: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s I0703 01:43:55.281227 4429 raft_consensus.cc:457] T 75f9ea8ba93f442aa064d4623d2c36a1 P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0703 01:43:55.281332 4429 raft_consensus.cc:2730] T 75f9ea8ba93f442aa064d4623d2c36a1 P 57bea6dc182c41a78304417a3cbfc4a7: Snoozing failure detection for 1.822s (starting election) I0703 01:43:55.281446 4429 raft_consensus.cc:479] T 75f9ea8ba93f442aa064d4623d2c36a1 P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } I0703 01:43:55.282063 4429 leader_election.cc:243] T 75f9ea8ba93f442aa064d4623d2c36a1 P 57bea6dc182c41a78304417a3cbfc4a7 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193:54919) I0703 01:43:55.288053 3573 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "75f9ea8ba93f442aa064d4623d2c36a1" candidate_uuid: "57bea6dc182c41a78304417a3cbfc4a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55223945fe3d4dd8ac03ba9dab346f54" is_pre_election: true I0703 01:43:55.288275 3573 raft_consensus.cc:2730] T 75f9ea8ba93f442aa064d4623d2c36a1 P 55223945fe3d4dd8ac03ba9dab346f54: Snoozing failure detection for 2.198s (vote granted) I0703 01:43:55.288362 3573 raft_consensus.cc:2243] T 75f9ea8ba93f442aa064d4623d2c36a1 P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57bea6dc182c41a78304417a3cbfc4a7 in term 0. I0703 01:43:55.288909 3630 leader_election.cc:379] T 75f9ea8ba93f442aa064d4623d2c36a1 P 57bea6dc182c41a78304417a3cbfc4a7 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193:54919) I0703 01:43:55.288978 3630 leader_election.cc:270] T 75f9ea8ba93f442aa064d4623d2c36a1 P 57bea6dc182c41a78304417a3cbfc4a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0703 01:43:55.289242 4429 raft_consensus.cc:2730] T 75f9ea8ba93f442aa064d4623d2c36a1 P 57bea6dc182c41a78304417a3cbfc4a7: Snoozing failure detection for 1.702s (election complete) I0703 01:43:55.289335 4429 raft_consensus.cc:2537] T 75f9ea8ba93f442aa064d4623d2c36a1 P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0703 01:43:55.289388 4429 raft_consensus.cc:457] T 75f9ea8ba93f442aa064d4623d2c36a1 P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0703 01:43:55.289427 4429 raft_consensus.cc:2730] T 75f9ea8ba93f442aa064d4623d2c36a1 P 57bea6dc182c41a78304417a3cbfc4a7: Snoozing failure detection for 1.880s (starting election) I0703 01:43:55.289466 4429 raft_consensus.cc:2782] T 75f9ea8ba93f442aa064d4623d2c36a1 P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:43:55.291625 4429 raft_consensus.cc:479] T 75f9ea8ba93f442aa064d4623d2c36a1 P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } I0703 01:43:55.291863 4429 leader_election.cc:243] T 75f9ea8ba93f442aa064d4623d2c36a1 P 57bea6dc182c41a78304417a3cbfc4a7 [CANDIDATE]: Term 1 election: Requesting vote from peer 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193:54919) I0703 01:43:55.292517 3573 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "75f9ea8ba93f442aa064d4623d2c36a1" candidate_uuid: "57bea6dc182c41a78304417a3cbfc4a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55223945fe3d4dd8ac03ba9dab346f54" I0703 01:43:55.292631 3573 raft_consensus.cc:2782] T 75f9ea8ba93f442aa064d4623d2c36a1 P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:43:55.292718 3573 raft_consensus.cc:2730] T 75f9ea8ba93f442aa064d4623d2c36a1 P 55223945fe3d4dd8ac03ba9dab346f54: Snoozing failure detection for 2.214s (vote granted) I0703 01:43:55.294811 3573 raft_consensus.cc:2243] T 75f9ea8ba93f442aa064d4623d2c36a1 P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57bea6dc182c41a78304417a3cbfc4a7 in term 1. I0703 01:43:55.295188 3630 leader_election.cc:379] T 75f9ea8ba93f442aa064d4623d2c36a1 P 57bea6dc182c41a78304417a3cbfc4a7 [CANDIDATE]: Term 1 election: Vote granted by peer 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193:54919) I0703 01:43:55.295244 3630 leader_election.cc:270] T 75f9ea8ba93f442aa064d4623d2c36a1 P 57bea6dc182c41a78304417a3cbfc4a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:43:55.295418 4429 raft_consensus.cc:2730] T 75f9ea8ba93f442aa064d4623d2c36a1 P 57bea6dc182c41a78304417a3cbfc4a7: Snoozing failure detection for 2.220s (election complete) I0703 01:43:55.295485 4429 raft_consensus.cc:2537] T 75f9ea8ba93f442aa064d4623d2c36a1 P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:43:55.296499 4429 raft_consensus.cc:593] T 75f9ea8ba93f442aa064d4623d2c36a1 P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 LEADER]: Becoming Leader. State: Replica: 57bea6dc182c41a78304417a3cbfc4a7, State: Running, Role: LEADER I0703 01:43:55.297019 4429 consensus_queue.cc:204] T 75f9ea8ba93f442aa064d4623d2c36a1 P 57bea6dc182c41a78304417a3cbfc4a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } I0703 01:43:55.299928 3453 catalog_manager.cc:3798] T 75f9ea8ba93f442aa064d4623d2c36a1 P 57bea6dc182c41a78304417a3cbfc4a7 reported cstate change: term changed from 0 to 1, leader changed from to 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194). New cstate: current_term: 1 leader_uuid: "57bea6dc182c41a78304417a3cbfc4a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } health_report { overall_health: UNKNOWN } } } I0703 01:43:55.328547 4431 raft_consensus.cc:457] T eada0e8b8608424594cabd35cf9891e3 P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0703 01:43:55.328649 4431 raft_consensus.cc:2730] T eada0e8b8608424594cabd35cf9891e3 P 55223945fe3d4dd8ac03ba9dab346f54: Snoozing failure detection for 2.150s (starting election) I0703 01:43:55.328719 4431 raft_consensus.cc:479] T eada0e8b8608424594cabd35cf9891e3 P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } I0703 01:43:55.329252 4431 leader_election.cc:243] T eada0e8b8608424594cabd35cf9891e3 P 55223945fe3d4dd8ac03ba9dab346f54 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746) I0703 01:43:55.335496 3696 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "eada0e8b8608424594cabd35cf9891e3" candidate_uuid: "55223945fe3d4dd8ac03ba9dab346f54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57bea6dc182c41a78304417a3cbfc4a7" is_pre_election: true I0703 01:43:55.335734 3696 raft_consensus.cc:2730] T eada0e8b8608424594cabd35cf9891e3 P 57bea6dc182c41a78304417a3cbfc4a7: Snoozing failure detection for 1.852s (vote granted) I0703 01:43:55.335811 3696 raft_consensus.cc:2243] T eada0e8b8608424594cabd35cf9891e3 P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55223945fe3d4dd8ac03ba9dab346f54 in term 0. I0703 01:43:55.336340 3504 leader_election.cc:379] T eada0e8b8608424594cabd35cf9891e3 P 55223945fe3d4dd8ac03ba9dab346f54 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746) I0703 01:43:55.336410 3504 leader_election.cc:270] T eada0e8b8608424594cabd35cf9891e3 P 55223945fe3d4dd8ac03ba9dab346f54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0703 01:43:55.336694 4431 raft_consensus.cc:2730] T eada0e8b8608424594cabd35cf9891e3 P 55223945fe3d4dd8ac03ba9dab346f54: Snoozing failure detection for 1.997s (election complete) I0703 01:43:55.336786 4431 raft_consensus.cc:2537] T eada0e8b8608424594cabd35cf9891e3 P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0703 01:43:55.336846 4431 raft_consensus.cc:457] T eada0e8b8608424594cabd35cf9891e3 P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0703 01:43:55.336901 4431 raft_consensus.cc:2730] T eada0e8b8608424594cabd35cf9891e3 P 55223945fe3d4dd8ac03ba9dab346f54: Snoozing failure detection for 1.634s (starting election) I0703 01:43:55.336961 4431 raft_consensus.cc:2782] T eada0e8b8608424594cabd35cf9891e3 P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:43:55.338583 4431 raft_consensus.cc:479] T eada0e8b8608424594cabd35cf9891e3 P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } I0703 01:43:55.338877 4431 leader_election.cc:243] T eada0e8b8608424594cabd35cf9891e3 P 55223945fe3d4dd8ac03ba9dab346f54 [CANDIDATE]: Term 1 election: Requesting vote from peer 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746) I0703 01:43:55.339485 3696 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "eada0e8b8608424594cabd35cf9891e3" candidate_uuid: "55223945fe3d4dd8ac03ba9dab346f54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57bea6dc182c41a78304417a3cbfc4a7" I0703 01:43:55.339592 3696 raft_consensus.cc:2782] T eada0e8b8608424594cabd35cf9891e3 P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:43:55.339659 3696 raft_consensus.cc:2730] T eada0e8b8608424594cabd35cf9891e3 P 57bea6dc182c41a78304417a3cbfc4a7: Snoozing failure detection for 1.585s (vote granted) I0703 01:43:55.341228 3696 raft_consensus.cc:2243] T eada0e8b8608424594cabd35cf9891e3 P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55223945fe3d4dd8ac03ba9dab346f54 in term 1. I0703 01:43:55.341650 3504 leader_election.cc:379] T eada0e8b8608424594cabd35cf9891e3 P 55223945fe3d4dd8ac03ba9dab346f54 [CANDIDATE]: Term 1 election: Vote granted by peer 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746) I0703 01:43:55.341706 3504 leader_election.cc:270] T eada0e8b8608424594cabd35cf9891e3 P 55223945fe3d4dd8ac03ba9dab346f54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:43:55.341864 4431 raft_consensus.cc:2730] T eada0e8b8608424594cabd35cf9891e3 P 55223945fe3d4dd8ac03ba9dab346f54: Snoozing failure detection for 2.063s (election complete) I0703 01:43:55.341933 4431 raft_consensus.cc:2537] T eada0e8b8608424594cabd35cf9891e3 P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:43:55.342831 4431 raft_consensus.cc:593] T eada0e8b8608424594cabd35cf9891e3 P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 LEADER]: Becoming Leader. State: Replica: 55223945fe3d4dd8ac03ba9dab346f54, State: Running, Role: LEADER I0703 01:43:55.343240 4431 consensus_queue.cc:204] T eada0e8b8608424594cabd35cf9891e3 P 55223945fe3d4dd8ac03ba9dab346f54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } I0703 01:43:55.346468 3453 catalog_manager.cc:3798] T eada0e8b8608424594cabd35cf9891e3 P 55223945fe3d4dd8ac03ba9dab346f54 reported cstate change: term changed from 0 to 1, leader changed from to 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193). New cstate: current_term: 1 leader_uuid: "55223945fe3d4dd8ac03ba9dab346f54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } health_report { overall_health: UNKNOWN } } } W0703 01:43:55.413832 3617 tablet.cc:1678] T 049bd2ef267f47fca634858420ac35b9 P 55223945fe3d4dd8ac03ba9dab346f54: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:43:55.477490 4429 raft_consensus.cc:457] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0703 01:43:55.477560 4429 raft_consensus.cc:2730] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7: Snoozing failure detection for 1.934s (starting election) I0703 01:43:55.477646 4429 raft_consensus.cc:479] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } I0703 01:43:55.477963 4429 leader_election.cc:243] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193:54919) I0703 01:43:55.478687 3573 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "049bd2ef267f47fca634858420ac35b9" candidate_uuid: "57bea6dc182c41a78304417a3cbfc4a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55223945fe3d4dd8ac03ba9dab346f54" is_pre_election: true I0703 01:43:55.478817 3573 raft_consensus.cc:2730] T 049bd2ef267f47fca634858420ac35b9 P 55223945fe3d4dd8ac03ba9dab346f54: Snoozing failure detection for 2.065s (vote granted) I0703 01:43:55.478879 3573 raft_consensus.cc:2243] T 049bd2ef267f47fca634858420ac35b9 P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57bea6dc182c41a78304417a3cbfc4a7 in term 0. I0703 01:43:55.479317 3630 leader_election.cc:379] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193:54919) I0703 01:43:55.479379 3630 leader_election.cc:270] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0703 01:43:55.479598 4429 raft_consensus.cc:2730] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7: Snoozing failure detection for 1.658s (election complete) I0703 01:43:55.479670 4429 raft_consensus.cc:2537] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0703 01:43:55.479720 4429 raft_consensus.cc:457] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0703 01:43:55.479781 4429 raft_consensus.cc:2730] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7: Snoozing failure detection for 1.799s (starting election) I0703 01:43:55.479840 4429 raft_consensus.cc:2782] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:43:55.481456 4429 raft_consensus.cc:479] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } I0703 01:43:55.481750 4429 leader_election.cc:243] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 [CANDIDATE]: Term 1 election: Requesting vote from peer 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193:54919) I0703 01:43:55.482347 3573 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "049bd2ef267f47fca634858420ac35b9" candidate_uuid: "57bea6dc182c41a78304417a3cbfc4a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55223945fe3d4dd8ac03ba9dab346f54" I0703 01:43:55.482453 3573 raft_consensus.cc:2782] T 049bd2ef267f47fca634858420ac35b9 P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:43:55.482537 3573 raft_consensus.cc:2730] T 049bd2ef267f47fca634858420ac35b9 P 55223945fe3d4dd8ac03ba9dab346f54: Snoozing failure detection for 2.195s (vote granted) I0703 01:43:55.484041 3573 raft_consensus.cc:2243] T 049bd2ef267f47fca634858420ac35b9 P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57bea6dc182c41a78304417a3cbfc4a7 in term 1. I0703 01:43:55.484419 3630 leader_election.cc:379] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 [CANDIDATE]: Term 1 election: Vote granted by peer 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193:54919) I0703 01:43:55.484486 3630 leader_election.cc:270] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:43:55.484697 4429 raft_consensus.cc:2730] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7: Snoozing failure detection for 1.939s (election complete) I0703 01:43:55.484763 4429 raft_consensus.cc:2537] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:43:55.484890 4429 raft_consensus.cc:593] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 LEADER]: Becoming Leader. State: Replica: 57bea6dc182c41a78304417a3cbfc4a7, State: Running, Role: LEADER I0703 01:43:55.485127 4429 consensus_queue.cc:204] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } I0703 01:43:55.487429 3453 catalog_manager.cc:3798] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 reported cstate change: term changed from 0 to 1, leader changed from to 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194). New cstate: current_term: 1 leader_uuid: "57bea6dc182c41a78304417a3cbfc4a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } health_report { overall_health: HEALTHY } } } W0703 01:43:55.779039 4442 flags.cc:406] Enabled unsafe flag: --never_fsync=true Using table 'rebalance_test_table_0' I0703 01:43:55.852056 3573 raft_consensus.cc:1081] T 75f9ea8ba93f442aa064d4623d2c36a1 P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 FOLLOWER]: Refusing update from remote peer 57bea6dc182c41a78304417a3cbfc4a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0703 01:43:55.852881 4429 consensus_queue.cc:962] T 75f9ea8ba93f442aa064d4623d2c36a1 P 57bea6dc182c41a78304417a3cbfc4a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0703 01:43:55.906908 3696 raft_consensus.cc:1081] T eada0e8b8608424594cabd35cf9891e3 P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 FOLLOWER]: Refusing update from remote peer 55223945fe3d4dd8ac03ba9dab346f54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0703 01:43:55.907951 4453 consensus_queue.cc:962] T eada0e8b8608424594cabd35cf9891e3 P 55223945fe3d4dd8ac03ba9dab346f54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0703 01:43:55.937700 3573 raft_consensus.cc:1081] T 049bd2ef267f47fca634858420ac35b9 P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 FOLLOWER]: Refusing update from remote peer 57bea6dc182c41a78304417a3cbfc4a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0703 01:43:55.938747 4429 consensus_queue.cc:962] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s Generator report time total : 211.721 ms time per row: 0.105861 ms I0703 01:43:56.084081 3452 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.5.240.254:58445: name: "rebalance_test_table_1" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 2 split_rows_range_bounds { } partition_schema { hash_bucket_schemas { columns { name: "key" } num_buckets: 3 seed: 0 } } I0703 01:43:56.090864 3552 tablet_service.cc:759] Processing CreateTablet for tablet 6e0179cbc11c41bdafeaad002709c067 (table=rebalance_test_table_1 [id=29732119416840ec89c7a3633f79c5fd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I0703 01:43:56.091253 3552 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 6e0179cbc11c41bdafeaad002709c067. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:43:56.091377 3798 tablet_service.cc:759] Processing CreateTablet for tablet 6e0179cbc11c41bdafeaad002709c067 (table=rebalance_test_table_1 [id=29732119416840ec89c7a3633f79c5fd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I0703 01:43:56.091428 3675 tablet_service.cc:759] Processing CreateTablet for tablet 930bbe4ac3234ed49a2207c4902541f1 (table=rebalance_test_table_1 [id=29732119416840ec89c7a3633f79c5fd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I0703 01:43:56.091725 3798 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 6e0179cbc11c41bdafeaad002709c067. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:43:56.091778 3675 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 930bbe4ac3234ed49a2207c4902541f1. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:43:56.092232 3553 tablet_service.cc:759] Processing CreateTablet for tablet c352f70a51154a85a5a5b0f3d14d405f (table=rebalance_test_table_1 [id=29732119416840ec89c7a3633f79c5fd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I0703 01:43:56.092625 3553 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet c352f70a51154a85a5a5b0f3d14d405f. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:43:56.092871 3797 tablet_service.cc:759] Processing CreateTablet for tablet 930bbe4ac3234ed49a2207c4902541f1 (table=rebalance_test_table_1 [id=29732119416840ec89c7a3633f79c5fd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I0703 01:43:56.093185 3797 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 930bbe4ac3234ed49a2207c4902541f1. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:43:56.094938 3796 tablet_service.cc:759] Processing CreateTablet for tablet c352f70a51154a85a5a5b0f3d14d405f (table=rebalance_test_table_1 [id=29732119416840ec89c7a3633f79c5fd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I0703 01:43:56.095819 3796 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet c352f70a51154a85a5a5b0f3d14d405f. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:43:56.098439 3675 ts_tablet_manager.cc:1173] T 930bbe4ac3234ed49a2207c4902541f1 P 57bea6dc182c41a78304417a3cbfc4a7: Registered tablet (data state: TABLET_DATA_READY) I0703 01:43:56.099107 3798 ts_tablet_manager.cc:1173] T 6e0179cbc11c41bdafeaad002709c067 P 8c1e8943f5654f71b030ec7c0b733742: Registered tablet (data state: TABLET_DATA_READY) I0703 01:43:56.099691 3552 ts_tablet_manager.cc:1173] T 6e0179cbc11c41bdafeaad002709c067 P 55223945fe3d4dd8ac03ba9dab346f54: Registered tablet (data state: TABLET_DATA_READY) I0703 01:43:56.101097 4478 ts_tablet_manager.cc:1028] T 6e0179cbc11c41bdafeaad002709c067 P 8c1e8943f5654f71b030ec7c0b733742: Bootstrapping tablet I0703 01:43:56.101227 4478 tablet_bootstrap.cc:427] T 6e0179cbc11c41bdafeaad002709c067 P 8c1e8943f5654f71b030ec7c0b733742: Bootstrap starting. I0703 01:43:56.101693 3553 ts_tablet_manager.cc:1173] T c352f70a51154a85a5a5b0f3d14d405f P 55223945fe3d4dd8ac03ba9dab346f54: Registered tablet (data state: TABLET_DATA_READY) I0703 01:43:56.104198 4478 tablet_bootstrap.cc:578] T 6e0179cbc11c41bdafeaad002709c067 P 8c1e8943f5654f71b030ec7c0b733742: No blocks or log segments found. Creating new log. I0703 01:43:56.106072 4477 ts_tablet_manager.cc:1028] T 930bbe4ac3234ed49a2207c4902541f1 P 57bea6dc182c41a78304417a3cbfc4a7: Bootstrapping tablet I0703 01:43:56.106227 3796 ts_tablet_manager.cc:1173] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742: Registered tablet (data state: TABLET_DATA_READY) I0703 01:43:56.106567 4477 tablet_bootstrap.cc:427] T 930bbe4ac3234ed49a2207c4902541f1 P 57bea6dc182c41a78304417a3cbfc4a7: Bootstrap starting. I0703 01:43:56.106592 3797 ts_tablet_manager.cc:1173] T 930bbe4ac3234ed49a2207c4902541f1 P 8c1e8943f5654f71b030ec7c0b733742: Registered tablet (data state: TABLET_DATA_READY) I0703 01:43:56.106724 4479 ts_tablet_manager.cc:1028] T 6e0179cbc11c41bdafeaad002709c067 P 55223945fe3d4dd8ac03ba9dab346f54: Bootstrapping tablet I0703 01:43:56.106964 4479 tablet_bootstrap.cc:427] T 6e0179cbc11c41bdafeaad002709c067 P 55223945fe3d4dd8ac03ba9dab346f54: Bootstrap starting. I0703 01:43:56.108305 4477 tablet_bootstrap.cc:578] T 930bbe4ac3234ed49a2207c4902541f1 P 57bea6dc182c41a78304417a3cbfc4a7: No blocks or log segments found. Creating new log. I0703 01:43:56.108367 4478 tablet_bootstrap.cc:427] T 6e0179cbc11c41bdafeaad002709c067 P 8c1e8943f5654f71b030ec7c0b733742: No bootstrap required, opened a new log I0703 01:43:56.108463 4478 ts_tablet_manager.cc:1045] T 6e0179cbc11c41bdafeaad002709c067 P 8c1e8943f5654f71b030ec7c0b733742: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s I0703 01:43:56.108798 4479 tablet_bootstrap.cc:578] T 6e0179cbc11c41bdafeaad002709c067 P 55223945fe3d4dd8ac03ba9dab346f54: No blocks or log segments found. Creating new log. I0703 01:43:56.109089 4478 raft_consensus.cc:325] T 6e0179cbc11c41bdafeaad002709c067 P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } I0703 01:43:56.109306 4478 raft_consensus.cc:351] T 6e0179cbc11c41bdafeaad002709c067 P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:43:56.109383 4478 raft_consensus.cc:628] T 6e0179cbc11c41bdafeaad002709c067 P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c1e8943f5654f71b030ec7c0b733742, State: Initialized, Role: FOLLOWER I0703 01:43:56.109658 4478 consensus_queue.cc:227] T 6e0179cbc11c41bdafeaad002709c067 P 8c1e8943f5654f71b030ec7c0b733742 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } I0703 01:43:56.111070 4477 tablet_bootstrap.cc:427] T 930bbe4ac3234ed49a2207c4902541f1 P 57bea6dc182c41a78304417a3cbfc4a7: No bootstrap required, opened a new log I0703 01:43:56.111187 4477 ts_tablet_manager.cc:1045] T 930bbe4ac3234ed49a2207c4902541f1 P 57bea6dc182c41a78304417a3cbfc4a7: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s I0703 01:43:56.111492 4478 ts_tablet_manager.cc:1073] T 6e0179cbc11c41bdafeaad002709c067 P 8c1e8943f5654f71b030ec7c0b733742: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s I0703 01:43:56.111654 4478 ts_tablet_manager.cc:1028] T 930bbe4ac3234ed49a2207c4902541f1 P 8c1e8943f5654f71b030ec7c0b733742: Bootstrapping tablet I0703 01:43:56.111783 4478 tablet_bootstrap.cc:427] T 930bbe4ac3234ed49a2207c4902541f1 P 8c1e8943f5654f71b030ec7c0b733742: Bootstrap starting. I0703 01:43:56.111847 4477 raft_consensus.cc:325] T 930bbe4ac3234ed49a2207c4902541f1 P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } I0703 01:43:56.112025 4477 raft_consensus.cc:351] T 930bbe4ac3234ed49a2207c4902541f1 P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:43:56.112072 4477 raft_consensus.cc:628] T 930bbe4ac3234ed49a2207c4902541f1 P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57bea6dc182c41a78304417a3cbfc4a7, State: Initialized, Role: FOLLOWER I0703 01:43:56.112210 4477 consensus_queue.cc:227] T 930bbe4ac3234ed49a2207c4902541f1 P 57bea6dc182c41a78304417a3cbfc4a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } I0703 01:43:56.112677 4477 ts_tablet_manager.cc:1073] T 930bbe4ac3234ed49a2207c4902541f1 P 57bea6dc182c41a78304417a3cbfc4a7: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s I0703 01:43:56.114217 4478 tablet_bootstrap.cc:578] T 930bbe4ac3234ed49a2207c4902541f1 P 8c1e8943f5654f71b030ec7c0b733742: No blocks or log segments found. Creating new log. I0703 01:43:56.114902 4479 tablet_bootstrap.cc:427] T 6e0179cbc11c41bdafeaad002709c067 P 55223945fe3d4dd8ac03ba9dab346f54: No bootstrap required, opened a new log I0703 01:43:56.115002 4479 ts_tablet_manager.cc:1045] T 6e0179cbc11c41bdafeaad002709c067 P 55223945fe3d4dd8ac03ba9dab346f54: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s I0703 01:43:56.115496 4479 raft_consensus.cc:325] T 6e0179cbc11c41bdafeaad002709c067 P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } I0703 01:43:56.115656 4479 raft_consensus.cc:351] T 6e0179cbc11c41bdafeaad002709c067 P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:43:56.115706 4479 raft_consensus.cc:628] T 6e0179cbc11c41bdafeaad002709c067 P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55223945fe3d4dd8ac03ba9dab346f54, State: Initialized, Role: FOLLOWER I0703 01:43:56.115906 4479 consensus_queue.cc:227] T 6e0179cbc11c41bdafeaad002709c067 P 55223945fe3d4dd8ac03ba9dab346f54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } I0703 01:43:56.116120 4478 tablet_bootstrap.cc:427] T 930bbe4ac3234ed49a2207c4902541f1 P 8c1e8943f5654f71b030ec7c0b733742: No bootstrap required, opened a new log I0703 01:43:56.116231 4478 ts_tablet_manager.cc:1045] T 930bbe4ac3234ed49a2207c4902541f1 P 8c1e8943f5654f71b030ec7c0b733742: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s I0703 01:43:56.116487 4479 ts_tablet_manager.cc:1073] T 6e0179cbc11c41bdafeaad002709c067 P 55223945fe3d4dd8ac03ba9dab346f54: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s I0703 01:43:56.116708 4479 ts_tablet_manager.cc:1028] T c352f70a51154a85a5a5b0f3d14d405f P 55223945fe3d4dd8ac03ba9dab346f54: Bootstrapping tablet I0703 01:43:56.116856 4479 tablet_bootstrap.cc:427] T c352f70a51154a85a5a5b0f3d14d405f P 55223945fe3d4dd8ac03ba9dab346f54: Bootstrap starting. I0703 01:43:56.116796 4478 raft_consensus.cc:325] T 930bbe4ac3234ed49a2207c4902541f1 P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } I0703 01:43:56.116988 4478 raft_consensus.cc:351] T 930bbe4ac3234ed49a2207c4902541f1 P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:43:56.117038 4478 raft_consensus.cc:628] T 930bbe4ac3234ed49a2207c4902541f1 P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c1e8943f5654f71b030ec7c0b733742, State: Initialized, Role: FOLLOWER I0703 01:43:56.117202 4478 consensus_queue.cc:227] T 930bbe4ac3234ed49a2207c4902541f1 P 8c1e8943f5654f71b030ec7c0b733742 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } I0703 01:43:56.118098 4479 tablet_bootstrap.cc:578] T c352f70a51154a85a5a5b0f3d14d405f P 55223945fe3d4dd8ac03ba9dab346f54: No blocks or log segments found. Creating new log. I0703 01:43:56.118366 4478 ts_tablet_manager.cc:1073] T 930bbe4ac3234ed49a2207c4902541f1 P 8c1e8943f5654f71b030ec7c0b733742: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s I0703 01:43:56.118742 4478 ts_tablet_manager.cc:1028] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742: Bootstrapping tablet I0703 01:43:56.118888 4478 tablet_bootstrap.cc:427] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742: Bootstrap starting. I0703 01:43:56.120836 4478 tablet_bootstrap.cc:578] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742: No blocks or log segments found. Creating new log. I0703 01:43:56.121337 4479 tablet_bootstrap.cc:427] T c352f70a51154a85a5a5b0f3d14d405f P 55223945fe3d4dd8ac03ba9dab346f54: No bootstrap required, opened a new log I0703 01:43:56.121419 4479 ts_tablet_manager.cc:1045] T c352f70a51154a85a5a5b0f3d14d405f P 55223945fe3d4dd8ac03ba9dab346f54: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s I0703 01:43:56.121906 4479 raft_consensus.cc:325] T c352f70a51154a85a5a5b0f3d14d405f P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } I0703 01:43:56.122016 4479 raft_consensus.cc:351] T c352f70a51154a85a5a5b0f3d14d405f P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:43:56.122063 4479 raft_consensus.cc:628] T c352f70a51154a85a5a5b0f3d14d405f P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55223945fe3d4dd8ac03ba9dab346f54, State: Initialized, Role: FOLLOWER I0703 01:43:56.122195 4479 consensus_queue.cc:227] T c352f70a51154a85a5a5b0f3d14d405f P 55223945fe3d4dd8ac03ba9dab346f54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } I0703 01:43:56.122721 4479 ts_tablet_manager.cc:1073] T c352f70a51154a85a5a5b0f3d14d405f P 55223945fe3d4dd8ac03ba9dab346f54: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s I0703 01:43:56.123163 4478 tablet_bootstrap.cc:427] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742: No bootstrap required, opened a new log I0703 01:43:56.123280 4478 ts_tablet_manager.cc:1045] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s I0703 01:43:56.123988 4478 raft_consensus.cc:325] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } I0703 01:43:56.124110 4478 raft_consensus.cc:351] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:43:56.124184 4478 raft_consensus.cc:628] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c1e8943f5654f71b030ec7c0b733742, State: Initialized, Role: FOLLOWER I0703 01:43:56.124428 4478 consensus_queue.cc:227] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } I0703 01:43:56.125272 4478 ts_tablet_manager.cc:1073] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s I0703 01:43:56.143712 4483 raft_consensus.cc:457] T 6e0179cbc11c41bdafeaad002709c067 P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0703 01:43:56.143793 4483 raft_consensus.cc:2730] T 6e0179cbc11c41bdafeaad002709c067 P 8c1e8943f5654f71b030ec7c0b733742: Snoozing failure detection for 1.537s (starting election) I0703 01:43:56.143882 4483 raft_consensus.cc:479] T 6e0179cbc11c41bdafeaad002709c067 P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } I0703 01:43:56.144189 4483 leader_election.cc:243] T 6e0179cbc11c41bdafeaad002709c067 P 8c1e8943f5654f71b030ec7c0b733742 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193:54919) I0703 01:43:56.149093 3573 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "6e0179cbc11c41bdafeaad002709c067" candidate_uuid: "8c1e8943f5654f71b030ec7c0b733742" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55223945fe3d4dd8ac03ba9dab346f54" is_pre_election: true I0703 01:43:56.149220 3573 raft_consensus.cc:2730] T 6e0179cbc11c41bdafeaad002709c067 P 55223945fe3d4dd8ac03ba9dab346f54: Snoozing failure detection for 1.958s (vote granted) I0703 01:43:56.149287 3573 raft_consensus.cc:2243] T 6e0179cbc11c41bdafeaad002709c067 P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c1e8943f5654f71b030ec7c0b733742 in term 0. I0703 01:43:56.149722 3752 leader_election.cc:379] T 6e0179cbc11c41bdafeaad002709c067 P 8c1e8943f5654f71b030ec7c0b733742 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193:54919) I0703 01:43:56.149785 3752 leader_election.cc:270] T 6e0179cbc11c41bdafeaad002709c067 P 8c1e8943f5654f71b030ec7c0b733742 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0703 01:43:56.149958 4483 raft_consensus.cc:2730] T 6e0179cbc11c41bdafeaad002709c067 P 8c1e8943f5654f71b030ec7c0b733742: Snoozing failure detection for 1.526s (election complete) I0703 01:43:56.150051 4483 raft_consensus.cc:2537] T 6e0179cbc11c41bdafeaad002709c067 P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0703 01:43:56.150107 4483 raft_consensus.cc:457] T 6e0179cbc11c41bdafeaad002709c067 P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0703 01:43:56.150164 4483 raft_consensus.cc:2730] T 6e0179cbc11c41bdafeaad002709c067 P 8c1e8943f5654f71b030ec7c0b733742: Snoozing failure detection for 1.639s (starting election) I0703 01:43:56.150230 4483 raft_consensus.cc:2782] T 6e0179cbc11c41bdafeaad002709c067 P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:43:56.151916 4483 raft_consensus.cc:479] T 6e0179cbc11c41bdafeaad002709c067 P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } I0703 01:43:56.152168 4483 leader_election.cc:243] T 6e0179cbc11c41bdafeaad002709c067 P 8c1e8943f5654f71b030ec7c0b733742 [CANDIDATE]: Term 1 election: Requesting vote from peer 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193:54919) I0703 01:43:56.152742 3573 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "6e0179cbc11c41bdafeaad002709c067" candidate_uuid: "8c1e8943f5654f71b030ec7c0b733742" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55223945fe3d4dd8ac03ba9dab346f54" I0703 01:43:56.152855 3573 raft_consensus.cc:2782] T 6e0179cbc11c41bdafeaad002709c067 P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:43:56.152916 3573 raft_consensus.cc:2730] T 6e0179cbc11c41bdafeaad002709c067 P 55223945fe3d4dd8ac03ba9dab346f54: Snoozing failure detection for 2.215s (vote granted) I0703 01:43:56.154173 3573 raft_consensus.cc:2243] T 6e0179cbc11c41bdafeaad002709c067 P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c1e8943f5654f71b030ec7c0b733742 in term 1. I0703 01:43:56.154538 3752 leader_election.cc:379] T 6e0179cbc11c41bdafeaad002709c067 P 8c1e8943f5654f71b030ec7c0b733742 [CANDIDATE]: Term 1 election: Vote granted by peer 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193:54919) I0703 01:43:56.154610 3752 leader_election.cc:270] T 6e0179cbc11c41bdafeaad002709c067 P 8c1e8943f5654f71b030ec7c0b733742 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:43:56.154762 4483 raft_consensus.cc:2730] T 6e0179cbc11c41bdafeaad002709c067 P 8c1e8943f5654f71b030ec7c0b733742: Snoozing failure detection for 2.039s (election complete) I0703 01:43:56.154829 4483 raft_consensus.cc:2537] T 6e0179cbc11c41bdafeaad002709c067 P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:43:56.156052 4483 raft_consensus.cc:593] T 6e0179cbc11c41bdafeaad002709c067 P 8c1e8943f5654f71b030ec7c0b733742 [term 1 LEADER]: Becoming Leader. State: Replica: 8c1e8943f5654f71b030ec7c0b733742, State: Running, Role: LEADER I0703 01:43:56.156473 4483 consensus_queue.cc:204] T 6e0179cbc11c41bdafeaad002709c067 P 8c1e8943f5654f71b030ec7c0b733742 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } I0703 01:43:56.160087 3449 catalog_manager.cc:3798] T 6e0179cbc11c41bdafeaad002709c067 P 8c1e8943f5654f71b030ec7c0b733742 reported cstate change: term changed from 0 to 1, leader changed from to 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195). New cstate: current_term: 1 leader_uuid: "8c1e8943f5654f71b030ec7c0b733742" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } health_report { overall_health: UNKNOWN } } } I0703 01:43:56.285004 4483 raft_consensus.cc:457] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0703 01:43:56.285073 4483 raft_consensus.cc:2730] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742: Snoozing failure detection for 1.712s (starting election) I0703 01:43:56.285146 4483 raft_consensus.cc:479] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } I0703 01:43:56.285409 4483 leader_election.cc:243] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193:54919) I0703 01:43:56.286051 3573 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "c352f70a51154a85a5a5b0f3d14d405f" candidate_uuid: "8c1e8943f5654f71b030ec7c0b733742" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55223945fe3d4dd8ac03ba9dab346f54" is_pre_election: true I0703 01:43:56.286180 3573 raft_consensus.cc:2730] T c352f70a51154a85a5a5b0f3d14d405f P 55223945fe3d4dd8ac03ba9dab346f54: Snoozing failure detection for 1.837s (vote granted) I0703 01:43:56.286244 3573 raft_consensus.cc:2243] T c352f70a51154a85a5a5b0f3d14d405f P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c1e8943f5654f71b030ec7c0b733742 in term 0. I0703 01:43:56.286649 3752 leader_election.cc:379] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193:54919) I0703 01:43:56.286708 3752 leader_election.cc:270] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0703 01:43:56.286873 4483 raft_consensus.cc:2730] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742: Snoozing failure detection for 2.093s (election complete) I0703 01:43:56.286947 4483 raft_consensus.cc:2537] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0703 01:43:56.286996 4483 raft_consensus.cc:457] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0703 01:43:56.287055 4483 raft_consensus.cc:2730] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742: Snoozing failure detection for 1.606s (starting election) I0703 01:43:56.287111 4483 raft_consensus.cc:2782] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:43:56.288349 4483 raft_consensus.cc:479] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } I0703 01:43:56.288616 4483 leader_election.cc:243] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 [CANDIDATE]: Term 1 election: Requesting vote from peer 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193:54919) I0703 01:43:56.289201 3573 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "c352f70a51154a85a5a5b0f3d14d405f" candidate_uuid: "8c1e8943f5654f71b030ec7c0b733742" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55223945fe3d4dd8ac03ba9dab346f54" I0703 01:43:56.289299 3573 raft_consensus.cc:2782] T c352f70a51154a85a5a5b0f3d14d405f P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:43:56.289377 3573 raft_consensus.cc:2730] T c352f70a51154a85a5a5b0f3d14d405f P 55223945fe3d4dd8ac03ba9dab346f54: Snoozing failure detection for 1.940s (vote granted) I0703 01:43:56.290622 3573 raft_consensus.cc:2243] T c352f70a51154a85a5a5b0f3d14d405f P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c1e8943f5654f71b030ec7c0b733742 in term 1. I0703 01:43:56.291043 3752 leader_election.cc:379] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 [CANDIDATE]: Term 1 election: Vote granted by peer 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193:54919) I0703 01:43:56.291100 3752 leader_election.cc:270] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:43:56.291254 4483 raft_consensus.cc:2730] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742: Snoozing failure detection for 2.046s (election complete) I0703 01:43:56.291314 4483 raft_consensus.cc:2537] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:43:56.291435 4483 raft_consensus.cc:593] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 [term 1 LEADER]: Becoming Leader. State: Replica: 8c1e8943f5654f71b030ec7c0b733742, State: Running, Role: LEADER I0703 01:43:56.291616 4483 consensus_queue.cc:204] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } I0703 01:43:56.294239 3449 catalog_manager.cc:3798] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 reported cstate change: term changed from 0 to 1, leader changed from to 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195). New cstate: current_term: 1 leader_uuid: "8c1e8943f5654f71b030ec7c0b733742" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } health_report { overall_health: UNKNOWN } } } I0703 01:43:56.403360 4470 raft_consensus.cc:457] T 930bbe4ac3234ed49a2207c4902541f1 P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0703 01:43:56.403441 4470 raft_consensus.cc:2730] T 930bbe4ac3234ed49a2207c4902541f1 P 57bea6dc182c41a78304417a3cbfc4a7: Snoozing failure detection for 1.574s (starting election) I0703 01:43:56.403551 4470 raft_consensus.cc:479] T 930bbe4ac3234ed49a2207c4902541f1 P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } I0703 01:43:56.403910 4470 leader_election.cc:243] T 930bbe4ac3234ed49a2207c4902541f1 P 57bea6dc182c41a78304417a3cbfc4a7 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064) I0703 01:43:56.409577 3818 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "930bbe4ac3234ed49a2207c4902541f1" candidate_uuid: "57bea6dc182c41a78304417a3cbfc4a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c1e8943f5654f71b030ec7c0b733742" is_pre_election: true I0703 01:43:56.409714 3818 raft_consensus.cc:2730] T 930bbe4ac3234ed49a2207c4902541f1 P 8c1e8943f5654f71b030ec7c0b733742: Snoozing failure detection for 1.800s (vote granted) I0703 01:43:56.409775 3818 raft_consensus.cc:2243] T 930bbe4ac3234ed49a2207c4902541f1 P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57bea6dc182c41a78304417a3cbfc4a7 in term 0. I0703 01:43:56.410212 3630 leader_election.cc:379] T 930bbe4ac3234ed49a2207c4902541f1 P 57bea6dc182c41a78304417a3cbfc4a7 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064) I0703 01:43:56.410274 3630 leader_election.cc:270] T 930bbe4ac3234ed49a2207c4902541f1 P 57bea6dc182c41a78304417a3cbfc4a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0703 01:43:56.410465 4470 raft_consensus.cc:2730] T 930bbe4ac3234ed49a2207c4902541f1 P 57bea6dc182c41a78304417a3cbfc4a7: Snoozing failure detection for 2.157s (election complete) I0703 01:43:56.410533 4470 raft_consensus.cc:2537] T 930bbe4ac3234ed49a2207c4902541f1 P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0703 01:43:56.410614 4470 raft_consensus.cc:457] T 930bbe4ac3234ed49a2207c4902541f1 P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0703 01:43:56.410665 4470 raft_consensus.cc:2730] T 930bbe4ac3234ed49a2207c4902541f1 P 57bea6dc182c41a78304417a3cbfc4a7: Snoozing failure detection for 1.858s (starting election) I0703 01:43:56.410727 4470 raft_consensus.cc:2782] T 930bbe4ac3234ed49a2207c4902541f1 P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:43:56.412125 4470 raft_consensus.cc:479] T 930bbe4ac3234ed49a2207c4902541f1 P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } I0703 01:43:56.412384 4470 leader_election.cc:243] T 930bbe4ac3234ed49a2207c4902541f1 P 57bea6dc182c41a78304417a3cbfc4a7 [CANDIDATE]: Term 1 election: Requesting vote from peer 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064) I0703 01:43:56.413035 3818 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "930bbe4ac3234ed49a2207c4902541f1" candidate_uuid: "57bea6dc182c41a78304417a3cbfc4a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c1e8943f5654f71b030ec7c0b733742" I0703 01:43:56.413148 3818 raft_consensus.cc:2782] T 930bbe4ac3234ed49a2207c4902541f1 P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:43:56.413216 3818 raft_consensus.cc:2730] T 930bbe4ac3234ed49a2207c4902541f1 P 8c1e8943f5654f71b030ec7c0b733742: Snoozing failure detection for 2.115s (vote granted) I0703 01:43:56.414464 3818 raft_consensus.cc:2243] T 930bbe4ac3234ed49a2207c4902541f1 P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57bea6dc182c41a78304417a3cbfc4a7 in term 1. I0703 01:43:56.414872 3630 leader_election.cc:379] T 930bbe4ac3234ed49a2207c4902541f1 P 57bea6dc182c41a78304417a3cbfc4a7 [CANDIDATE]: Term 1 election: Vote granted by peer 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064) I0703 01:43:56.414929 3630 leader_election.cc:270] T 930bbe4ac3234ed49a2207c4902541f1 P 57bea6dc182c41a78304417a3cbfc4a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:43:56.415107 4470 raft_consensus.cc:2730] T 930bbe4ac3234ed49a2207c4902541f1 P 57bea6dc182c41a78304417a3cbfc4a7: Snoozing failure detection for 2.223s (election complete) I0703 01:43:56.415176 4470 raft_consensus.cc:2537] T 930bbe4ac3234ed49a2207c4902541f1 P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:43:56.415299 4470 raft_consensus.cc:593] T 930bbe4ac3234ed49a2207c4902541f1 P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 LEADER]: Becoming Leader. State: Replica: 57bea6dc182c41a78304417a3cbfc4a7, State: Running, Role: LEADER I0703 01:43:56.415495 4470 consensus_queue.cc:204] T 930bbe4ac3234ed49a2207c4902541f1 P 57bea6dc182c41a78304417a3cbfc4a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } I0703 01:43:56.417807 3449 catalog_manager.cc:3798] T 930bbe4ac3234ed49a2207c4902541f1 P 57bea6dc182c41a78304417a3cbfc4a7 reported cstate change: term changed from 0 to 1, leader changed from to 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194). New cstate: current_term: 1 leader_uuid: "57bea6dc182c41a78304417a3cbfc4a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } health_report { overall_health: HEALTHY } } } W0703 01:43:56.674926 4494 flags.cc:406] Enabled unsafe flag: --never_fsync=true I0703 01:43:56.718329 3573 raft_consensus.cc:1081] T 6e0179cbc11c41bdafeaad002709c067 P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 FOLLOWER]: Refusing update from remote peer 8c1e8943f5654f71b030ec7c0b733742: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0703 01:43:56.719029 4483 consensus_queue.cc:962] T 6e0179cbc11c41bdafeaad002709c067 P 8c1e8943f5654f71b030ec7c0b733742 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s Using table 'rebalance_test_table_1' I0703 01:43:56.766896 3573 raft_consensus.cc:1081] T c352f70a51154a85a5a5b0f3d14d405f P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 FOLLOWER]: Refusing update from remote peer 8c1e8943f5654f71b030ec7c0b733742: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0703 01:43:56.767617 4483 consensus_queue.cc:962] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0703 01:43:56.834241 3818 raft_consensus.cc:1081] T 930bbe4ac3234ed49a2207c4902541f1 P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Refusing update from remote peer 57bea6dc182c41a78304417a3cbfc4a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0703 01:43:56.835209 4470 consensus_queue.cc:962] T 930bbe4ac3234ed49a2207c4902541f1 P 57bea6dc182c41a78304417a3cbfc4a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0703 01:43:56.835551 4483 raft_consensus.cc:457] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0703 01:43:56.835656 4483 raft_consensus.cc:2730] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742: Snoozing failure detection for 2.393s (starting election) I0703 01:43:56.835773 4483 raft_consensus.cc:479] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } } peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } I0703 01:43:56.836230 4483 leader_election.cc:243] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377) W0703 01:43:56.858217 3749 leader_election.cc:299] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377): Network error: Client connection negotiation failed: client connection to 127.5.240.199:55377: connect: Connection refused (error 111) I0703 01:43:56.858399 3749 leader_election.cc:270] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0703 01:43:56.862684 4483 raft_consensus.cc:2730] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742: Snoozing failure detection for 2.143s (election complete) I0703 01:43:56.863915 4483 raft_consensus.cc:2485] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority Generator report time total : 229.338 ms time per row: 0.114669 ms I0703 01:43:56.999260 3449 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.5.240.254:58445: name: "rebalance_test_table_2" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 2 split_rows_range_bounds { } partition_schema { hash_bucket_schemas { columns { name: "key" } num_buckets: 3 seed: 0 } } I0703 01:43:57.006247 3675 tablet_service.cc:759] Processing CreateTablet for tablet 150033054dcf498d8d71fd996af55e88 (table=rebalance_test_table_2 [id=24e8eff3fcb34269ba893ac2456e5712]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I0703 01:43:57.006294 3796 tablet_service.cc:759] Processing CreateTablet for tablet 150033054dcf498d8d71fd996af55e88 (table=rebalance_test_table_2 [id=24e8eff3fcb34269ba893ac2456e5712]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I0703 01:43:57.006573 3675 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 150033054dcf498d8d71fd996af55e88. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:43:57.006695 3796 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 150033054dcf498d8d71fd996af55e88. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:43:57.008015 3552 tablet_service.cc:759] Processing CreateTablet for tablet 2051955b4b644a03a9adad816d350596 (table=rebalance_test_table_2 [id=24e8eff3fcb34269ba893ac2456e5712]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I0703 01:43:57.008162 3797 tablet_service.cc:759] Processing CreateTablet for tablet eea4e9534e0d4d25a827fae00bbeae4d (table=rebalance_test_table_2 [id=24e8eff3fcb34269ba893ac2456e5712]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I0703 01:43:57.008273 3552 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 2051955b4b644a03a9adad816d350596. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:43:57.008496 3797 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet eea4e9534e0d4d25a827fae00bbeae4d. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:43:57.009325 3676 tablet_service.cc:759] Processing CreateTablet for tablet 2051955b4b644a03a9adad816d350596 (table=rebalance_test_table_2 [id=24e8eff3fcb34269ba893ac2456e5712]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I0703 01:43:57.011740 3552 ts_tablet_manager.cc:1173] T 2051955b4b644a03a9adad816d350596 P 55223945fe3d4dd8ac03ba9dab346f54: Registered tablet (data state: TABLET_DATA_READY) I0703 01:43:57.012125 3676 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 2051955b4b644a03a9adad816d350596. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:43:57.012091 3674 tablet_service.cc:759] Processing CreateTablet for tablet eea4e9534e0d4d25a827fae00bbeae4d (table=rebalance_test_table_2 [id=24e8eff3fcb34269ba893ac2456e5712]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I0703 01:43:57.012583 3674 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet eea4e9534e0d4d25a827fae00bbeae4d. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:43:57.013347 3796 ts_tablet_manager.cc:1173] T 150033054dcf498d8d71fd996af55e88 P 8c1e8943f5654f71b030ec7c0b733742: Registered tablet (data state: TABLET_DATA_READY) I0703 01:43:57.013662 3797 ts_tablet_manager.cc:1173] T eea4e9534e0d4d25a827fae00bbeae4d P 8c1e8943f5654f71b030ec7c0b733742: Registered tablet (data state: TABLET_DATA_READY) I0703 01:43:57.014840 4532 ts_tablet_manager.cc:1028] T 150033054dcf498d8d71fd996af55e88 P 8c1e8943f5654f71b030ec7c0b733742: Bootstrapping tablet I0703 01:43:57.014982 4532 tablet_bootstrap.cc:427] T 150033054dcf498d8d71fd996af55e88 P 8c1e8943f5654f71b030ec7c0b733742: Bootstrap starting. I0703 01:43:57.016398 4532 tablet_bootstrap.cc:578] T 150033054dcf498d8d71fd996af55e88 P 8c1e8943f5654f71b030ec7c0b733742: No blocks or log segments found. Creating new log. I0703 01:43:57.016511 4531 ts_tablet_manager.cc:1028] T 2051955b4b644a03a9adad816d350596 P 55223945fe3d4dd8ac03ba9dab346f54: Bootstrapping tablet I0703 01:43:57.016703 4531 tablet_bootstrap.cc:427] T 2051955b4b644a03a9adad816d350596 P 55223945fe3d4dd8ac03ba9dab346f54: Bootstrap starting. I0703 01:43:57.017808 3675 ts_tablet_manager.cc:1173] T 150033054dcf498d8d71fd996af55e88 P 57bea6dc182c41a78304417a3cbfc4a7: Registered tablet (data state: TABLET_DATA_READY) I0703 01:43:57.018023 4531 tablet_bootstrap.cc:578] T 2051955b4b644a03a9adad816d350596 P 55223945fe3d4dd8ac03ba9dab346f54: No blocks or log segments found. Creating new log. I0703 01:43:57.020534 3676 ts_tablet_manager.cc:1173] T 2051955b4b644a03a9adad816d350596 P 57bea6dc182c41a78304417a3cbfc4a7: Registered tablet (data state: TABLET_DATA_READY) I0703 01:43:57.021884 3674 ts_tablet_manager.cc:1173] T eea4e9534e0d4d25a827fae00bbeae4d P 57bea6dc182c41a78304417a3cbfc4a7: Registered tablet (data state: TABLET_DATA_READY) I0703 01:43:57.022733 4534 ts_tablet_manager.cc:1028] T 150033054dcf498d8d71fd996af55e88 P 57bea6dc182c41a78304417a3cbfc4a7: Bootstrapping tablet I0703 01:43:57.022850 4534 tablet_bootstrap.cc:427] T 150033054dcf498d8d71fd996af55e88 P 57bea6dc182c41a78304417a3cbfc4a7: Bootstrap starting. I0703 01:43:57.023352 4532 tablet_bootstrap.cc:427] T 150033054dcf498d8d71fd996af55e88 P 8c1e8943f5654f71b030ec7c0b733742: No bootstrap required, opened a new log I0703 01:43:57.023438 4532 ts_tablet_manager.cc:1045] T 150033054dcf498d8d71fd996af55e88 P 8c1e8943f5654f71b030ec7c0b733742: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s I0703 01:43:57.023792 4534 tablet_bootstrap.cc:578] T 150033054dcf498d8d71fd996af55e88 P 57bea6dc182c41a78304417a3cbfc4a7: No blocks or log segments found. Creating new log. I0703 01:43:57.023988 4532 raft_consensus.cc:325] T 150033054dcf498d8d71fd996af55e88 P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } I0703 01:43:57.024163 4532 raft_consensus.cc:351] T 150033054dcf498d8d71fd996af55e88 P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:43:57.024216 4532 raft_consensus.cc:628] T 150033054dcf498d8d71fd996af55e88 P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c1e8943f5654f71b030ec7c0b733742, State: Initialized, Role: FOLLOWER I0703 01:43:57.024380 4532 consensus_queue.cc:227] T 150033054dcf498d8d71fd996af55e88 P 8c1e8943f5654f71b030ec7c0b733742 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } I0703 01:43:57.024927 4531 tablet_bootstrap.cc:427] T 2051955b4b644a03a9adad816d350596 P 55223945fe3d4dd8ac03ba9dab346f54: No bootstrap required, opened a new log I0703 01:43:57.025033 4531 ts_tablet_manager.cc:1045] T 2051955b4b644a03a9adad816d350596 P 55223945fe3d4dd8ac03ba9dab346f54: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s I0703 01:43:57.025045 4532 ts_tablet_manager.cc:1073] T 150033054dcf498d8d71fd996af55e88 P 8c1e8943f5654f71b030ec7c0b733742: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s I0703 01:43:57.025223 4532 ts_tablet_manager.cc:1028] T eea4e9534e0d4d25a827fae00bbeae4d P 8c1e8943f5654f71b030ec7c0b733742: Bootstrapping tablet I0703 01:43:57.025339 4532 tablet_bootstrap.cc:427] T eea4e9534e0d4d25a827fae00bbeae4d P 8c1e8943f5654f71b030ec7c0b733742: Bootstrap starting. I0703 01:43:57.025666 4531 raft_consensus.cc:325] T 2051955b4b644a03a9adad816d350596 P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } I0703 01:43:57.025797 4531 raft_consensus.cc:351] T 2051955b4b644a03a9adad816d350596 P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:43:57.025882 4531 raft_consensus.cc:628] T 2051955b4b644a03a9adad816d350596 P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55223945fe3d4dd8ac03ba9dab346f54, State: Initialized, Role: FOLLOWER I0703 01:43:57.026049 4531 consensus_queue.cc:227] T 2051955b4b644a03a9adad816d350596 P 55223945fe3d4dd8ac03ba9dab346f54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } I0703 01:43:57.026561 4531 ts_tablet_manager.cc:1073] T 2051955b4b644a03a9adad816d350596 P 55223945fe3d4dd8ac03ba9dab346f54: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s I0703 01:43:57.027138 4532 tablet_bootstrap.cc:578] T eea4e9534e0d4d25a827fae00bbeae4d P 8c1e8943f5654f71b030ec7c0b733742: No blocks or log segments found. Creating new log. I0703 01:43:57.027158 4534 tablet_bootstrap.cc:427] T 150033054dcf498d8d71fd996af55e88 P 57bea6dc182c41a78304417a3cbfc4a7: No bootstrap required, opened a new log I0703 01:43:57.027245 4534 ts_tablet_manager.cc:1045] T 150033054dcf498d8d71fd996af55e88 P 57bea6dc182c41a78304417a3cbfc4a7: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s I0703 01:43:57.027770 4534 raft_consensus.cc:325] T 150033054dcf498d8d71fd996af55e88 P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } I0703 01:43:57.027954 4534 raft_consensus.cc:351] T 150033054dcf498d8d71fd996af55e88 P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:43:57.028021 4534 raft_consensus.cc:628] T 150033054dcf498d8d71fd996af55e88 P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57bea6dc182c41a78304417a3cbfc4a7, State: Initialized, Role: FOLLOWER I0703 01:43:57.028201 4534 consensus_queue.cc:227] T 150033054dcf498d8d71fd996af55e88 P 57bea6dc182c41a78304417a3cbfc4a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } I0703 01:43:57.028816 4534 ts_tablet_manager.cc:1073] T 150033054dcf498d8d71fd996af55e88 P 57bea6dc182c41a78304417a3cbfc4a7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s I0703 01:43:57.028996 4534 ts_tablet_manager.cc:1028] T 2051955b4b644a03a9adad816d350596 P 57bea6dc182c41a78304417a3cbfc4a7: Bootstrapping tablet I0703 01:43:57.029165 4534 tablet_bootstrap.cc:427] T 2051955b4b644a03a9adad816d350596 P 57bea6dc182c41a78304417a3cbfc4a7: Bootstrap starting. I0703 01:43:57.030146 4534 tablet_bootstrap.cc:578] T 2051955b4b644a03a9adad816d350596 P 57bea6dc182c41a78304417a3cbfc4a7: No blocks or log segments found. Creating new log. I0703 01:43:57.030656 4532 tablet_bootstrap.cc:427] T eea4e9534e0d4d25a827fae00bbeae4d P 8c1e8943f5654f71b030ec7c0b733742: No bootstrap required, opened a new log I0703 01:43:57.030762 4532 ts_tablet_manager.cc:1045] T eea4e9534e0d4d25a827fae00bbeae4d P 8c1e8943f5654f71b030ec7c0b733742: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s I0703 01:43:57.031275 4532 raft_consensus.cc:325] T eea4e9534e0d4d25a827fae00bbeae4d P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } I0703 01:43:57.031388 4532 raft_consensus.cc:351] T eea4e9534e0d4d25a827fae00bbeae4d P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:43:57.031422 4532 raft_consensus.cc:628] T eea4e9534e0d4d25a827fae00bbeae4d P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c1e8943f5654f71b030ec7c0b733742, State: Initialized, Role: FOLLOWER I0703 01:43:57.031554 4532 consensus_queue.cc:227] T eea4e9534e0d4d25a827fae00bbeae4d P 8c1e8943f5654f71b030ec7c0b733742 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } I0703 01:43:57.032011 4532 ts_tablet_manager.cc:1073] T eea4e9534e0d4d25a827fae00bbeae4d P 8c1e8943f5654f71b030ec7c0b733742: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s I0703 01:43:57.032027 4534 tablet_bootstrap.cc:427] T 2051955b4b644a03a9adad816d350596 P 57bea6dc182c41a78304417a3cbfc4a7: No bootstrap required, opened a new log I0703 01:43:57.032121 4534 ts_tablet_manager.cc:1045] T 2051955b4b644a03a9adad816d350596 P 57bea6dc182c41a78304417a3cbfc4a7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s I0703 01:43:57.032649 4534 raft_consensus.cc:325] T 2051955b4b644a03a9adad816d350596 P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } I0703 01:43:57.032754 4534 raft_consensus.cc:351] T 2051955b4b644a03a9adad816d350596 P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:43:57.032794 4534 raft_consensus.cc:628] T 2051955b4b644a03a9adad816d350596 P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57bea6dc182c41a78304417a3cbfc4a7, State: Initialized, Role: FOLLOWER I0703 01:43:57.032959 4534 consensus_queue.cc:227] T 2051955b4b644a03a9adad816d350596 P 57bea6dc182c41a78304417a3cbfc4a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } I0703 01:43:57.033434 4534 ts_tablet_manager.cc:1073] T 2051955b4b644a03a9adad816d350596 P 57bea6dc182c41a78304417a3cbfc4a7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s I0703 01:43:57.033625 4534 ts_tablet_manager.cc:1028] T eea4e9534e0d4d25a827fae00bbeae4d P 57bea6dc182c41a78304417a3cbfc4a7: Bootstrapping tablet I0703 01:43:57.033721 4534 tablet_bootstrap.cc:427] T eea4e9534e0d4d25a827fae00bbeae4d P 57bea6dc182c41a78304417a3cbfc4a7: Bootstrap starting. I0703 01:43:57.034555 4534 tablet_bootstrap.cc:578] T eea4e9534e0d4d25a827fae00bbeae4d P 57bea6dc182c41a78304417a3cbfc4a7: No blocks or log segments found. Creating new log. I0703 01:43:57.035945 4534 tablet_bootstrap.cc:427] T eea4e9534e0d4d25a827fae00bbeae4d P 57bea6dc182c41a78304417a3cbfc4a7: No bootstrap required, opened a new log I0703 01:43:57.036034 4534 ts_tablet_manager.cc:1045] T eea4e9534e0d4d25a827fae00bbeae4d P 57bea6dc182c41a78304417a3cbfc4a7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s I0703 01:43:57.036445 4534 raft_consensus.cc:325] T eea4e9534e0d4d25a827fae00bbeae4d P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } I0703 01:43:57.036576 4534 raft_consensus.cc:351] T eea4e9534e0d4d25a827fae00bbeae4d P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:43:57.036644 4534 raft_consensus.cc:628] T eea4e9534e0d4d25a827fae00bbeae4d P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57bea6dc182c41a78304417a3cbfc4a7, State: Initialized, Role: FOLLOWER I0703 01:43:57.036790 4534 consensus_queue.cc:227] T eea4e9534e0d4d25a827fae00bbeae4d P 57bea6dc182c41a78304417a3cbfc4a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } I0703 01:43:57.037227 4534 ts_tablet_manager.cc:1073] T eea4e9534e0d4d25a827fae00bbeae4d P 57bea6dc182c41a78304417a3cbfc4a7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s I0703 01:43:57.080392 4470 raft_consensus.cc:457] T eea4e9534e0d4d25a827fae00bbeae4d P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0703 01:43:57.080449 4470 raft_consensus.cc:2730] T eea4e9534e0d4d25a827fae00bbeae4d P 57bea6dc182c41a78304417a3cbfc4a7: Snoozing failure detection for 2.160s (starting election) I0703 01:43:57.080543 4470 raft_consensus.cc:479] T eea4e9534e0d4d25a827fae00bbeae4d P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } I0703 01:43:57.080832 4470 leader_election.cc:243] T eea4e9534e0d4d25a827fae00bbeae4d P 57bea6dc182c41a78304417a3cbfc4a7 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064) I0703 01:43:57.081496 3817 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "eea4e9534e0d4d25a827fae00bbeae4d" candidate_uuid: "57bea6dc182c41a78304417a3cbfc4a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c1e8943f5654f71b030ec7c0b733742" is_pre_election: true I0703 01:43:57.081661 3817 raft_consensus.cc:2730] T eea4e9534e0d4d25a827fae00bbeae4d P 8c1e8943f5654f71b030ec7c0b733742: Snoozing failure detection for 1.821s (vote granted) I0703 01:43:57.081732 3817 raft_consensus.cc:2243] T eea4e9534e0d4d25a827fae00bbeae4d P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57bea6dc182c41a78304417a3cbfc4a7 in term 0. I0703 01:43:57.082165 3630 leader_election.cc:379] T eea4e9534e0d4d25a827fae00bbeae4d P 57bea6dc182c41a78304417a3cbfc4a7 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064) I0703 01:43:57.082226 3630 leader_election.cc:270] T eea4e9534e0d4d25a827fae00bbeae4d P 57bea6dc182c41a78304417a3cbfc4a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0703 01:43:57.082393 4470 raft_consensus.cc:2730] T eea4e9534e0d4d25a827fae00bbeae4d P 57bea6dc182c41a78304417a3cbfc4a7: Snoozing failure detection for 1.561s (election complete) I0703 01:43:57.082454 4470 raft_consensus.cc:2537] T eea4e9534e0d4d25a827fae00bbeae4d P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0703 01:43:57.082538 4470 raft_consensus.cc:457] T eea4e9534e0d4d25a827fae00bbeae4d P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0703 01:43:57.082589 4470 raft_consensus.cc:2730] T eea4e9534e0d4d25a827fae00bbeae4d P 57bea6dc182c41a78304417a3cbfc4a7: Snoozing failure detection for 1.881s (starting election) I0703 01:43:57.082650 4470 raft_consensus.cc:2782] T eea4e9534e0d4d25a827fae00bbeae4d P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:43:57.083811 4470 raft_consensus.cc:479] T eea4e9534e0d4d25a827fae00bbeae4d P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } I0703 01:43:57.084064 4470 leader_election.cc:243] T eea4e9534e0d4d25a827fae00bbeae4d P 57bea6dc182c41a78304417a3cbfc4a7 [CANDIDATE]: Term 1 election: Requesting vote from peer 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064) I0703 01:43:57.084723 3817 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "eea4e9534e0d4d25a827fae00bbeae4d" candidate_uuid: "57bea6dc182c41a78304417a3cbfc4a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c1e8943f5654f71b030ec7c0b733742" I0703 01:43:57.084837 3817 raft_consensus.cc:2782] T eea4e9534e0d4d25a827fae00bbeae4d P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:43:57.084900 3817 raft_consensus.cc:2730] T eea4e9534e0d4d25a827fae00bbeae4d P 8c1e8943f5654f71b030ec7c0b733742: Snoozing failure detection for 1.978s (vote granted) I0703 01:43:57.086133 3817 raft_consensus.cc:2243] T eea4e9534e0d4d25a827fae00bbeae4d P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57bea6dc182c41a78304417a3cbfc4a7 in term 1. I0703 01:43:57.086545 3630 leader_election.cc:379] T eea4e9534e0d4d25a827fae00bbeae4d P 57bea6dc182c41a78304417a3cbfc4a7 [CANDIDATE]: Term 1 election: Vote granted by peer 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064) I0703 01:43:57.086621 3630 leader_election.cc:270] T eea4e9534e0d4d25a827fae00bbeae4d P 57bea6dc182c41a78304417a3cbfc4a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:43:57.086822 4470 raft_consensus.cc:2730] T eea4e9534e0d4d25a827fae00bbeae4d P 57bea6dc182c41a78304417a3cbfc4a7: Snoozing failure detection for 1.967s (election complete) I0703 01:43:57.086889 4470 raft_consensus.cc:2537] T eea4e9534e0d4d25a827fae00bbeae4d P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:43:57.087004 4470 raft_consensus.cc:593] T eea4e9534e0d4d25a827fae00bbeae4d P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 LEADER]: Becoming Leader. State: Replica: 57bea6dc182c41a78304417a3cbfc4a7, State: Running, Role: LEADER I0703 01:43:57.087174 4470 consensus_queue.cc:204] T eea4e9534e0d4d25a827fae00bbeae4d P 57bea6dc182c41a78304417a3cbfc4a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } I0703 01:43:57.088879 3449 catalog_manager.cc:3798] T eea4e9534e0d4d25a827fae00bbeae4d P 57bea6dc182c41a78304417a3cbfc4a7 reported cstate change: term changed from 0 to 1, leader changed from to 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194). New cstate: current_term: 1 leader_uuid: "57bea6dc182c41a78304417a3cbfc4a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } health_report { overall_health: HEALTHY } } } I0703 01:43:57.168247 4488 raft_consensus.cc:457] T 150033054dcf498d8d71fd996af55e88 P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0703 01:43:57.168313 4488 raft_consensus.cc:2730] T 150033054dcf498d8d71fd996af55e88 P 8c1e8943f5654f71b030ec7c0b733742: Snoozing failure detection for 2.217s (starting election) I0703 01:43:57.168401 4488 raft_consensus.cc:479] T 150033054dcf498d8d71fd996af55e88 P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } I0703 01:43:57.168709 4488 leader_election.cc:243] T 150033054dcf498d8d71fd996af55e88 P 8c1e8943f5654f71b030ec7c0b733742 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746) I0703 01:43:57.172999 3696 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "150033054dcf498d8d71fd996af55e88" candidate_uuid: "8c1e8943f5654f71b030ec7c0b733742" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57bea6dc182c41a78304417a3cbfc4a7" is_pre_election: true I0703 01:43:57.173115 3696 raft_consensus.cc:2730] T 150033054dcf498d8d71fd996af55e88 P 57bea6dc182c41a78304417a3cbfc4a7: Snoozing failure detection for 2.132s (vote granted) I0703 01:43:57.173171 3696 raft_consensus.cc:2243] T 150033054dcf498d8d71fd996af55e88 P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c1e8943f5654f71b030ec7c0b733742 in term 0. I0703 01:43:57.173569 3749 leader_election.cc:379] T 150033054dcf498d8d71fd996af55e88 P 8c1e8943f5654f71b030ec7c0b733742 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746) I0703 01:43:57.173636 3749 leader_election.cc:270] T 150033054dcf498d8d71fd996af55e88 P 8c1e8943f5654f71b030ec7c0b733742 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0703 01:43:57.173854 4488 raft_consensus.cc:2730] T 150033054dcf498d8d71fd996af55e88 P 8c1e8943f5654f71b030ec7c0b733742: Snoozing failure detection for 1.991s (election complete) I0703 01:43:57.173915 4488 raft_consensus.cc:2537] T 150033054dcf498d8d71fd996af55e88 P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0703 01:43:57.173974 4488 raft_consensus.cc:457] T 150033054dcf498d8d71fd996af55e88 P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0703 01:43:57.174027 4488 raft_consensus.cc:2730] T 150033054dcf498d8d71fd996af55e88 P 8c1e8943f5654f71b030ec7c0b733742: Snoozing failure detection for 1.512s (starting election) I0703 01:43:57.174108 4488 raft_consensus.cc:2782] T 150033054dcf498d8d71fd996af55e88 P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:43:57.175278 4488 raft_consensus.cc:479] T 150033054dcf498d8d71fd996af55e88 P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } I0703 01:43:57.175523 4488 leader_election.cc:243] T 150033054dcf498d8d71fd996af55e88 P 8c1e8943f5654f71b030ec7c0b733742 [CANDIDATE]: Term 1 election: Requesting vote from peer 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746) I0703 01:43:57.176167 3696 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "150033054dcf498d8d71fd996af55e88" candidate_uuid: "8c1e8943f5654f71b030ec7c0b733742" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57bea6dc182c41a78304417a3cbfc4a7" I0703 01:43:57.176265 3696 raft_consensus.cc:2782] T 150033054dcf498d8d71fd996af55e88 P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:43:57.176316 3696 raft_consensus.cc:2730] T 150033054dcf498d8d71fd996af55e88 P 57bea6dc182c41a78304417a3cbfc4a7: Snoozing failure detection for 1.782s (vote granted) I0703 01:43:57.177459 3696 raft_consensus.cc:2243] T 150033054dcf498d8d71fd996af55e88 P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c1e8943f5654f71b030ec7c0b733742 in term 1. I0703 01:43:57.177852 3749 leader_election.cc:379] T 150033054dcf498d8d71fd996af55e88 P 8c1e8943f5654f71b030ec7c0b733742 [CANDIDATE]: Term 1 election: Vote granted by peer 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746) I0703 01:43:57.177909 3749 leader_election.cc:270] T 150033054dcf498d8d71fd996af55e88 P 8c1e8943f5654f71b030ec7c0b733742 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:43:57.178084 4488 raft_consensus.cc:2730] T 150033054dcf498d8d71fd996af55e88 P 8c1e8943f5654f71b030ec7c0b733742: Snoozing failure detection for 1.500s (election complete) I0703 01:43:57.178154 4488 raft_consensus.cc:2537] T 150033054dcf498d8d71fd996af55e88 P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:43:57.178268 4488 raft_consensus.cc:593] T 150033054dcf498d8d71fd996af55e88 P 8c1e8943f5654f71b030ec7c0b733742 [term 1 LEADER]: Becoming Leader. State: Replica: 8c1e8943f5654f71b030ec7c0b733742, State: Running, Role: LEADER I0703 01:43:57.178534 4488 consensus_queue.cc:204] T 150033054dcf498d8d71fd996af55e88 P 8c1e8943f5654f71b030ec7c0b733742 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } I0703 01:43:57.180240 3449 catalog_manager.cc:3798] T 150033054dcf498d8d71fd996af55e88 P 8c1e8943f5654f71b030ec7c0b733742 reported cstate change: term changed from 0 to 1, leader changed from to 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195). New cstate: current_term: 1 leader_uuid: "8c1e8943f5654f71b030ec7c0b733742" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } health_report { overall_health: UNKNOWN } } } I0703 01:43:57.296027 4461 raft_consensus.cc:457] T 2051955b4b644a03a9adad816d350596 P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0703 01:43:57.296121 4461 raft_consensus.cc:2730] T 2051955b4b644a03a9adad816d350596 P 55223945fe3d4dd8ac03ba9dab346f54: Snoozing failure detection for 1.813s (starting election) I0703 01:43:57.296232 4461 raft_consensus.cc:479] T 2051955b4b644a03a9adad816d350596 P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } I0703 01:43:57.296622 4461 leader_election.cc:243] T 2051955b4b644a03a9adad816d350596 P 55223945fe3d4dd8ac03ba9dab346f54 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746) I0703 01:43:57.297313 3696 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "2051955b4b644a03a9adad816d350596" candidate_uuid: "55223945fe3d4dd8ac03ba9dab346f54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57bea6dc182c41a78304417a3cbfc4a7" is_pre_election: true I0703 01:43:57.297452 3696 raft_consensus.cc:2730] T 2051955b4b644a03a9adad816d350596 P 57bea6dc182c41a78304417a3cbfc4a7: Snoozing failure detection for 2.223s (vote granted) I0703 01:43:57.297505 3696 raft_consensus.cc:2243] T 2051955b4b644a03a9adad816d350596 P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55223945fe3d4dd8ac03ba9dab346f54 in term 0. I0703 01:43:57.297931 3504 leader_election.cc:379] T 2051955b4b644a03a9adad816d350596 P 55223945fe3d4dd8ac03ba9dab346f54 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746) I0703 01:43:57.297991 3504 leader_election.cc:270] T 2051955b4b644a03a9adad816d350596 P 55223945fe3d4dd8ac03ba9dab346f54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0703 01:43:57.298188 4461 raft_consensus.cc:2730] T 2051955b4b644a03a9adad816d350596 P 55223945fe3d4dd8ac03ba9dab346f54: Snoozing failure detection for 1.770s (election complete) I0703 01:43:57.298252 4461 raft_consensus.cc:2537] T 2051955b4b644a03a9adad816d350596 P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0703 01:43:57.298315 4461 raft_consensus.cc:457] T 2051955b4b644a03a9adad816d350596 P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0703 01:43:57.298367 4461 raft_consensus.cc:2730] T 2051955b4b644a03a9adad816d350596 P 55223945fe3d4dd8ac03ba9dab346f54: Snoozing failure detection for 2.005s (starting election) I0703 01:43:57.298431 4461 raft_consensus.cc:2782] T 2051955b4b644a03a9adad816d350596 P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:43:57.299782 4461 raft_consensus.cc:479] T 2051955b4b644a03a9adad816d350596 P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } I0703 01:43:57.300015 4461 leader_election.cc:243] T 2051955b4b644a03a9adad816d350596 P 55223945fe3d4dd8ac03ba9dab346f54 [CANDIDATE]: Term 1 election: Requesting vote from peer 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746) I0703 01:43:57.300614 3696 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "2051955b4b644a03a9adad816d350596" candidate_uuid: "55223945fe3d4dd8ac03ba9dab346f54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57bea6dc182c41a78304417a3cbfc4a7" I0703 01:43:57.300700 3696 raft_consensus.cc:2782] T 2051955b4b644a03a9adad816d350596 P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:43:57.300762 3696 raft_consensus.cc:2730] T 2051955b4b644a03a9adad816d350596 P 57bea6dc182c41a78304417a3cbfc4a7: Snoozing failure detection for 1.707s (vote granted) I0703 01:43:57.301975 3696 raft_consensus.cc:2243] T 2051955b4b644a03a9adad816d350596 P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55223945fe3d4dd8ac03ba9dab346f54 in term 1. I0703 01:43:57.302321 3504 leader_election.cc:379] T 2051955b4b644a03a9adad816d350596 P 55223945fe3d4dd8ac03ba9dab346f54 [CANDIDATE]: Term 1 election: Vote granted by peer 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746) I0703 01:43:57.302376 3504 leader_election.cc:270] T 2051955b4b644a03a9adad816d350596 P 55223945fe3d4dd8ac03ba9dab346f54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:43:57.302565 4461 raft_consensus.cc:2730] T 2051955b4b644a03a9adad816d350596 P 55223945fe3d4dd8ac03ba9dab346f54: Snoozing failure detection for 1.837s (election complete) I0703 01:43:57.302633 4461 raft_consensus.cc:2537] T 2051955b4b644a03a9adad816d350596 P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:43:57.302930 4461 raft_consensus.cc:593] T 2051955b4b644a03a9adad816d350596 P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 LEADER]: Becoming Leader. State: Replica: 55223945fe3d4dd8ac03ba9dab346f54, State: Running, Role: LEADER I0703 01:43:57.303217 4461 consensus_queue.cc:204] T 2051955b4b644a03a9adad816d350596 P 55223945fe3d4dd8ac03ba9dab346f54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } I0703 01:43:57.305757 3449 catalog_manager.cc:3798] T 2051955b4b644a03a9adad816d350596 P 55223945fe3d4dd8ac03ba9dab346f54 reported cstate change: term changed from 0 to 1, leader changed from to 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193). New cstate: current_term: 1 leader_uuid: "55223945fe3d4dd8ac03ba9dab346f54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } health_report { overall_health: HEALTHY } } } W0703 01:43:57.587536 4544 flags.cc:406] Enabled unsafe flag: --never_fsync=true Using table 'rebalance_test_table_2' I0703 01:43:57.636757 3817 raft_consensus.cc:1081] T eea4e9534e0d4d25a827fae00bbeae4d P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Refusing update from remote peer 57bea6dc182c41a78304417a3cbfc4a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0703 01:43:57.637323 4470 consensus_queue.cc:962] T eea4e9534e0d4d25a827fae00bbeae4d P 57bea6dc182c41a78304417a3cbfc4a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0703 01:43:57.733718 3696 raft_consensus.cc:1081] T 150033054dcf498d8d71fd996af55e88 P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 FOLLOWER]: Refusing update from remote peer 8c1e8943f5654f71b030ec7c0b733742: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0703 01:43:57.734290 3695 raft_consensus.cc:1081] T 2051955b4b644a03a9adad816d350596 P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 FOLLOWER]: Refusing update from remote peer 55223945fe3d4dd8ac03ba9dab346f54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0703 01:43:57.734458 4488 consensus_queue.cc:962] T 150033054dcf498d8d71fd996af55e88 P 8c1e8943f5654f71b030ec7c0b733742 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0703 01:43:57.735055 4461 consensus_queue.cc:962] T 2051955b4b644a03a9adad816d350596 P 55223945fe3d4dd8ac03ba9dab346f54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s Generator report time total : 195.268 ms time per row: 0.0976342 ms I0703 01:43:57.883462 3444 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.5.240.254:58445: name: "rebalance_test_table_3" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 2 split_rows_range_bounds { } partition_schema { hash_bucket_schemas { columns { name: "key" } num_buckets: 3 seed: 0 } } I0703 01:43:57.890015 3796 tablet_service.cc:759] Processing CreateTablet for tablet 674ed5a88db04825a4eaec2fa996a852 (table=rebalance_test_table_3 [id=2d96fe2971a440c3b073db7c0073362b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I0703 01:43:57.890450 3796 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 674ed5a88db04825a4eaec2fa996a852. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:43:57.890396 3552 tablet_service.cc:759] Processing CreateTablet for tablet 674ed5a88db04825a4eaec2fa996a852 (table=rebalance_test_table_3 [id=2d96fe2971a440c3b073db7c0073362b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I0703 01:43:57.890625 3553 tablet_service.cc:759] Processing CreateTablet for tablet 60d3228294e945c5b343ae62e76dc2f0 (table=rebalance_test_table_3 [id=2d96fe2971a440c3b073db7c0073362b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I0703 01:43:57.890808 3552 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 674ed5a88db04825a4eaec2fa996a852. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:43:57.890977 3553 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 60d3228294e945c5b343ae62e76dc2f0. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:43:57.891640 3551 tablet_service.cc:759] Processing CreateTablet for tablet 6cc1e565d73247dc90536ad298c9562f (table=rebalance_test_table_3 [id=2d96fe2971a440c3b073db7c0073362b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I0703 01:43:57.892019 3551 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 6cc1e565d73247dc90536ad298c9562f. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:43:57.892962 3675 tablet_service.cc:759] Processing CreateTablet for tablet 6cc1e565d73247dc90536ad298c9562f (table=rebalance_test_table_3 [id=2d96fe2971a440c3b073db7c0073362b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I0703 01:43:57.893292 3675 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 6cc1e565d73247dc90536ad298c9562f. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:43:57.894783 3797 tablet_service.cc:759] Processing CreateTablet for tablet 60d3228294e945c5b343ae62e76dc2f0 (table=rebalance_test_table_3 [id=2d96fe2971a440c3b073db7c0073362b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I0703 01:43:57.895151 3797 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 60d3228294e945c5b343ae62e76dc2f0. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:43:57.898048 3796 ts_tablet_manager.cc:1173] T 674ed5a88db04825a4eaec2fa996a852 P 8c1e8943f5654f71b030ec7c0b733742: Registered tablet (data state: TABLET_DATA_READY) I0703 01:43:57.898340 3675 ts_tablet_manager.cc:1173] T 6cc1e565d73247dc90536ad298c9562f P 57bea6dc182c41a78304417a3cbfc4a7: Registered tablet (data state: TABLET_DATA_READY) I0703 01:43:57.903803 3797 ts_tablet_manager.cc:1173] T 60d3228294e945c5b343ae62e76dc2f0 P 8c1e8943f5654f71b030ec7c0b733742: Registered tablet (data state: TABLET_DATA_READY) I0703 01:43:57.904189 4583 ts_tablet_manager.cc:1028] T 674ed5a88db04825a4eaec2fa996a852 P 8c1e8943f5654f71b030ec7c0b733742: Bootstrapping tablet I0703 01:43:57.904335 4583 tablet_bootstrap.cc:427] T 674ed5a88db04825a4eaec2fa996a852 P 8c1e8943f5654f71b030ec7c0b733742: Bootstrap starting. I0703 01:43:57.904378 3553 ts_tablet_manager.cc:1173] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54: Registered tablet (data state: TABLET_DATA_READY) I0703 01:43:57.904670 3551 ts_tablet_manager.cc:1173] T 6cc1e565d73247dc90536ad298c9562f P 55223945fe3d4dd8ac03ba9dab346f54: Registered tablet (data state: TABLET_DATA_READY) I0703 01:43:57.905169 3552 ts_tablet_manager.cc:1173] T 674ed5a88db04825a4eaec2fa996a852 P 55223945fe3d4dd8ac03ba9dab346f54: Registered tablet (data state: TABLET_DATA_READY) I0703 01:43:57.905575 4583 tablet_bootstrap.cc:578] T 674ed5a88db04825a4eaec2fa996a852 P 8c1e8943f5654f71b030ec7c0b733742: No blocks or log segments found. Creating new log. I0703 01:43:57.908417 4585 ts_tablet_manager.cc:1028] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54: Bootstrapping tablet I0703 01:43:57.909013 4585 tablet_bootstrap.cc:427] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54: Bootstrap starting. I0703 01:43:57.909029 4584 ts_tablet_manager.cc:1028] T 6cc1e565d73247dc90536ad298c9562f P 57bea6dc182c41a78304417a3cbfc4a7: Bootstrapping tablet I0703 01:43:57.909235 4584 tablet_bootstrap.cc:427] T 6cc1e565d73247dc90536ad298c9562f P 57bea6dc182c41a78304417a3cbfc4a7: Bootstrap starting. I0703 01:43:57.910341 4585 tablet_bootstrap.cc:578] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54: No blocks or log segments found. Creating new log. I0703 01:43:57.910781 4584 tablet_bootstrap.cc:578] T 6cc1e565d73247dc90536ad298c9562f P 57bea6dc182c41a78304417a3cbfc4a7: No blocks or log segments found. Creating new log. I0703 01:43:57.911967 4583 tablet_bootstrap.cc:427] T 674ed5a88db04825a4eaec2fa996a852 P 8c1e8943f5654f71b030ec7c0b733742: No bootstrap required, opened a new log I0703 01:43:57.912061 4583 ts_tablet_manager.cc:1045] T 674ed5a88db04825a4eaec2fa996a852 P 8c1e8943f5654f71b030ec7c0b733742: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s I0703 01:43:57.912675 4583 raft_consensus.cc:325] T 674ed5a88db04825a4eaec2fa996a852 P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } I0703 01:43:57.912838 4583 raft_consensus.cc:351] T 674ed5a88db04825a4eaec2fa996a852 P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:43:57.912874 4583 raft_consensus.cc:628] T 674ed5a88db04825a4eaec2fa996a852 P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c1e8943f5654f71b030ec7c0b733742, State: Initialized, Role: FOLLOWER I0703 01:43:57.913017 4583 consensus_queue.cc:227] T 674ed5a88db04825a4eaec2fa996a852 P 8c1e8943f5654f71b030ec7c0b733742 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } I0703 01:43:57.913594 4583 ts_tablet_manager.cc:1073] T 674ed5a88db04825a4eaec2fa996a852 P 8c1e8943f5654f71b030ec7c0b733742: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s I0703 01:43:57.913746 4583 ts_tablet_manager.cc:1028] T 60d3228294e945c5b343ae62e76dc2f0 P 8c1e8943f5654f71b030ec7c0b733742: Bootstrapping tablet I0703 01:43:57.913877 4583 tablet_bootstrap.cc:427] T 60d3228294e945c5b343ae62e76dc2f0 P 8c1e8943f5654f71b030ec7c0b733742: Bootstrap starting. I0703 01:43:57.914950 4583 tablet_bootstrap.cc:578] T 60d3228294e945c5b343ae62e76dc2f0 P 8c1e8943f5654f71b030ec7c0b733742: No blocks or log segments found. Creating new log. I0703 01:43:57.918500 4585 tablet_bootstrap.cc:427] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54: No bootstrap required, opened a new log I0703 01:43:57.918601 4585 ts_tablet_manager.cc:1045] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s I0703 01:43:57.919132 4585 raft_consensus.cc:325] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } I0703 01:43:57.919272 4585 raft_consensus.cc:351] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:43:57.919327 4584 tablet_bootstrap.cc:427] T 6cc1e565d73247dc90536ad298c9562f P 57bea6dc182c41a78304417a3cbfc4a7: No bootstrap required, opened a new log I0703 01:43:57.919318 4585 raft_consensus.cc:628] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55223945fe3d4dd8ac03ba9dab346f54, State: Initialized, Role: FOLLOWER I0703 01:43:57.919462 4584 ts_tablet_manager.cc:1045] T 6cc1e565d73247dc90536ad298c9562f P 57bea6dc182c41a78304417a3cbfc4a7: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.006s I0703 01:43:57.919503 4585 consensus_queue.cc:227] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } I0703 01:43:57.919970 4585 ts_tablet_manager.cc:1073] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s I0703 01:43:57.920102 4585 ts_tablet_manager.cc:1028] T 6cc1e565d73247dc90536ad298c9562f P 55223945fe3d4dd8ac03ba9dab346f54: Bootstrapping tablet I0703 01:43:57.920198 4585 tablet_bootstrap.cc:427] T 6cc1e565d73247dc90536ad298c9562f P 55223945fe3d4dd8ac03ba9dab346f54: Bootstrap starting. I0703 01:43:57.920294 4584 raft_consensus.cc:325] T 6cc1e565d73247dc90536ad298c9562f P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } I0703 01:43:57.920611 4584 raft_consensus.cc:351] T 6cc1e565d73247dc90536ad298c9562f P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:43:57.920666 4584 raft_consensus.cc:628] T 6cc1e565d73247dc90536ad298c9562f P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57bea6dc182c41a78304417a3cbfc4a7, State: Initialized, Role: FOLLOWER I0703 01:43:57.921176 4585 tablet_bootstrap.cc:578] T 6cc1e565d73247dc90536ad298c9562f P 55223945fe3d4dd8ac03ba9dab346f54: No blocks or log segments found. Creating new log. I0703 01:43:57.920825 4584 consensus_queue.cc:227] T 6cc1e565d73247dc90536ad298c9562f P 57bea6dc182c41a78304417a3cbfc4a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } I0703 01:43:57.922011 4584 ts_tablet_manager.cc:1073] T 6cc1e565d73247dc90536ad298c9562f P 57bea6dc182c41a78304417a3cbfc4a7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s I0703 01:43:57.923606 4583 tablet_bootstrap.cc:427] T 60d3228294e945c5b343ae62e76dc2f0 P 8c1e8943f5654f71b030ec7c0b733742: No bootstrap required, opened a new log I0703 01:43:57.923759 4583 ts_tablet_manager.cc:1045] T 60d3228294e945c5b343ae62e76dc2f0 P 8c1e8943f5654f71b030ec7c0b733742: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.002s I0703 01:43:57.923777 4585 tablet_bootstrap.cc:427] T 6cc1e565d73247dc90536ad298c9562f P 55223945fe3d4dd8ac03ba9dab346f54: No bootstrap required, opened a new log I0703 01:43:57.923952 4585 ts_tablet_manager.cc:1045] T 6cc1e565d73247dc90536ad298c9562f P 55223945fe3d4dd8ac03ba9dab346f54: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s I0703 01:43:57.924397 4585 raft_consensus.cc:325] T 6cc1e565d73247dc90536ad298c9562f P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } I0703 01:43:57.924579 4585 raft_consensus.cc:351] T 6cc1e565d73247dc90536ad298c9562f P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:43:57.925019 4585 raft_consensus.cc:628] T 6cc1e565d73247dc90536ad298c9562f P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55223945fe3d4dd8ac03ba9dab346f54, State: Initialized, Role: FOLLOWER I0703 01:43:57.925035 4583 raft_consensus.cc:325] T 60d3228294e945c5b343ae62e76dc2f0 P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } I0703 01:43:57.925191 4583 raft_consensus.cc:351] T 60d3228294e945c5b343ae62e76dc2f0 P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:43:57.925266 4583 raft_consensus.cc:628] T 60d3228294e945c5b343ae62e76dc2f0 P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c1e8943f5654f71b030ec7c0b733742, State: Initialized, Role: FOLLOWER I0703 01:43:57.925526 4583 consensus_queue.cc:227] T 60d3228294e945c5b343ae62e76dc2f0 P 8c1e8943f5654f71b030ec7c0b733742 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } I0703 01:43:57.925715 4585 consensus_queue.cc:227] T 6cc1e565d73247dc90536ad298c9562f P 55223945fe3d4dd8ac03ba9dab346f54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } I0703 01:43:57.929595 4585 ts_tablet_manager.cc:1073] T 6cc1e565d73247dc90536ad298c9562f P 55223945fe3d4dd8ac03ba9dab346f54: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s I0703 01:43:57.929797 4585 ts_tablet_manager.cc:1028] T 674ed5a88db04825a4eaec2fa996a852 P 55223945fe3d4dd8ac03ba9dab346f54: Bootstrapping tablet I0703 01:43:57.929941 4585 tablet_bootstrap.cc:427] T 674ed5a88db04825a4eaec2fa996a852 P 55223945fe3d4dd8ac03ba9dab346f54: Bootstrap starting. I0703 01:43:57.931399 4585 tablet_bootstrap.cc:578] T 674ed5a88db04825a4eaec2fa996a852 P 55223945fe3d4dd8ac03ba9dab346f54: No blocks or log segments found. Creating new log. I0703 01:43:57.931810 4583 ts_tablet_manager.cc:1073] T 60d3228294e945c5b343ae62e76dc2f0 P 8c1e8943f5654f71b030ec7c0b733742: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s I0703 01:43:57.936631 4585 tablet_bootstrap.cc:427] T 674ed5a88db04825a4eaec2fa996a852 P 55223945fe3d4dd8ac03ba9dab346f54: No bootstrap required, opened a new log I0703 01:43:57.936759 4585 ts_tablet_manager.cc:1045] T 674ed5a88db04825a4eaec2fa996a852 P 55223945fe3d4dd8ac03ba9dab346f54: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.003s I0703 01:43:57.937259 4585 raft_consensus.cc:325] T 674ed5a88db04825a4eaec2fa996a852 P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } I0703 01:43:57.937448 4585 raft_consensus.cc:351] T 674ed5a88db04825a4eaec2fa996a852 P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:43:57.937536 4585 raft_consensus.cc:628] T 674ed5a88db04825a4eaec2fa996a852 P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55223945fe3d4dd8ac03ba9dab346f54, State: Initialized, Role: FOLLOWER I0703 01:43:57.937716 4585 consensus_queue.cc:227] T 674ed5a88db04825a4eaec2fa996a852 P 55223945fe3d4dd8ac03ba9dab346f54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } I0703 01:43:57.938571 4585 ts_tablet_manager.cc:1073] T 674ed5a88db04825a4eaec2fa996a852 P 55223945fe3d4dd8ac03ba9dab346f54: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s I0703 01:43:58.246176 4542 raft_consensus.cc:457] T 6cc1e565d73247dc90536ad298c9562f P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0703 01:43:58.246495 4542 raft_consensus.cc:2730] T 6cc1e565d73247dc90536ad298c9562f P 55223945fe3d4dd8ac03ba9dab346f54: Snoozing failure detection for 1.689s (starting election) I0703 01:43:58.246623 4542 raft_consensus.cc:479] T 6cc1e565d73247dc90536ad298c9562f P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } I0703 01:43:58.246938 4542 leader_election.cc:243] T 6cc1e565d73247dc90536ad298c9562f P 55223945fe3d4dd8ac03ba9dab346f54 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746) I0703 01:43:58.247735 3695 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "6cc1e565d73247dc90536ad298c9562f" candidate_uuid: "55223945fe3d4dd8ac03ba9dab346f54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57bea6dc182c41a78304417a3cbfc4a7" is_pre_election: true I0703 01:43:58.247902 3695 raft_consensus.cc:2730] T 6cc1e565d73247dc90536ad298c9562f P 57bea6dc182c41a78304417a3cbfc4a7: Snoozing failure detection for 1.584s (vote granted) I0703 01:43:58.247968 3695 raft_consensus.cc:2243] T 6cc1e565d73247dc90536ad298c9562f P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55223945fe3d4dd8ac03ba9dab346f54 in term 0. I0703 01:43:58.248468 3504 leader_election.cc:379] T 6cc1e565d73247dc90536ad298c9562f P 55223945fe3d4dd8ac03ba9dab346f54 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746) I0703 01:43:58.248535 3504 leader_election.cc:270] T 6cc1e565d73247dc90536ad298c9562f P 55223945fe3d4dd8ac03ba9dab346f54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0703 01:43:58.248785 4542 raft_consensus.cc:2730] T 6cc1e565d73247dc90536ad298c9562f P 55223945fe3d4dd8ac03ba9dab346f54: Snoozing failure detection for 1.760s (election complete) I0703 01:43:58.248852 4542 raft_consensus.cc:2537] T 6cc1e565d73247dc90536ad298c9562f P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0703 01:43:58.248906 4542 raft_consensus.cc:457] T 6cc1e565d73247dc90536ad298c9562f P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0703 01:43:58.248953 4542 raft_consensus.cc:2730] T 6cc1e565d73247dc90536ad298c9562f P 55223945fe3d4dd8ac03ba9dab346f54: Snoozing failure detection for 1.554s (starting election) I0703 01:43:58.249013 4542 raft_consensus.cc:2782] T 6cc1e565d73247dc90536ad298c9562f P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:43:58.250380 4542 raft_consensus.cc:479] T 6cc1e565d73247dc90536ad298c9562f P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } I0703 01:43:58.250649 4542 leader_election.cc:243] T 6cc1e565d73247dc90536ad298c9562f P 55223945fe3d4dd8ac03ba9dab346f54 [CANDIDATE]: Term 1 election: Requesting vote from peer 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746) I0703 01:43:58.251242 3695 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "6cc1e565d73247dc90536ad298c9562f" candidate_uuid: "55223945fe3d4dd8ac03ba9dab346f54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57bea6dc182c41a78304417a3cbfc4a7" I0703 01:43:58.251348 3695 raft_consensus.cc:2782] T 6cc1e565d73247dc90536ad298c9562f P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:43:58.251422 3695 raft_consensus.cc:2730] T 6cc1e565d73247dc90536ad298c9562f P 57bea6dc182c41a78304417a3cbfc4a7: Snoozing failure detection for 2.246s (vote granted) I0703 01:43:58.252689 3695 raft_consensus.cc:2243] T 6cc1e565d73247dc90536ad298c9562f P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55223945fe3d4dd8ac03ba9dab346f54 in term 1. I0703 01:43:58.253103 3504 leader_election.cc:379] T 6cc1e565d73247dc90536ad298c9562f P 55223945fe3d4dd8ac03ba9dab346f54 [CANDIDATE]: Term 1 election: Vote granted by peer 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746) I0703 01:43:58.253159 3504 leader_election.cc:270] T 6cc1e565d73247dc90536ad298c9562f P 55223945fe3d4dd8ac03ba9dab346f54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:43:58.253330 4542 raft_consensus.cc:2730] T 6cc1e565d73247dc90536ad298c9562f P 55223945fe3d4dd8ac03ba9dab346f54: Snoozing failure detection for 1.643s (election complete) I0703 01:43:58.253388 4542 raft_consensus.cc:2537] T 6cc1e565d73247dc90536ad298c9562f P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:43:58.253494 4542 raft_consensus.cc:593] T 6cc1e565d73247dc90536ad298c9562f P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 LEADER]: Becoming Leader. State: Replica: 55223945fe3d4dd8ac03ba9dab346f54, State: Running, Role: LEADER I0703 01:43:58.253859 4542 consensus_queue.cc:204] T 6cc1e565d73247dc90536ad298c9562f P 55223945fe3d4dd8ac03ba9dab346f54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } I0703 01:43:58.255532 3447 catalog_manager.cc:3798] T 6cc1e565d73247dc90536ad298c9562f P 55223945fe3d4dd8ac03ba9dab346f54 reported cstate change: term changed from 0 to 1, leader changed from to 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193). New cstate: current_term: 1 leader_uuid: "55223945fe3d4dd8ac03ba9dab346f54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } health_report { overall_health: HEALTHY } } } I0703 01:43:58.258555 4542 raft_consensus.cc:457] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0703 01:43:58.258610 4542 raft_consensus.cc:2730] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54: Snoozing failure detection for 1.949s (starting election) I0703 01:43:58.258664 4542 raft_consensus.cc:479] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } I0703 01:43:58.258882 4542 leader_election.cc:243] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064) I0703 01:43:58.263278 3817 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "60d3228294e945c5b343ae62e76dc2f0" candidate_uuid: "55223945fe3d4dd8ac03ba9dab346f54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c1e8943f5654f71b030ec7c0b733742" is_pre_election: true I0703 01:43:58.263414 3817 raft_consensus.cc:2730] T 60d3228294e945c5b343ae62e76dc2f0 P 8c1e8943f5654f71b030ec7c0b733742: Snoozing failure detection for 2.227s (vote granted) I0703 01:43:58.263471 3817 raft_consensus.cc:2243] T 60d3228294e945c5b343ae62e76dc2f0 P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55223945fe3d4dd8ac03ba9dab346f54 in term 0. I0703 01:43:58.263878 3507 leader_election.cc:379] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064) I0703 01:43:58.263933 3507 leader_election.cc:270] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0703 01:43:58.264109 4542 raft_consensus.cc:2730] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54: Snoozing failure detection for 2.145s (election complete) I0703 01:43:58.264170 4542 raft_consensus.cc:2537] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0703 01:43:58.264207 4542 raft_consensus.cc:457] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0703 01:43:58.264246 4542 raft_consensus.cc:2730] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54: Snoozing failure detection for 1.942s (starting election) I0703 01:43:58.264286 4542 raft_consensus.cc:2782] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:43:58.265477 4542 raft_consensus.cc:479] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } I0703 01:43:58.265719 4542 leader_election.cc:243] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 [CANDIDATE]: Term 1 election: Requesting vote from peer 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064) I0703 01:43:58.266301 3817 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "60d3228294e945c5b343ae62e76dc2f0" candidate_uuid: "55223945fe3d4dd8ac03ba9dab346f54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c1e8943f5654f71b030ec7c0b733742" I0703 01:43:58.266398 3817 raft_consensus.cc:2782] T 60d3228294e945c5b343ae62e76dc2f0 P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:43:58.266454 3817 raft_consensus.cc:2730] T 60d3228294e945c5b343ae62e76dc2f0 P 8c1e8943f5654f71b030ec7c0b733742: Snoozing failure detection for 2.066s (vote granted) I0703 01:43:58.267737 3817 raft_consensus.cc:2243] T 60d3228294e945c5b343ae62e76dc2f0 P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55223945fe3d4dd8ac03ba9dab346f54 in term 1. I0703 01:43:58.268067 3507 leader_election.cc:379] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 [CANDIDATE]: Term 1 election: Vote granted by peer 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064) I0703 01:43:58.268121 3507 leader_election.cc:270] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:43:58.268292 4542 raft_consensus.cc:2730] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54: Snoozing failure detection for 2.028s (election complete) I0703 01:43:58.268349 4542 raft_consensus.cc:2537] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:43:58.268452 4542 raft_consensus.cc:593] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 LEADER]: Becoming Leader. State: Replica: 55223945fe3d4dd8ac03ba9dab346f54, State: Running, Role: LEADER I0703 01:43:58.268661 4542 consensus_queue.cc:204] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } I0703 01:43:58.270390 3447 catalog_manager.cc:3798] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 reported cstate change: term changed from 0 to 1, leader changed from to 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193). New cstate: current_term: 1 leader_uuid: "55223945fe3d4dd8ac03ba9dab346f54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } health_report { overall_health: UNKNOWN } } } I0703 01:43:58.299939 4572 raft_consensus.cc:457] T 674ed5a88db04825a4eaec2fa996a852 P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0703 01:43:58.299989 4572 raft_consensus.cc:2730] T 674ed5a88db04825a4eaec2fa996a852 P 8c1e8943f5654f71b030ec7c0b733742: Snoozing failure detection for 1.781s (starting election) I0703 01:43:58.300062 4572 raft_consensus.cc:479] T 674ed5a88db04825a4eaec2fa996a852 P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } I0703 01:43:58.300348 4572 leader_election.cc:243] T 674ed5a88db04825a4eaec2fa996a852 P 8c1e8943f5654f71b030ec7c0b733742 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193:54919) I0703 01:43:58.300966 3572 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "674ed5a88db04825a4eaec2fa996a852" candidate_uuid: "8c1e8943f5654f71b030ec7c0b733742" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55223945fe3d4dd8ac03ba9dab346f54" is_pre_election: true I0703 01:43:58.301100 3572 raft_consensus.cc:2730] T 674ed5a88db04825a4eaec2fa996a852 P 55223945fe3d4dd8ac03ba9dab346f54: Snoozing failure detection for 1.754s (vote granted) I0703 01:43:58.301164 3572 raft_consensus.cc:2243] T 674ed5a88db04825a4eaec2fa996a852 P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c1e8943f5654f71b030ec7c0b733742 in term 0. I0703 01:43:58.301652 3752 leader_election.cc:379] T 674ed5a88db04825a4eaec2fa996a852 P 8c1e8943f5654f71b030ec7c0b733742 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193:54919) I0703 01:43:58.301736 3752 leader_election.cc:270] T 674ed5a88db04825a4eaec2fa996a852 P 8c1e8943f5654f71b030ec7c0b733742 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0703 01:43:58.302016 4572 raft_consensus.cc:2730] T 674ed5a88db04825a4eaec2fa996a852 P 8c1e8943f5654f71b030ec7c0b733742: Snoozing failure detection for 1.701s (election complete) I0703 01:43:58.302110 4572 raft_consensus.cc:2537] T 674ed5a88db04825a4eaec2fa996a852 P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0703 01:43:58.302196 4572 raft_consensus.cc:457] T 674ed5a88db04825a4eaec2fa996a852 P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0703 01:43:58.302278 4572 raft_consensus.cc:2730] T 674ed5a88db04825a4eaec2fa996a852 P 8c1e8943f5654f71b030ec7c0b733742: Snoozing failure detection for 1.656s (starting election) I0703 01:43:58.302366 4572 raft_consensus.cc:2782] T 674ed5a88db04825a4eaec2fa996a852 P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:43:58.303709 4572 raft_consensus.cc:479] T 674ed5a88db04825a4eaec2fa996a852 P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } I0703 01:43:58.303972 4572 leader_election.cc:243] T 674ed5a88db04825a4eaec2fa996a852 P 8c1e8943f5654f71b030ec7c0b733742 [CANDIDATE]: Term 1 election: Requesting vote from peer 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193:54919) I0703 01:43:58.304530 3572 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "674ed5a88db04825a4eaec2fa996a852" candidate_uuid: "8c1e8943f5654f71b030ec7c0b733742" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55223945fe3d4dd8ac03ba9dab346f54" I0703 01:43:58.304636 3572 raft_consensus.cc:2782] T 674ed5a88db04825a4eaec2fa996a852 P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:43:58.304689 3572 raft_consensus.cc:2730] T 674ed5a88db04825a4eaec2fa996a852 P 55223945fe3d4dd8ac03ba9dab346f54: Snoozing failure detection for 2.108s (vote granted) I0703 01:43:58.305912 3572 raft_consensus.cc:2243] T 674ed5a88db04825a4eaec2fa996a852 P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c1e8943f5654f71b030ec7c0b733742 in term 1. I0703 01:43:58.306257 3752 leader_election.cc:379] T 674ed5a88db04825a4eaec2fa996a852 P 8c1e8943f5654f71b030ec7c0b733742 [CANDIDATE]: Term 1 election: Vote granted by peer 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193:54919) I0703 01:43:58.306320 3752 leader_election.cc:270] T 674ed5a88db04825a4eaec2fa996a852 P 8c1e8943f5654f71b030ec7c0b733742 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:43:58.306493 4572 raft_consensus.cc:2730] T 674ed5a88db04825a4eaec2fa996a852 P 8c1e8943f5654f71b030ec7c0b733742: Snoozing failure detection for 1.584s (election complete) I0703 01:43:58.306573 4572 raft_consensus.cc:2537] T 674ed5a88db04825a4eaec2fa996a852 P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:43:58.306839 4572 raft_consensus.cc:593] T 674ed5a88db04825a4eaec2fa996a852 P 8c1e8943f5654f71b030ec7c0b733742 [term 1 LEADER]: Becoming Leader. State: Replica: 8c1e8943f5654f71b030ec7c0b733742, State: Running, Role: LEADER I0703 01:43:58.307113 4572 consensus_queue.cc:204] T 674ed5a88db04825a4eaec2fa996a852 P 8c1e8943f5654f71b030ec7c0b733742 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } I0703 01:43:58.308797 3447 catalog_manager.cc:3798] T 674ed5a88db04825a4eaec2fa996a852 P 8c1e8943f5654f71b030ec7c0b733742 reported cstate change: term changed from 0 to 1, leader changed from to 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195). New cstate: current_term: 1 leader_uuid: "8c1e8943f5654f71b030ec7c0b733742" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } health_report { overall_health: UNKNOWN } } } W0703 01:43:58.559656 4598 flags.cc:406] Enabled unsafe flag: --never_fsync=true Using table 'rebalance_test_table_3' I0703 01:43:58.684322 3572 raft_consensus.cc:1081] T 674ed5a88db04825a4eaec2fa996a852 P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 FOLLOWER]: Refusing update from remote peer 8c1e8943f5654f71b030ec7c0b733742: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0703 01:43:58.685006 4572 consensus_queue.cc:962] T 674ed5a88db04825a4eaec2fa996a852 P 8c1e8943f5654f71b030ec7c0b733742 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0703 01:43:58.697180 3817 raft_consensus.cc:1081] T 60d3228294e945c5b343ae62e76dc2f0 P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Refusing update from remote peer 55223945fe3d4dd8ac03ba9dab346f54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0703 01:43:58.698125 4542 consensus_queue.cc:962] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0703 01:43:58.707757 3695 raft_consensus.cc:1081] T 6cc1e565d73247dc90536ad298c9562f P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 FOLLOWER]: Refusing update from remote peer 55223945fe3d4dd8ac03ba9dab346f54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0703 01:43:58.709025 4461 consensus_queue.cc:962] T 6cc1e565d73247dc90536ad298c9562f P 55223945fe3d4dd8ac03ba9dab346f54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s Generator report time total : 194.928 ms time per row: 0.097464 ms I0703 01:43:58.847226 3444 catalog_manager.cc:1339] Servicing CreateTable request from {username='jenkins'} at 127.5.240.254:58445: name: "rebalance_test_table_4" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "int_val" type: INT32 is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } columns { name: "string_val" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 } } num_replicas: 2 split_rows_range_bounds { } partition_schema { hash_bucket_schemas { columns { name: "key" } num_buckets: 3 seed: 0 } } I0703 01:43:58.854056 3797 tablet_service.cc:759] Processing CreateTablet for tablet 25b33ca4b32a4a2493d03564bfe1bf7e (table=rebalance_test_table_4 [id=d5e3657f924944ca895ef4125aaa82a1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I0703 01:43:58.854221 3675 tablet_service.cc:759] Processing CreateTablet for tablet 25b33ca4b32a4a2493d03564bfe1bf7e (table=rebalance_test_table_4 [id=d5e3657f924944ca895ef4125aaa82a1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED I0703 01:43:58.854460 3797 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 25b33ca4b32a4a2493d03564bfe1bf7e. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:43:58.854446 3674 tablet_service.cc:759] Processing CreateTablet for tablet 4a613c1be55c4888a4df36d3f1bfc54c (table=rebalance_test_table_4 [id=d5e3657f924944ca895ef4125aaa82a1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I0703 01:43:58.854605 3675 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 25b33ca4b32a4a2493d03564bfe1bf7e. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:43:58.854777 3674 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 4a613c1be55c4888a4df36d3f1bfc54c. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:43:58.855808 3553 tablet_service.cc:759] Processing CreateTablet for tablet 6c8253ac9a9a405885f1f57c59a84fd5 (table=rebalance_test_table_4 [id=d5e3657f924944ca895ef4125aaa82a1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I0703 01:43:58.856137 3553 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 6c8253ac9a9a405885f1f57c59a84fd5. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:43:58.858865 3797 ts_tablet_manager.cc:1173] T 25b33ca4b32a4a2493d03564bfe1bf7e P 8c1e8943f5654f71b030ec7c0b733742: Registered tablet (data state: TABLET_DATA_READY) I0703 01:43:58.859566 3796 tablet_service.cc:759] Processing CreateTablet for tablet 4a613c1be55c4888a4df36d3f1bfc54c (table=rebalance_test_table_4 [id=d5e3657f924944ca895ef4125aaa82a1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED I0703 01:43:58.859901 3796 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 4a613c1be55c4888a4df36d3f1bfc54c. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:43:58.860496 3674 ts_tablet_manager.cc:1173] T 4a613c1be55c4888a4df36d3f1bfc54c P 57bea6dc182c41a78304417a3cbfc4a7: Registered tablet (data state: TABLET_DATA_READY) I0703 01:43:58.860621 3675 ts_tablet_manager.cc:1173] T 25b33ca4b32a4a2493d03564bfe1bf7e P 57bea6dc182c41a78304417a3cbfc4a7: Registered tablet (data state: TABLET_DATA_READY) I0703 01:43:58.861023 4634 ts_tablet_manager.cc:1028] T 25b33ca4b32a4a2493d03564bfe1bf7e P 8c1e8943f5654f71b030ec7c0b733742: Bootstrapping tablet I0703 01:43:58.861253 4634 tablet_bootstrap.cc:427] T 25b33ca4b32a4a2493d03564bfe1bf7e P 8c1e8943f5654f71b030ec7c0b733742: Bootstrap starting. I0703 01:43:58.861637 3676 tablet_service.cc:759] Processing CreateTablet for tablet 6c8253ac9a9a405885f1f57c59a84fd5 (table=rebalance_test_table_4 [id=d5e3657f924944ca895ef4125aaa82a1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED I0703 01:43:58.862048 3676 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 6c8253ac9a9a405885f1f57c59a84fd5. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:43:58.862982 4634 tablet_bootstrap.cc:578] T 25b33ca4b32a4a2493d03564bfe1bf7e P 8c1e8943f5654f71b030ec7c0b733742: No blocks or log segments found. Creating new log. I0703 01:43:58.863996 4635 ts_tablet_manager.cc:1028] T 25b33ca4b32a4a2493d03564bfe1bf7e P 57bea6dc182c41a78304417a3cbfc4a7: Bootstrapping tablet I0703 01:43:58.864151 4635 tablet_bootstrap.cc:427] T 25b33ca4b32a4a2493d03564bfe1bf7e P 57bea6dc182c41a78304417a3cbfc4a7: Bootstrap starting. I0703 01:43:58.864557 3553 ts_tablet_manager.cc:1173] T 6c8253ac9a9a405885f1f57c59a84fd5 P 55223945fe3d4dd8ac03ba9dab346f54: Registered tablet (data state: TABLET_DATA_READY) I0703 01:43:58.865486 4635 tablet_bootstrap.cc:578] T 25b33ca4b32a4a2493d03564bfe1bf7e P 57bea6dc182c41a78304417a3cbfc4a7: No blocks or log segments found. Creating new log. I0703 01:43:58.866684 4637 ts_tablet_manager.cc:1028] T 6c8253ac9a9a405885f1f57c59a84fd5 P 55223945fe3d4dd8ac03ba9dab346f54: Bootstrapping tablet I0703 01:43:58.866816 4637 tablet_bootstrap.cc:427] T 6c8253ac9a9a405885f1f57c59a84fd5 P 55223945fe3d4dd8ac03ba9dab346f54: Bootstrap starting. I0703 01:43:58.867260 3796 ts_tablet_manager.cc:1173] T 4a613c1be55c4888a4df36d3f1bfc54c P 8c1e8943f5654f71b030ec7c0b733742: Registered tablet (data state: TABLET_DATA_READY) I0703 01:43:58.868582 3676 ts_tablet_manager.cc:1173] T 6c8253ac9a9a405885f1f57c59a84fd5 P 57bea6dc182c41a78304417a3cbfc4a7: Registered tablet (data state: TABLET_DATA_READY) I0703 01:43:58.868857 4637 tablet_bootstrap.cc:578] T 6c8253ac9a9a405885f1f57c59a84fd5 P 55223945fe3d4dd8ac03ba9dab346f54: No blocks or log segments found. Creating new log. I0703 01:43:58.869385 4634 tablet_bootstrap.cc:427] T 25b33ca4b32a4a2493d03564bfe1bf7e P 8c1e8943f5654f71b030ec7c0b733742: No bootstrap required, opened a new log I0703 01:43:58.869395 4635 tablet_bootstrap.cc:427] T 25b33ca4b32a4a2493d03564bfe1bf7e P 57bea6dc182c41a78304417a3cbfc4a7: No bootstrap required, opened a new log I0703 01:43:58.869482 4634 ts_tablet_manager.cc:1045] T 25b33ca4b32a4a2493d03564bfe1bf7e P 8c1e8943f5654f71b030ec7c0b733742: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s I0703 01:43:58.869496 4635 ts_tablet_manager.cc:1045] T 25b33ca4b32a4a2493d03564bfe1bf7e P 57bea6dc182c41a78304417a3cbfc4a7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s I0703 01:43:58.870015 4634 raft_consensus.cc:325] T 25b33ca4b32a4a2493d03564bfe1bf7e P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } I0703 01:43:58.870151 4634 raft_consensus.cc:351] T 25b33ca4b32a4a2493d03564bfe1bf7e P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:43:58.870194 4634 raft_consensus.cc:628] T 25b33ca4b32a4a2493d03564bfe1bf7e P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c1e8943f5654f71b030ec7c0b733742, State: Initialized, Role: FOLLOWER I0703 01:43:58.870095 4635 raft_consensus.cc:325] T 25b33ca4b32a4a2493d03564bfe1bf7e P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } I0703 01:43:58.870249 4635 raft_consensus.cc:351] T 25b33ca4b32a4a2493d03564bfe1bf7e P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:43:58.870285 4635 raft_consensus.cc:628] T 25b33ca4b32a4a2493d03564bfe1bf7e P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57bea6dc182c41a78304417a3cbfc4a7, State: Initialized, Role: FOLLOWER I0703 01:43:58.870353 4634 consensus_queue.cc:227] T 25b33ca4b32a4a2493d03564bfe1bf7e P 8c1e8943f5654f71b030ec7c0b733742 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } I0703 01:43:58.870453 4635 consensus_queue.cc:227] T 25b33ca4b32a4a2493d03564bfe1bf7e P 57bea6dc182c41a78304417a3cbfc4a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } I0703 01:43:58.870951 4634 ts_tablet_manager.cc:1073] T 25b33ca4b32a4a2493d03564bfe1bf7e P 8c1e8943f5654f71b030ec7c0b733742: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s I0703 01:43:58.871004 4635 ts_tablet_manager.cc:1073] T 25b33ca4b32a4a2493d03564bfe1bf7e P 57bea6dc182c41a78304417a3cbfc4a7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s I0703 01:43:58.871095 4634 ts_tablet_manager.cc:1028] T 4a613c1be55c4888a4df36d3f1bfc54c P 8c1e8943f5654f71b030ec7c0b733742: Bootstrapping tablet I0703 01:43:58.871125 4635 ts_tablet_manager.cc:1028] T 4a613c1be55c4888a4df36d3f1bfc54c P 57bea6dc182c41a78304417a3cbfc4a7: Bootstrapping tablet I0703 01:43:58.871206 4634 tablet_bootstrap.cc:427] T 4a613c1be55c4888a4df36d3f1bfc54c P 8c1e8943f5654f71b030ec7c0b733742: Bootstrap starting. I0703 01:43:58.871218 4635 tablet_bootstrap.cc:427] T 4a613c1be55c4888a4df36d3f1bfc54c P 57bea6dc182c41a78304417a3cbfc4a7: Bootstrap starting. I0703 01:43:58.872347 4634 tablet_bootstrap.cc:578] T 4a613c1be55c4888a4df36d3f1bfc54c P 8c1e8943f5654f71b030ec7c0b733742: No blocks or log segments found. Creating new log. I0703 01:43:58.873255 4635 tablet_bootstrap.cc:578] T 4a613c1be55c4888a4df36d3f1bfc54c P 57bea6dc182c41a78304417a3cbfc4a7: No blocks or log segments found. Creating new log. I0703 01:43:58.874945 4637 tablet_bootstrap.cc:427] T 6c8253ac9a9a405885f1f57c59a84fd5 P 55223945fe3d4dd8ac03ba9dab346f54: No bootstrap required, opened a new log I0703 01:43:58.875051 4637 ts_tablet_manager.cc:1045] T 6c8253ac9a9a405885f1f57c59a84fd5 P 55223945fe3d4dd8ac03ba9dab346f54: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s I0703 01:43:58.875566 4634 tablet_bootstrap.cc:427] T 4a613c1be55c4888a4df36d3f1bfc54c P 8c1e8943f5654f71b030ec7c0b733742: No bootstrap required, opened a new log I0703 01:43:58.875664 4634 ts_tablet_manager.cc:1045] T 4a613c1be55c4888a4df36d3f1bfc54c P 8c1e8943f5654f71b030ec7c0b733742: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s I0703 01:43:58.875738 4637 raft_consensus.cc:325] T 6c8253ac9a9a405885f1f57c59a84fd5 P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } I0703 01:43:58.875900 4637 raft_consensus.cc:351] T 6c8253ac9a9a405885f1f57c59a84fd5 P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:43:58.875952 4637 raft_consensus.cc:628] T 6c8253ac9a9a405885f1f57c59a84fd5 P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55223945fe3d4dd8ac03ba9dab346f54, State: Initialized, Role: FOLLOWER I0703 01:43:58.876116 4637 consensus_queue.cc:227] T 6c8253ac9a9a405885f1f57c59a84fd5 P 55223945fe3d4dd8ac03ba9dab346f54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } I0703 01:43:58.876237 4634 raft_consensus.cc:325] T 4a613c1be55c4888a4df36d3f1bfc54c P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } I0703 01:43:58.876363 4634 raft_consensus.cc:351] T 4a613c1be55c4888a4df36d3f1bfc54c P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:43:58.876411 4634 raft_consensus.cc:628] T 4a613c1be55c4888a4df36d3f1bfc54c P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c1e8943f5654f71b030ec7c0b733742, State: Initialized, Role: FOLLOWER I0703 01:43:58.876605 4634 consensus_queue.cc:227] T 4a613c1be55c4888a4df36d3f1bfc54c P 8c1e8943f5654f71b030ec7c0b733742 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } I0703 01:43:58.876742 4637 ts_tablet_manager.cc:1073] T 6c8253ac9a9a405885f1f57c59a84fd5 P 55223945fe3d4dd8ac03ba9dab346f54: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s I0703 01:43:58.877262 4634 ts_tablet_manager.cc:1073] T 4a613c1be55c4888a4df36d3f1bfc54c P 8c1e8943f5654f71b030ec7c0b733742: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s I0703 01:43:58.877727 4635 tablet_bootstrap.cc:427] T 4a613c1be55c4888a4df36d3f1bfc54c P 57bea6dc182c41a78304417a3cbfc4a7: No bootstrap required, opened a new log I0703 01:43:58.877838 4635 ts_tablet_manager.cc:1045] T 4a613c1be55c4888a4df36d3f1bfc54c P 57bea6dc182c41a78304417a3cbfc4a7: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s I0703 01:43:58.878459 4635 raft_consensus.cc:325] T 4a613c1be55c4888a4df36d3f1bfc54c P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } I0703 01:43:58.878571 4635 raft_consensus.cc:351] T 4a613c1be55c4888a4df36d3f1bfc54c P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:43:58.878657 4635 raft_consensus.cc:628] T 4a613c1be55c4888a4df36d3f1bfc54c P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57bea6dc182c41a78304417a3cbfc4a7, State: Initialized, Role: FOLLOWER I0703 01:43:58.878870 4635 consensus_queue.cc:227] T 4a613c1be55c4888a4df36d3f1bfc54c P 57bea6dc182c41a78304417a3cbfc4a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } I0703 01:43:58.879554 4635 ts_tablet_manager.cc:1073] T 4a613c1be55c4888a4df36d3f1bfc54c P 57bea6dc182c41a78304417a3cbfc4a7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s I0703 01:43:58.879707 4635 ts_tablet_manager.cc:1028] T 6c8253ac9a9a405885f1f57c59a84fd5 P 57bea6dc182c41a78304417a3cbfc4a7: Bootstrapping tablet I0703 01:43:58.879814 4635 tablet_bootstrap.cc:427] T 6c8253ac9a9a405885f1f57c59a84fd5 P 57bea6dc182c41a78304417a3cbfc4a7: Bootstrap starting. I0703 01:43:58.880858 4635 tablet_bootstrap.cc:578] T 6c8253ac9a9a405885f1f57c59a84fd5 P 57bea6dc182c41a78304417a3cbfc4a7: No blocks or log segments found. Creating new log. I0703 01:43:58.882356 4635 tablet_bootstrap.cc:427] T 6c8253ac9a9a405885f1f57c59a84fd5 P 57bea6dc182c41a78304417a3cbfc4a7: No bootstrap required, opened a new log I0703 01:43:58.882447 4635 ts_tablet_manager.cc:1045] T 6c8253ac9a9a405885f1f57c59a84fd5 P 57bea6dc182c41a78304417a3cbfc4a7: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s I0703 01:43:58.882890 4635 raft_consensus.cc:325] T 6c8253ac9a9a405885f1f57c59a84fd5 P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } I0703 01:43:58.883008 4635 raft_consensus.cc:351] T 6c8253ac9a9a405885f1f57c59a84fd5 P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely I0703 01:43:58.883055 4635 raft_consensus.cc:628] T 6c8253ac9a9a405885f1f57c59a84fd5 P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57bea6dc182c41a78304417a3cbfc4a7, State: Initialized, Role: FOLLOWER I0703 01:43:58.883217 4635 consensus_queue.cc:227] T 6c8253ac9a9a405885f1f57c59a84fd5 P 57bea6dc182c41a78304417a3cbfc4a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } I0703 01:43:58.883702 4635 ts_tablet_manager.cc:1073] T 6c8253ac9a9a405885f1f57c59a84fd5 P 57bea6dc182c41a78304417a3cbfc4a7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s I0703 01:43:58.945196 4556 raft_consensus.cc:457] T 4a613c1be55c4888a4df36d3f1bfc54c P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0703 01:43:58.945271 4556 raft_consensus.cc:2730] T 4a613c1be55c4888a4df36d3f1bfc54c P 57bea6dc182c41a78304417a3cbfc4a7: Snoozing failure detection for 1.807s (starting election) I0703 01:43:58.945353 4556 raft_consensus.cc:479] T 4a613c1be55c4888a4df36d3f1bfc54c P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } I0703 01:43:58.945637 4556 leader_election.cc:243] T 4a613c1be55c4888a4df36d3f1bfc54c P 57bea6dc182c41a78304417a3cbfc4a7 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064) I0703 01:43:58.946262 3817 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "4a613c1be55c4888a4df36d3f1bfc54c" candidate_uuid: "57bea6dc182c41a78304417a3cbfc4a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c1e8943f5654f71b030ec7c0b733742" is_pre_election: true I0703 01:43:58.946389 3817 raft_consensus.cc:2730] T 4a613c1be55c4888a4df36d3f1bfc54c P 8c1e8943f5654f71b030ec7c0b733742: Snoozing failure detection for 1.728s (vote granted) I0703 01:43:58.946463 3817 raft_consensus.cc:2243] T 4a613c1be55c4888a4df36d3f1bfc54c P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57bea6dc182c41a78304417a3cbfc4a7 in term 0. I0703 01:43:58.946902 3630 leader_election.cc:379] T 4a613c1be55c4888a4df36d3f1bfc54c P 57bea6dc182c41a78304417a3cbfc4a7 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064) I0703 01:43:58.946955 3630 leader_election.cc:270] T 4a613c1be55c4888a4df36d3f1bfc54c P 57bea6dc182c41a78304417a3cbfc4a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0703 01:43:58.947121 4556 raft_consensus.cc:2730] T 4a613c1be55c4888a4df36d3f1bfc54c P 57bea6dc182c41a78304417a3cbfc4a7: Snoozing failure detection for 1.730s (election complete) I0703 01:43:58.947182 4556 raft_consensus.cc:2537] T 4a613c1be55c4888a4df36d3f1bfc54c P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0703 01:43:58.947245 4556 raft_consensus.cc:457] T 4a613c1be55c4888a4df36d3f1bfc54c P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0703 01:43:58.947299 4556 raft_consensus.cc:2730] T 4a613c1be55c4888a4df36d3f1bfc54c P 57bea6dc182c41a78304417a3cbfc4a7: Snoozing failure detection for 1.720s (starting election) I0703 01:43:58.947358 4556 raft_consensus.cc:2782] T 4a613c1be55c4888a4df36d3f1bfc54c P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:43:58.948622 4556 raft_consensus.cc:479] T 4a613c1be55c4888a4df36d3f1bfc54c P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } I0703 01:43:58.948881 4556 leader_election.cc:243] T 4a613c1be55c4888a4df36d3f1bfc54c P 57bea6dc182c41a78304417a3cbfc4a7 [CANDIDATE]: Term 1 election: Requesting vote from peer 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064) I0703 01:43:58.949455 3817 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "4a613c1be55c4888a4df36d3f1bfc54c" candidate_uuid: "57bea6dc182c41a78304417a3cbfc4a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c1e8943f5654f71b030ec7c0b733742" I0703 01:43:58.949582 3817 raft_consensus.cc:2782] T 4a613c1be55c4888a4df36d3f1bfc54c P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:43:58.949648 3817 raft_consensus.cc:2730] T 4a613c1be55c4888a4df36d3f1bfc54c P 8c1e8943f5654f71b030ec7c0b733742: Snoozing failure detection for 1.933s (vote granted) I0703 01:43:58.950883 3817 raft_consensus.cc:2243] T 4a613c1be55c4888a4df36d3f1bfc54c P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57bea6dc182c41a78304417a3cbfc4a7 in term 1. I0703 01:43:58.951243 3630 leader_election.cc:379] T 4a613c1be55c4888a4df36d3f1bfc54c P 57bea6dc182c41a78304417a3cbfc4a7 [CANDIDATE]: Term 1 election: Vote granted by peer 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064) I0703 01:43:58.951292 3630 leader_election.cc:270] T 4a613c1be55c4888a4df36d3f1bfc54c P 57bea6dc182c41a78304417a3cbfc4a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:43:58.951442 4556 raft_consensus.cc:2730] T 4a613c1be55c4888a4df36d3f1bfc54c P 57bea6dc182c41a78304417a3cbfc4a7: Snoozing failure detection for 2.167s (election complete) I0703 01:43:58.951511 4556 raft_consensus.cc:2537] T 4a613c1be55c4888a4df36d3f1bfc54c P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:43:58.951824 4556 raft_consensus.cc:593] T 4a613c1be55c4888a4df36d3f1bfc54c P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 LEADER]: Becoming Leader. State: Replica: 57bea6dc182c41a78304417a3cbfc4a7, State: Running, Role: LEADER I0703 01:43:58.952033 4556 consensus_queue.cc:204] T 4a613c1be55c4888a4df36d3f1bfc54c P 57bea6dc182c41a78304417a3cbfc4a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } I0703 01:43:58.953857 3445 catalog_manager.cc:3798] T 4a613c1be55c4888a4df36d3f1bfc54c P 57bea6dc182c41a78304417a3cbfc4a7 reported cstate change: term changed from 0 to 1, leader changed from to 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194). New cstate: current_term: 1 leader_uuid: "57bea6dc182c41a78304417a3cbfc4a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } health_report { overall_health: HEALTHY } } } I0703 01:43:59.005996 4619 raft_consensus.cc:457] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0703 01:43:59.006067 4619 raft_consensus.cc:2730] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742: Snoozing failure detection for 3.377s (starting election) I0703 01:43:59.006160 4619 raft_consensus.cc:479] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } } peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } I0703 01:43:59.006570 4619 leader_election.cc:243] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377) W0703 01:43:59.007534 3749 leader_election.cc:299] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377): Network error: Client connection negotiation failed: client connection to 127.5.240.199:55377: connect: Connection refused (error 111) I0703 01:43:59.007598 3749 leader_election.cc:270] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0703 01:43:59.007797 4619 raft_consensus.cc:2730] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742: Snoozing failure detection for 3.926s (election complete) I0703 01:43:59.007881 4619 raft_consensus.cc:2485] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0703 01:43:59.011324 4619 raft_consensus.cc:457] T 25b33ca4b32a4a2493d03564bfe1bf7e P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0703 01:43:59.011378 4619 raft_consensus.cc:2730] T 25b33ca4b32a4a2493d03564bfe1bf7e P 8c1e8943f5654f71b030ec7c0b733742: Snoozing failure detection for 2.173s (starting election) I0703 01:43:59.011448 4619 raft_consensus.cc:479] T 25b33ca4b32a4a2493d03564bfe1bf7e P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } I0703 01:43:59.011694 4619 leader_election.cc:243] T 25b33ca4b32a4a2493d03564bfe1bf7e P 8c1e8943f5654f71b030ec7c0b733742 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746) I0703 01:43:59.012315 3695 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "25b33ca4b32a4a2493d03564bfe1bf7e" candidate_uuid: "8c1e8943f5654f71b030ec7c0b733742" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57bea6dc182c41a78304417a3cbfc4a7" is_pre_election: true I0703 01:43:59.012449 3695 raft_consensus.cc:2730] T 25b33ca4b32a4a2493d03564bfe1bf7e P 57bea6dc182c41a78304417a3cbfc4a7: Snoozing failure detection for 1.644s (vote granted) I0703 01:43:59.012509 3695 raft_consensus.cc:2243] T 25b33ca4b32a4a2493d03564bfe1bf7e P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c1e8943f5654f71b030ec7c0b733742 in term 0. I0703 01:43:59.012923 3749 leader_election.cc:379] T 25b33ca4b32a4a2493d03564bfe1bf7e P 8c1e8943f5654f71b030ec7c0b733742 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746) I0703 01:43:59.012974 3749 leader_election.cc:270] T 25b33ca4b32a4a2493d03564bfe1bf7e P 8c1e8943f5654f71b030ec7c0b733742 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0703 01:43:59.013164 4619 raft_consensus.cc:2730] T 25b33ca4b32a4a2493d03564bfe1bf7e P 8c1e8943f5654f71b030ec7c0b733742: Snoozing failure detection for 2.230s (election complete) I0703 01:43:59.013222 4619 raft_consensus.cc:2537] T 25b33ca4b32a4a2493d03564bfe1bf7e P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0703 01:43:59.013283 4619 raft_consensus.cc:457] T 25b33ca4b32a4a2493d03564bfe1bf7e P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0703 01:43:59.013331 4619 raft_consensus.cc:2730] T 25b33ca4b32a4a2493d03564bfe1bf7e P 8c1e8943f5654f71b030ec7c0b733742: Snoozing failure detection for 2.164s (starting election) I0703 01:43:59.013386 4619 raft_consensus.cc:2782] T 25b33ca4b32a4a2493d03564bfe1bf7e P 8c1e8943f5654f71b030ec7c0b733742 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:43:59.014681 4619 raft_consensus.cc:479] T 25b33ca4b32a4a2493d03564bfe1bf7e P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } I0703 01:43:59.014917 4619 leader_election.cc:243] T 25b33ca4b32a4a2493d03564bfe1bf7e P 8c1e8943f5654f71b030ec7c0b733742 [CANDIDATE]: Term 1 election: Requesting vote from peer 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746) I0703 01:43:59.015524 3695 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "25b33ca4b32a4a2493d03564bfe1bf7e" candidate_uuid: "8c1e8943f5654f71b030ec7c0b733742" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57bea6dc182c41a78304417a3cbfc4a7" I0703 01:43:59.015643 3695 raft_consensus.cc:2782] T 25b33ca4b32a4a2493d03564bfe1bf7e P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:43:59.015712 3695 raft_consensus.cc:2730] T 25b33ca4b32a4a2493d03564bfe1bf7e P 57bea6dc182c41a78304417a3cbfc4a7: Snoozing failure detection for 1.943s (vote granted) I0703 01:43:59.016947 3695 raft_consensus.cc:2243] T 25b33ca4b32a4a2493d03564bfe1bf7e P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c1e8943f5654f71b030ec7c0b733742 in term 1. I0703 01:43:59.017318 3749 leader_election.cc:379] T 25b33ca4b32a4a2493d03564bfe1bf7e P 8c1e8943f5654f71b030ec7c0b733742 [CANDIDATE]: Term 1 election: Vote granted by peer 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746) I0703 01:43:59.017369 3749 leader_election.cc:270] T 25b33ca4b32a4a2493d03564bfe1bf7e P 8c1e8943f5654f71b030ec7c0b733742 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:43:59.017537 4619 raft_consensus.cc:2730] T 25b33ca4b32a4a2493d03564bfe1bf7e P 8c1e8943f5654f71b030ec7c0b733742: Snoozing failure detection for 1.558s (election complete) I0703 01:43:59.017634 4619 raft_consensus.cc:2537] T 25b33ca4b32a4a2493d03564bfe1bf7e P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:43:59.017719 4619 raft_consensus.cc:593] T 25b33ca4b32a4a2493d03564bfe1bf7e P 8c1e8943f5654f71b030ec7c0b733742 [term 1 LEADER]: Becoming Leader. State: Replica: 8c1e8943f5654f71b030ec7c0b733742, State: Running, Role: LEADER I0703 01:43:59.017997 4619 consensus_queue.cc:204] T 25b33ca4b32a4a2493d03564bfe1bf7e P 8c1e8943f5654f71b030ec7c0b733742 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } I0703 01:43:59.019623 3445 catalog_manager.cc:3798] T 25b33ca4b32a4a2493d03564bfe1bf7e P 8c1e8943f5654f71b030ec7c0b733742 reported cstate change: term changed from 0 to 1, leader changed from to 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195). New cstate: current_term: 1 leader_uuid: "8c1e8943f5654f71b030ec7c0b733742" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } health_report { overall_health: UNKNOWN } } } I0703 01:43:59.114102 4542 raft_consensus.cc:457] T 6c8253ac9a9a405885f1f57c59a84fd5 P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0703 01:43:59.114168 4542 raft_consensus.cc:2730] T 6c8253ac9a9a405885f1f57c59a84fd5 P 55223945fe3d4dd8ac03ba9dab346f54: Snoozing failure detection for 1.975s (starting election) I0703 01:43:59.114244 4542 raft_consensus.cc:479] T 6c8253ac9a9a405885f1f57c59a84fd5 P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } I0703 01:43:59.114534 4542 leader_election.cc:243] T 6c8253ac9a9a405885f1f57c59a84fd5 P 55223945fe3d4dd8ac03ba9dab346f54 [CANDIDATE]: Term 1 pre-election: Requesting pre-vote from peer 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746) I0703 01:43:59.115162 3695 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "6c8253ac9a9a405885f1f57c59a84fd5" candidate_uuid: "55223945fe3d4dd8ac03ba9dab346f54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57bea6dc182c41a78304417a3cbfc4a7" is_pre_election: true I0703 01:43:59.115293 3695 raft_consensus.cc:2730] T 6c8253ac9a9a405885f1f57c59a84fd5 P 57bea6dc182c41a78304417a3cbfc4a7: Snoozing failure detection for 1.583s (vote granted) I0703 01:43:59.115360 3695 raft_consensus.cc:2243] T 6c8253ac9a9a405885f1f57c59a84fd5 P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55223945fe3d4dd8ac03ba9dab346f54 in term 0. I0703 01:43:59.115775 3504 leader_election.cc:379] T 6c8253ac9a9a405885f1f57c59a84fd5 P 55223945fe3d4dd8ac03ba9dab346f54 [CANDIDATE]: Term 1 pre-election: Vote granted by peer 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746) I0703 01:43:59.115835 3504 leader_election.cc:270] T 6c8253ac9a9a405885f1f57c59a84fd5 P 55223945fe3d4dd8ac03ba9dab346f54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. I0703 01:43:59.116024 4542 raft_consensus.cc:2730] T 6c8253ac9a9a405885f1f57c59a84fd5 P 55223945fe3d4dd8ac03ba9dab346f54: Snoozing failure detection for 1.784s (election complete) I0703 01:43:59.116113 4542 raft_consensus.cc:2537] T 6c8253ac9a9a405885f1f57c59a84fd5 P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Leader pre-election won for term 1 I0703 01:43:59.116163 4542 raft_consensus.cc:457] T 6c8253ac9a9a405885f1f57c59a84fd5 P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0703 01:43:59.116225 4542 raft_consensus.cc:2730] T 6c8253ac9a9a405885f1f57c59a84fd5 P 55223945fe3d4dd8ac03ba9dab346f54: Snoozing failure detection for 2.085s (starting election) I0703 01:43:59.116277 4542 raft_consensus.cc:2782] T 6c8253ac9a9a405885f1f57c59a84fd5 P 55223945fe3d4dd8ac03ba9dab346f54 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:43:59.117557 4542 raft_consensus.cc:479] T 6c8253ac9a9a405885f1f57c59a84fd5 P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } I0703 01:43:59.117812 4542 leader_election.cc:243] T 6c8253ac9a9a405885f1f57c59a84fd5 P 55223945fe3d4dd8ac03ba9dab346f54 [CANDIDATE]: Term 1 election: Requesting vote from peer 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746) I0703 01:43:59.118424 3695 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "6c8253ac9a9a405885f1f57c59a84fd5" candidate_uuid: "55223945fe3d4dd8ac03ba9dab346f54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57bea6dc182c41a78304417a3cbfc4a7" I0703 01:43:59.118557 3695 raft_consensus.cc:2782] T 6c8253ac9a9a405885f1f57c59a84fd5 P 57bea6dc182c41a78304417a3cbfc4a7 [term 0 FOLLOWER]: Advancing to term 1 I0703 01:43:59.118621 3695 raft_consensus.cc:2730] T 6c8253ac9a9a405885f1f57c59a84fd5 P 57bea6dc182c41a78304417a3cbfc4a7: Snoozing failure detection for 1.673s (vote granted) I0703 01:43:59.119802 3695 raft_consensus.cc:2243] T 6c8253ac9a9a405885f1f57c59a84fd5 P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55223945fe3d4dd8ac03ba9dab346f54 in term 1. I0703 01:43:59.120244 3504 leader_election.cc:379] T 6c8253ac9a9a405885f1f57c59a84fd5 P 55223945fe3d4dd8ac03ba9dab346f54 [CANDIDATE]: Term 1 election: Vote granted by peer 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746) I0703 01:43:59.120299 3504 leader_election.cc:270] T 6c8253ac9a9a405885f1f57c59a84fd5 P 55223945fe3d4dd8ac03ba9dab346f54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. I0703 01:43:59.120468 4542 raft_consensus.cc:2730] T 6c8253ac9a9a405885f1f57c59a84fd5 P 55223945fe3d4dd8ac03ba9dab346f54: Snoozing failure detection for 1.631s (election complete) I0703 01:43:59.120522 4542 raft_consensus.cc:2537] T 6c8253ac9a9a405885f1f57c59a84fd5 P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 FOLLOWER]: Leader election won for term 1 I0703 01:43:59.120656 4542 raft_consensus.cc:593] T 6c8253ac9a9a405885f1f57c59a84fd5 P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 LEADER]: Becoming Leader. State: Replica: 55223945fe3d4dd8ac03ba9dab346f54, State: Running, Role: LEADER I0703 01:43:59.120839 4542 consensus_queue.cc:204] T 6c8253ac9a9a405885f1f57c59a84fd5 P 55223945fe3d4dd8ac03ba9dab346f54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } I0703 01:43:59.123299 3445 catalog_manager.cc:3798] T 6c8253ac9a9a405885f1f57c59a84fd5 P 55223945fe3d4dd8ac03ba9dab346f54 reported cstate change: term changed from 0 to 1, leader changed from to 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193). New cstate: current_term: 1 leader_uuid: "55223945fe3d4dd8ac03ba9dab346f54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } health_report { overall_health: UNKNOWN } } } W0703 01:43:59.375185 4648 flags.cc:406] Enabled unsafe flag: --never_fsync=true Using table 'rebalance_test_table_4' I0703 01:43:59.470650 3817 raft_consensus.cc:1081] T 4a613c1be55c4888a4df36d3f1bfc54c P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Refusing update from remote peer 57bea6dc182c41a78304417a3cbfc4a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 1. (index mismatch) I0703 01:43:59.471367 4556 consensus_queue.cc:962] T 4a613c1be55c4888a4df36d3f1bfc54c P 57bea6dc182c41a78304417a3cbfc4a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0703 01:43:59.517742 3696 raft_consensus.cc:1081] T 25b33ca4b32a4a2493d03564bfe1bf7e P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 FOLLOWER]: Refusing update from remote peer 8c1e8943f5654f71b030ec7c0b733742: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0703 01:43:59.517742 3695 raft_consensus.cc:1081] T 6c8253ac9a9a405885f1f57c59a84fd5 P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 FOLLOWER]: Refusing update from remote peer 55223945fe3d4dd8ac03ba9dab346f54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch) I0703 01:43:59.518434 4619 consensus_queue.cc:962] T 25b33ca4b32a4a2493d03564bfe1bf7e P 8c1e8943f5654f71b030ec7c0b733742 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s I0703 01:43:59.518726 4542 consensus_queue.cc:962] T 6c8253ac9a9a405885f1f57c59a84fd5 P 55223945fe3d4dd8ac03ba9dab346f54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s Generator report time total : 194.627 ms time per row: 0.0973134 ms I0703 01:43:59.665871 6083 test_util.cc:208] Using random seed: 1354455744 I0703 01:43:59.675922 6083 test_workload.cc:306] TestWorkload: Skipping table creation because table rebalance_test_table_0 already exists I0703 01:43:59.676277 6083 test_util.cc:208] Using random seed: 1354466082 I0703 01:43:59.692059 6083 test_workload.cc:306] TestWorkload: Skipping table creation because table rebalance_test_table_1 already exists I0703 01:43:59.692468 6083 test_util.cc:208] Using random seed: 1354482246 I0703 01:43:59.719909 6083 test_workload.cc:306] TestWorkload: Skipping table creation because table rebalance_test_table_2 already exists I0703 01:43:59.720474 6083 test_util.cc:208] Using random seed: 1354510262 I0703 01:43:59.752926 6083 test_workload.cc:306] TestWorkload: Skipping table creation because table rebalance_test_table_3 already exists I0703 01:43:59.753326 6083 test_util.cc:208] Using random seed: 1354543104 I0703 01:43:59.800746 6083 test_workload.cc:306] TestWorkload: Skipping table creation because table rebalance_test_table_4 already exists I0703 01:43:59.805415 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data --block_manager=log --rpc_bind_addresses=127.5.240.196:51288 --local_ip_for_outbound_sockets=127.5.240.196 --webserver_port=38993 --webserver_interface=127.5.240.196 --tserver_master_addrs=127.5.240.254:32866 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:44:00.413764 4737 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:44:00.416404 4737 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:44:00.422999 4737 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0703 01:44:00.423465 4737 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:44:00.423672 4737 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.196 I0703 01:44:00.447984 4737 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.196:51288 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb --webserver_interface=127.5.240.196 --webserver_port=38993 --tserver_master_addrs=127.5.240.254:32866 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.196 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:44:00.475411 4737 tablet_server_main.cc:85] Initializing tablet server... I0703 01:44:00.477833 4737 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 273708us I0703 01:44:00.478659 4737 fs_manager.cc:261] Metadata directory not provided I0703 01:44:00.478734 4737 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:44:00.500514 4737 fs_manager.cc:397] Time spent opening directory manager: real 0.019s user 0.003s sys 0.007s I0703 01:44:00.503100 4737 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:44:00.506327 4737 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:44:00.518977 4737 fs_manager.cc:417] Time spent opening block manager: real 0.010s user 0.000s sys 0.005s I0703 01:44:00.519143 4737 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal uuid: "213596374e4142dca6ecfc701ba1c566" format_stamp: "Formatted at 2018-07-03 08:43:49 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:44:00.528827 4737 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:44:00.778638 4737 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:44:00.782904 4737 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:44:00.796846 4737 tablet_server_main.cc:90] Starting tablet server... I0703 01:44:01.554973 4737 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.196:51288 I0703 01:44:01.555680 4737 webserver.cc:175] Starting webserver on 127.5.240.196:38993 I0703 01:44:01.555775 4737 webserver.cc:186] Document root disabled I0703 01:44:01.561987 4737 webserver.cc:313] Webserver started. Bound to: http://127.5.240.196:38993/ I0703 01:44:01.562891 4737 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-3/data/info.pb I0703 01:44:01.570022 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 4737 I0703 01:44:01.570540 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data --block_manager=log --rpc_bind_addresses=127.5.240.197:51329 --local_ip_for_outbound_sockets=127.5.240.197 --webserver_port=36989 --webserver_interface=127.5.240.197 --tserver_master_addrs=127.5.240.254:32866 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0703 01:44:01.622362 4737 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:44:01.663198 4878 heartbeater.cc:346] Connected to a master server at 127.5.240.254:32866 I0703 01:44:01.663399 4878 heartbeater.cc:426] Registering TS with master... I0703 01:44:01.664808 4878 heartbeater.cc:475] Master 127.5.240.254:32866 requested a full tablet report, sending... I0703 01:44:01.668190 3445 ts_manager.cc:104] Re-registered known tserver with Master: 213596374e4142dca6ecfc701ba1c566 (127.5.240.196:51288) I0703 01:44:01.669530 3445 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.196:35894 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:44:02.119719 4882 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:44:02.121129 4882 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:44:02.127099 4882 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0703 01:44:02.127209 4882 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:44:02.127393 4882 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.197 I0703 01:44:02.160580 4882 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.197:51329 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb --webserver_interface=127.5.240.197 --webserver_port=36989 --tserver_master_addrs=127.5.240.254:32866 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.197 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:44:02.174374 4882 tablet_server_main.cc:85] Initializing tablet server... I0703 01:44:02.180281 4882 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 274708us I0703 01:44:02.182333 4882 fs_manager.cc:261] Metadata directory not provided I0703 01:44:02.183106 4882 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:44:02.228852 4882 fs_manager.cc:397] Time spent opening directory manager: real 0.041s user 0.005s sys 0.007s I0703 01:44:02.230106 4882 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:44:02.230739 4882 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:44:02.249524 4882 fs_manager.cc:417] Time spent opening block manager: real 0.014s user 0.000s sys 0.001s I0703 01:44:02.250319 4882 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal uuid: "aa8b9c58b466436faa5086e85b5224a7" format_stamp: "Formatted at 2018-07-03 08:43:49 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:44:02.254485 4882 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:44:02.402040 4882 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:44:02.410348 4882 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:44:02.417254 4882 tablet_server_main.cc:90] Starting tablet server... I0703 01:44:02.672775 4878 heartbeater.cc:467] Master 127.5.240.254:32866 was elected leader, sending a full tablet report... I0703 01:44:02.934367 4744 raft_consensus.cc:457] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0703 01:44:02.934517 4744 raft_consensus.cc:2730] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742: Snoozing failure detection for 3.738s (starting election) I0703 01:44:02.934639 4744 raft_consensus.cc:479] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } } peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } I0703 01:44:02.935058 4744 leader_election.cc:243] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377) W0703 01:44:02.943457 3749 leader_election.cc:299] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377): Network error: Client connection negotiation failed: client connection to 127.5.240.199:55377: connect: Connection refused (error 111) I0703 01:44:02.943809 3749 leader_election.cc:270] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0703 01:44:02.944245 4744 raft_consensus.cc:2730] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742: Snoozing failure detection for 6.423s (election complete) I0703 01:44:02.944339 4744 raft_consensus.cc:2485] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0703 01:44:03.279245 4882 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.197:51329 I0703 01:44:03.280242 4882 webserver.cc:175] Starting webserver on 127.5.240.197:36989 I0703 01:44:03.280321 4882 webserver.cc:186] Document root disabled I0703 01:44:03.286434 4882 webserver.cc:313] Webserver started. Bound to: http://127.5.240.197:36989/ I0703 01:44:03.287322 4882 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-4/data/info.pb I0703 01:44:03.290390 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 4882 I0703 01:44:03.290995 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data --block_manager=log --rpc_bind_addresses=127.5.240.198:51033 --local_ip_for_outbound_sockets=127.5.240.198 --webserver_port=39793 --webserver_interface=127.5.240.198 --tserver_master_addrs=127.5.240.254:32866 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0703 01:44:03.323959 4882 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:44:03.386956 5011 heartbeater.cc:346] Connected to a master server at 127.5.240.254:32866 I0703 01:44:03.387136 5011 heartbeater.cc:426] Registering TS with master... I0703 01:44:03.388106 5011 heartbeater.cc:475] Master 127.5.240.254:32866 requested a full tablet report, sending... I0703 01:44:03.392540 3445 ts_manager.cc:104] Re-registered known tserver with Master: aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197:51329) I0703 01:44:03.393898 3445 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.197:56687 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:44:03.857817 5012 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:44:03.859058 5012 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:44:03.868079 5012 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0703 01:44:03.868185 5012 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:44:03.868366 5012 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.198 I0703 01:44:03.899731 5012 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.198:51033 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data/info.pb --webserver_interface=127.5.240.198 --webserver_port=39793 --tserver_master_addrs=127.5.240.254:32866 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.198 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:44:03.914300 5012 tablet_server_main.cc:85] Initializing tablet server... I0703 01:44:03.917342 5012 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 275208us I0703 01:44:03.919054 5012 fs_manager.cc:261] Metadata directory not provided I0703 01:44:03.919178 5012 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:44:03.989671 5012 fs_manager.cc:397] Time spent opening directory manager: real 0.067s user 0.004s sys 0.010s I0703 01:44:03.990756 5012 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:44:03.993619 5012 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:44:04.006578 5012 fs_manager.cc:417] Time spent opening block manager: real 0.007s user 0.002s sys 0.003s I0703 01:44:04.006686 5012 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/wal uuid: "6932f1cd716a48fe8d6c5573acfc48bf" format_stamp: "Formatted at 2018-07-03 08:43:50 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:44:04.010712 5012 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:44:04.224413 5012 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:44:04.232925 5012 ts_tablet_manager.cc:356] Loaded tablet metadata (0 live tablets) I0703 01:44:04.238965 5012 tablet_server_main.cc:90] Starting tablet server... I0703 01:44:04.396445 5011 heartbeater.cc:467] Master 127.5.240.254:32866 was elected leader, sending a full tablet report... I0703 01:44:05.042763 5012 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.198:51033 I0703 01:44:05.043423 5012 webserver.cc:175] Starting webserver on 127.5.240.198:39793 I0703 01:44:05.043503 5012 webserver.cc:186] Document root disabled I0703 01:44:05.050876 5012 webserver.cc:313] Webserver started. Bound to: http://127.5.240.198:39793/ I0703 01:44:05.053010 5012 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-5/data/info.pb I0703 01:44:05.061079 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 5012 I0703 01:44:05.061638 6083 external_mini_cluster.cc:794] Running /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/wal --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data --block_manager=log --rpc_bind_addresses=127.5.240.199:55377 --local_ip_for_outbound_sockets=127.5.240.199 --webserver_port=38435 --webserver_interface=127.5.240.199 --tserver_master_addrs=127.5.240.254:32866 --never_fsync --ipki_server_key_size=1024 --enable_minidumps=false --redact=none --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/logs --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data/info.pb --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports --unlock_experimental_flags --unlock_unsafe_flags --raft_prepare_replacement_before_eviction=true with env {} I0703 01:44:05.082864 5012 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:44:05.146419 5147 heartbeater.cc:346] Connected to a master server at 127.5.240.254:32866 I0703 01:44:05.146630 5147 heartbeater.cc:426] Registering TS with master... I0703 01:44:05.148022 5147 heartbeater.cc:475] Master 127.5.240.254:32866 requested a full tablet report, sending... I0703 01:44:05.149616 3445 ts_manager.cc:104] Re-registered known tserver with Master: 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198:51033) I0703 01:44:05.150934 3445 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.198:42414 WARNING: Logging before InitGoogleLogging() is written to STDERR W0703 01:44:05.560632 5151 flags.cc:406] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true W0703 01:44:05.562736 5151 flags.cc:406] Enabled unsafe flag: --never_fsync=true W0703 01:44:05.570025 5151 flags.cc:406] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true W0703 01:44:05.570138 5151 flags.cc:406] Enabled experimental flag: --ipki_server_key_size=1024 W0703 01:44:05.570305 5151 flags.cc:406] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.240.199 I0703 01:44:05.602610 5151 tablet_server_main.cc:78] Tablet server non-default flags: --fs_data_dirs=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data --fs_wal_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/wal --ipki_server_key_size=1024 --rpc_bind_addresses=127.5.240.199:55377 --rpc_server_allow_ephemeral_ports=true --metrics_log_interval_ms=1000 --server_dump_info_format=pb --server_dump_info_path=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data/info.pb --webserver_interface=127.5.240.199 --webserver_port=38435 --tserver_master_addrs=127.5.240.254:32866 --never_fsync=true --redact=none --unlock_experimental_flags=true --unlock_unsafe_flags=true --enable_minidumps=false --local_ip_for_outbound_sockets=127.5.240.199 --log_dir=/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/logs --logbuflevel=-1 --logtostderr=true Tablet server version: kudu 1.8.0-SNAPSHOT revision 1ae050e4d57bc333de28bcbc62e072e8bafd04b3 build type FASTDEBUG built by jenkins at 03 Jul 2018 00:28:42 PST on quasar-kdakzq-1.vpc.cloudera.com build id 91347 ASAN enabled I0703 01:44:05.614758 5151 tablet_server_main.cc:85] Initializing tablet server... I0703 01:44:05.618062 5151 system_ntp.cc:199] NTP initialized. Skew: 500ppm Current error: 276208us I0703 01:44:05.619962 5151 fs_manager.cc:261] Metadata directory not provided I0703 01:44:05.620084 5151 fs_manager.cc:267] Using write-ahead log directory (fs_wal_dir) as metadata directory I0703 01:44:05.640910 5151 fs_manager.cc:397] Time spent opening directory manager: real 0.018s user 0.003s sys 0.008s I0703 01:44:05.642040 5151 env_posix.cc:1674] Not raising this process' open files per process limit of 32768; it is already as high as it can go I0703 01:44:05.642644 5151 file_cache.cc:470] Constructed file cache lbm with capacity 13107 I0703 01:44:05.654543 5151 fs_manager.cc:417] Time spent opening block manager: real 0.008s user 0.000s sys 0.005s I0703 01:44:05.654672 5151 fs_manager.cc:434] Opened local filesystem: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data,/data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/wal uuid: "1157c91cc6f54a68a1d99017ac8444aa" format_stamp: "Formatted at 2018-07-03 08:43:50 on quasar-kdakzq-1.vpc.cloudera.com" I0703 01:44:05.656337 5151 fs_report.cc:352] FS layout report -------------------- wal directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/wal metadata directory: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/wal 1 data directories: /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data/data Total live blocks: 0 Total live bytes: 0 Total live bytes (after alignment): 0 Total number of LBM containers: 0 (0 full) Did not check for missing blocks Did not check for orphaned blocks Total full LBM containers with extra space: 0 (0 repaired) Total full LBM container extra space in bytes: 0 (0 repaired) Total incomplete LBM containers: 0 (0 repaired) Total LBM partial records: 0 (0 repaired) I0703 01:44:05.824805 5151 env_posix.cc:1669] Raising this process' running threads per effective uid limit from 32768 to 65536 I0703 01:44:05.847679 5151 ts_tablet_manager.cc:344] Loading tablet metadata (0/1 complete) I0703 01:44:05.847767 5151 ts_tablet_manager.cc:1005] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa: Loading tablet metadata I0703 01:44:05.870863 5151 ts_tablet_manager.cc:356] Loaded tablet metadata (1 live tablets) I0703 01:44:05.874665 5151 ts_tablet_manager.cc:1173] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa: Registered tablet (data state: TABLET_DATA_READY) I0703 01:44:05.883870 5170 ts_tablet_manager.cc:1028] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa: Bootstrapping tablet I0703 01:44:05.886224 5170 tablet_bootstrap.cc:427] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa: Bootstrap starting. I0703 01:44:05.893256 5151 tablet_server_main.cc:90] Starting tablet server... I0703 01:44:05.988754 5170 log.cc:525] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa: Log is configured to *not* fsync() on all Append() calls I0703 01:44:06.004019 5170 tablet_bootstrap.cc:427] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates I0703 01:44:06.004778 5170 tablet_bootstrap.cc:427] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa: Bootstrap complete. I0703 01:44:06.006237 5170 ts_tablet_manager.cc:1045] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa: Time spent bootstrapping tablet: real 0.120s user 0.011s sys 0.016s I0703 01:44:06.029120 5170 raft_consensus.cc:325] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa [term 1 FOLLOWER]: Replica starting. Triggering 1 pending transactions. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } } peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } I0703 01:44:06.060742 5170 raft_consensus.cc:628] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1157c91cc6f54a68a1d99017ac8444aa, State: Initialized, Role: FOLLOWER I0703 01:44:06.061311 5170 consensus_queue.cc:227] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } } peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } I0703 01:44:06.069955 5170 ts_tablet_manager.cc:1073] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa: Time spent starting tablet: real 0.063s user 0.013s sys 0.010s I0703 01:44:06.154639 5147 heartbeater.cc:467] Master 127.5.240.254:32866 was elected leader, sending a full tablet report... I0703 01:44:06.781559 5151 rpc_server.cc:205] RPC server started. Bound to: 127.5.240.199:55377 I0703 01:44:06.782191 5151 webserver.cc:175] Starting webserver on 127.5.240.199:38435 I0703 01:44:06.782272 5151 webserver.cc:186] Document root disabled I0703 01:44:06.787614 5151 webserver.cc:313] Webserver started. Bound to: http://127.5.240.199:38435/ I0703 01:44:06.788328 5151 server_base.cc:607] Dumped server information to /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/data/info.pb I0703 01:44:06.792942 6083 external_mini_cluster.cc:856] Started /data/somelongdirectorytoavoidrpathissues/src/kudu/build/asan/bin/kudu-tserver as pid 5151 W0703 01:44:06.807819 5283 tablet.cc:1678] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:44:06.808354 5151 tablet_server_main.cc:93] Tablet server successfully started. I0703 01:44:06.823281 5281 heartbeater.cc:346] Connected to a master server at 127.5.240.254:32866 I0703 01:44:06.823457 5281 heartbeater.cc:426] Registering TS with master... I0703 01:44:06.824023 5281 heartbeater.cc:475] Master 127.5.240.254:32866 requested a full tablet report, sending... I0703 01:44:06.825790 3445 ts_manager.cc:104] Re-registered known tserver with Master: 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377) I0703 01:44:06.827301 3445 master_service.cc:249] Signed X509 certificate for tserver {username='jenkins'} at 127.5.240.199:37862 I0703 01:44:07.629552 5301 raft_consensus.cc:457] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout) I0703 01:44:07.630007 5301 raft_consensus.cc:2730] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa: Snoozing failure detection for 2.089s (starting election) I0703 01:44:07.630158 5301 raft_consensus.cc:479] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } } peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } I0703 01:44:07.631789 5301 leader_election.cc:243] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064) I0703 01:44:07.654635 3818 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "dc02a796568c4677a2eebfacae32fc38" candidate_uuid: "1157c91cc6f54a68a1d99017ac8444aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c1e8943f5654f71b030ec7c0b733742" is_pre_election: true I0703 01:44:07.654898 3818 raft_consensus.cc:2730] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742: Snoozing failure detection for 11.319s (vote granted) I0703 01:44:07.655000 3818 raft_consensus.cc:2243] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1157c91cc6f54a68a1d99017ac8444aa in term 1. I0703 01:44:07.656399 5164 leader_election.cc:379] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa [CANDIDATE]: Term 2 pre-election: Vote granted by peer 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064) I0703 01:44:07.656478 5164 leader_election.cc:270] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. I0703 01:44:07.656960 5301 raft_consensus.cc:2730] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa: Snoozing failure detection for 1.557s (election complete) I0703 01:44:07.657032 5301 raft_consensus.cc:2537] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa [term 1 FOLLOWER]: Leader pre-election won for term 2 I0703 01:44:07.657104 5301 raft_consensus.cc:457] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout) I0703 01:44:07.657152 5301 raft_consensus.cc:2730] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa: Snoozing failure detection for 2.055s (starting election) I0703 01:44:07.657213 5301 raft_consensus.cc:2782] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa [term 1 FOLLOWER]: Advancing to term 2 I0703 01:44:07.660676 5301 raft_consensus.cc:479] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } } peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } I0703 01:44:07.661020 5301 leader_election.cc:243] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa [CANDIDATE]: Term 2 election: Requesting vote from peer 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064) I0703 01:44:07.674017 3817 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "dc02a796568c4677a2eebfacae32fc38" candidate_uuid: "1157c91cc6f54a68a1d99017ac8444aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8c1e8943f5654f71b030ec7c0b733742" I0703 01:44:07.674232 3817 raft_consensus.cc:2782] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Advancing to term 2 I0703 01:44:07.674363 3817 raft_consensus.cc:2730] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742: Snoozing failure detection for 10.414s (vote granted) I0703 01:44:07.678737 3817 raft_consensus.cc:2243] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1157c91cc6f54a68a1d99017ac8444aa in term 2. I0703 01:44:07.680135 5164 leader_election.cc:379] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa [CANDIDATE]: Term 2 election: Vote granted by peer 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064) I0703 01:44:07.680213 5164 leader_election.cc:270] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0703 01:44:07.680439 5301 raft_consensus.cc:2730] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa: Snoozing failure detection for 1.976s (election complete) I0703 01:44:07.680541 5301 raft_consensus.cc:2537] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa [term 2 FOLLOWER]: Leader election won for term 2 I0703 01:44:07.682927 5301 raft_consensus.cc:593] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa [term 2 LEADER]: Becoming Leader. State: Replica: 1157c91cc6f54a68a1d99017ac8444aa, State: Running, Role: LEADER I0703 01:44:07.683352 5301 consensus_queue.cc:204] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } } peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } I0703 01:44:07.719625 5281 heartbeater.cc:467] Master 127.5.240.254:32866 was elected leader, sending a full tablet report... I0703 01:44:07.721470 3449 catalog_manager.cc:3798] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "1157c91cc6f54a68a1d99017ac8444aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } health_report { overall_health: UNKNOWN } } } I0703 01:44:08.198004 3818 raft_consensus.cc:1081] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [term 2 FOLLOWER]: Refusing update from remote peer 1157c91cc6f54a68a1d99017ac8444aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch) I0703 01:44:08.198896 5309 consensus_queue.cc:962] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s I0703 01:44:08.558063 3816 consensus_peers.cc:455] T 150033054dcf498d8d71fd996af55e88 P 8c1e8943f5654f71b030ec7c0b733742 -> Peer 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746): Closing peer: 57bea6dc182c41a78304417a3cbfc4a7 I0703 01:44:08.558393 3816 consensus_queue.cc:204] T 150033054dcf498d8d71fd996af55e88 P 8c1e8943f5654f71b030ec7c0b733742 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 199, Committed index: 199, Last appended: 1.199, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 200 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: NON_VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: true } } I0703 01:44:08.569433 3693 raft_consensus.cc:1081] T 150033054dcf498d8d71fd996af55e88 P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 FOLLOWER]: Refusing update from remote peer 8c1e8943f5654f71b030ec7c0b733742: Log matching property violated. Preceding OpId in replica: term: 1 index: 199. Preceding OpId from leader: term: 1 index: 200. (index mismatch) I0703 01:44:08.570194 5154 consensus_queue.cc:962] T 150033054dcf498d8d71fd996af55e88 P 8c1e8943f5654f71b030ec7c0b733742 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 200, Last known committed idx: 199, Time since last communication: 0.000s W0703 01:44:08.583161 3749 consensus_peers.cc:433] T 150033054dcf498d8d71fd996af55e88 P 8c1e8943f5654f71b030ec7c0b733742 -> Peer aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197:51329): Couldn't send request to peer aa8b9c58b466436faa5086e85b5224a7 for tablet 150033054dcf498d8d71fd996af55e88. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 150033054dcf498d8d71fd996af55e88. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:44:08.587816 4744 raft_consensus.cc:2684] T 150033054dcf498d8d71fd996af55e88 P 8c1e8943f5654f71b030ec7c0b733742 [term 1 LEADER]: Committing config change with OpId 1.200: config changed from index -1 to 200, NON_VOTER aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197) added. New config: { opid_index: 200 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: NON_VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: true } } } I0703 01:44:08.589658 3694 raft_consensus.cc:2684] T 150033054dcf498d8d71fd996af55e88 P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 FOLLOWER]: Committing config change with OpId 1.200: config changed from index -1 to 200, NON_VOTER aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197) added. New config: { opid_index: 200 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: NON_VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: true } } } I0703 01:44:08.594983 3449 catalog_manager.cc:3798] T 150033054dcf498d8d71fd996af55e88 P 8c1e8943f5654f71b030ec7c0b733742 reported cstate change: config changed from index -1 to 200, NON_VOTER aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197) added. New cstate: current_term: 1 leader_uuid: "8c1e8943f5654f71b030ec7c0b733742" committed_config { opid_index: 200 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: NON_VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I0703 01:44:08.656746 3693 consensus_peers.cc:455] T eea4e9534e0d4d25a827fae00bbeae4d P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064): Closing peer: 8c1e8943f5654f71b030ec7c0b733742 I0703 01:44:08.657065 3693 consensus_queue.cc:204] T eea4e9534e0d4d25a827fae00bbeae4d P 57bea6dc182c41a78304417a3cbfc4a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 198, Committed index: 198, Last appended: 1.198, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 199 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: NON_VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: true } } I0703 01:44:08.674013 3816 raft_consensus.cc:1081] T eea4e9534e0d4d25a827fae00bbeae4d P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Refusing update from remote peer 57bea6dc182c41a78304417a3cbfc4a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 198. Preceding OpId from leader: term: 1 index: 199. (index mismatch) I0703 01:44:08.678351 4556 consensus_queue.cc:962] T eea4e9534e0d4d25a827fae00bbeae4d P 57bea6dc182c41a78304417a3cbfc4a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 199, Last known committed idx: 198, Time since last communication: 0.000s I0703 01:44:08.685755 4556 raft_consensus.cc:2684] T eea4e9534e0d4d25a827fae00bbeae4d P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 LEADER]: Committing config change with OpId 1.199: config changed from index -1 to 199, NON_VOTER 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198) added. New config: { opid_index: 199 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: NON_VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: true } } } I0703 01:44:08.693832 3449 catalog_manager.cc:3798] T eea4e9534e0d4d25a827fae00bbeae4d P 57bea6dc182c41a78304417a3cbfc4a7 reported cstate change: config changed from index -1 to 199, NON_VOTER 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198) added. New cstate: current_term: 1 leader_uuid: "57bea6dc182c41a78304417a3cbfc4a7" committed_config { opid_index: 199 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: NON_VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I0703 01:44:08.702016 3817 raft_consensus.cc:2684] T eea4e9534e0d4d25a827fae00bbeae4d P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Committing config change with OpId 1.199: config changed from index -1 to 199, NON_VOTER 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198) added. New config: { opid_index: 199 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: NON_VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: true } } } W0703 01:44:08.716482 3630 consensus_peers.cc:433] T eea4e9534e0d4d25a827fae00bbeae4d P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198:51033): Couldn't send request to peer 6932f1cd716a48fe8d6c5573acfc48bf for tablet eea4e9534e0d4d25a827fae00bbeae4d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eea4e9534e0d4d25a827fae00bbeae4d. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:44:08.747139 3693 consensus_peers.cc:455] T 75f9ea8ba93f442aa064d4623d2c36a1 P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193:54919): Closing peer: 55223945fe3d4dd8ac03ba9dab346f54 I0703 01:44:08.748744 3693 consensus_queue.cc:204] T 75f9ea8ba93f442aa064d4623d2c36a1 P 57bea6dc182c41a78304417a3cbfc4a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 197, Committed index: 197, Last appended: 1.197, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 198 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: true } } I0703 01:44:08.755158 3572 raft_consensus.cc:1081] T 75f9ea8ba93f442aa064d4623d2c36a1 P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 FOLLOWER]: Refusing update from remote peer 57bea6dc182c41a78304417a3cbfc4a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 197. Preceding OpId from leader: term: 1 index: 198. (index mismatch) I0703 01:44:08.759671 4556 consensus_queue.cc:962] T 75f9ea8ba93f442aa064d4623d2c36a1 P 57bea6dc182c41a78304417a3cbfc4a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 198, Last known committed idx: 197, Time since last communication: 0.000s I0703 01:44:08.766875 5247 raft_consensus.cc:2684] T 75f9ea8ba93f442aa064d4623d2c36a1 P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 LEADER]: Committing config change with OpId 1.198: config changed from index -1 to 198, NON_VOTER 213596374e4142dca6ecfc701ba1c566 (127.5.240.196) added. New config: { opid_index: 198 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: true } } } I0703 01:44:08.772114 3449 catalog_manager.cc:3798] T 75f9ea8ba93f442aa064d4623d2c36a1 P 57bea6dc182c41a78304417a3cbfc4a7 reported cstate change: config changed from index -1 to 198, NON_VOTER 213596374e4142dca6ecfc701ba1c566 (127.5.240.196) added. New cstate: current_term: 1 leader_uuid: "57bea6dc182c41a78304417a3cbfc4a7" committed_config { opid_index: 198 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } W0703 01:44:08.780187 3630 consensus_peers.cc:433] T 75f9ea8ba93f442aa064d4623d2c36a1 P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer 213596374e4142dca6ecfc701ba1c566 (127.5.240.196:51288): Couldn't send request to peer 213596374e4142dca6ecfc701ba1c566 for tablet 75f9ea8ba93f442aa064d4623d2c36a1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 75f9ea8ba93f442aa064d4623d2c36a1. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:44:08.782949 3570 raft_consensus.cc:2684] T 75f9ea8ba93f442aa064d4623d2c36a1 P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 FOLLOWER]: Committing config change with OpId 1.198: config changed from index -1 to 198, NON_VOTER 213596374e4142dca6ecfc701ba1c566 (127.5.240.196) added. New config: { opid_index: 198 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: true } } } I0703 01:44:08.834004 3815 consensus_peers.cc:455] T 25b33ca4b32a4a2493d03564bfe1bf7e P 8c1e8943f5654f71b030ec7c0b733742 -> Peer 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746): Closing peer: 57bea6dc182c41a78304417a3cbfc4a7 I0703 01:44:08.834281 3815 consensus_queue.cc:204] T 25b33ca4b32a4a2493d03564bfe1bf7e P 8c1e8943f5654f71b030ec7c0b733742 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 220, Committed index: 220, Last appended: 1.220, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: true } } I0703 01:44:08.837736 3693 raft_consensus.cc:1081] T 25b33ca4b32a4a2493d03564bfe1bf7e P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 FOLLOWER]: Refusing update from remote peer 8c1e8943f5654f71b030ec7c0b733742: Log matching property violated. Preceding OpId in replica: term: 1 index: 220. Preceding OpId from leader: term: 1 index: 221. (index mismatch) I0703 01:44:08.838848 5154 consensus_queue.cc:962] T 25b33ca4b32a4a2493d03564bfe1bf7e P 8c1e8943f5654f71b030ec7c0b733742 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 221, Last known committed idx: 220, Time since last communication: 0.000s I0703 01:44:08.845326 4744 raft_consensus.cc:2684] T 25b33ca4b32a4a2493d03564bfe1bf7e P 8c1e8943f5654f71b030ec7c0b733742 [term 1 LEADER]: Committing config change with OpId 1.221: config changed from index -1 to 221, NON_VOTER 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199) added. New config: { opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: true } } } I0703 01:44:08.846330 3694 raft_consensus.cc:2684] T 25b33ca4b32a4a2493d03564bfe1bf7e P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 FOLLOWER]: Committing config change with OpId 1.221: config changed from index -1 to 221, NON_VOTER 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199) added. New config: { opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: true } } } W0703 01:44:08.846971 3749 consensus_peers.cc:433] T 25b33ca4b32a4a2493d03564bfe1bf7e P 8c1e8943f5654f71b030ec7c0b733742 -> Peer 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377): Couldn't send request to peer 1157c91cc6f54a68a1d99017ac8444aa for tablet 25b33ca4b32a4a2493d03564bfe1bf7e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 25b33ca4b32a4a2493d03564bfe1bf7e. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:44:08.851019 3449 catalog_manager.cc:3798] T 25b33ca4b32a4a2493d03564bfe1bf7e P 8c1e8943f5654f71b030ec7c0b733742 reported cstate change: config changed from index -1 to 221, NON_VOTER 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199) added. New cstate: current_term: 1 leader_uuid: "8c1e8943f5654f71b030ec7c0b733742" committed_config { opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I0703 01:44:08.892931 3816 consensus_peers.cc:455] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 -> Peer 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193:54919): Closing peer: 55223945fe3d4dd8ac03ba9dab346f54 I0703 01:44:08.893144 3816 consensus_queue.cc:204] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 237, Committed index: 237, Last appended: 1.237, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: NON_VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: true } } I0703 01:44:08.897308 3572 raft_consensus.cc:1081] T c352f70a51154a85a5a5b0f3d14d405f P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 FOLLOWER]: Refusing update from remote peer 8c1e8943f5654f71b030ec7c0b733742: Log matching property violated. Preceding OpId in replica: term: 1 index: 237. Preceding OpId from leader: term: 1 index: 238. (index mismatch) W0703 01:44:08.899682 3749 consensus_peers.cc:433] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 -> Peer aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197:51329): Couldn't send request to peer aa8b9c58b466436faa5086e85b5224a7 for tablet c352f70a51154a85a5a5b0f3d14d405f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c352f70a51154a85a5a5b0f3d14d405f. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:44:08.901325 5154 consensus_queue.cc:962] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 238, Last known committed idx: 237, Time since last communication: 0.002s I0703 01:44:08.904037 4744 raft_consensus.cc:2684] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 [term 1 LEADER]: Committing config change with OpId 1.238: config changed from index -1 to 238, NON_VOTER aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197) added. New config: { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: NON_VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: true } } } I0703 01:44:08.906347 3570 raft_consensus.cc:2684] T c352f70a51154a85a5a5b0f3d14d405f P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 FOLLOWER]: Committing config change with OpId 1.238: config changed from index -1 to 238, NON_VOTER aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197) added. New config: { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: NON_VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: true } } } I0703 01:44:08.910557 3449 catalog_manager.cc:3798] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 reported cstate change: config changed from index -1 to 238, NON_VOTER aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197) added. New cstate: current_term: 1 leader_uuid: "8c1e8943f5654f71b030ec7c0b733742" committed_config { opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: NON_VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I0703 01:44:08.978325 3696 consensus_peers.cc:455] T 4a613c1be55c4888a4df36d3f1bfc54c P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064): Closing peer: 8c1e8943f5654f71b030ec7c0b733742 I0703 01:44:08.978935 3696 consensus_queue.cc:204] T 4a613c1be55c4888a4df36d3f1bfc54c P 57bea6dc182c41a78304417a3cbfc4a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 221, Committed index: 221, Last appended: 1.222, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: NON_VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: true } } I0703 01:44:08.988859 3816 raft_consensus.cc:1081] T 4a613c1be55c4888a4df36d3f1bfc54c P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Refusing update from remote peer 57bea6dc182c41a78304417a3cbfc4a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 222. Preceding OpId from leader: term: 1 index: 223. (index mismatch) W0703 01:44:08.988591 3630 consensus_peers.cc:433] T 4a613c1be55c4888a4df36d3f1bfc54c P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198:51033): Couldn't send request to peer 6932f1cd716a48fe8d6c5573acfc48bf for tablet 4a613c1be55c4888a4df36d3f1bfc54c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4a613c1be55c4888a4df36d3f1bfc54c. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:44:08.996803 5400 consensus_queue.cc:962] T 4a613c1be55c4888a4df36d3f1bfc54c P 57bea6dc182c41a78304417a3cbfc4a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 223, Last known committed idx: 221, Time since last communication: 0.001s I0703 01:44:09.013763 4556 raft_consensus.cc:2684] T 4a613c1be55c4888a4df36d3f1bfc54c P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 LEADER]: Committing config change with OpId 1.223: config changed from index -1 to 223, NON_VOTER 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198) added. New config: { opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: NON_VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: true } } } I0703 01:44:09.019825 3445 catalog_manager.cc:3798] T 4a613c1be55c4888a4df36d3f1bfc54c P 57bea6dc182c41a78304417a3cbfc4a7 reported cstate change: config changed from index -1 to 223, NON_VOTER 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198) added. New cstate: current_term: 1 leader_uuid: "57bea6dc182c41a78304417a3cbfc4a7" committed_config { opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: NON_VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I0703 01:44:09.027341 3816 raft_consensus.cc:2684] T 4a613c1be55c4888a4df36d3f1bfc54c P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Committing config change with OpId 1.223: config changed from index -1 to 223, NON_VOTER 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198) added. New config: { opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: NON_VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: true } } } I0703 01:44:09.061692 5235 consensus_peers.cc:455] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa -> Peer 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064): Closing peer: 8c1e8943f5654f71b030ec7c0b733742 I0703 01:44:09.062048 5235 consensus_queue.cc:204] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { replace: true } } peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: true } } I0703 01:44:09.067291 3815 raft_consensus.cc:1081] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [term 2 FOLLOWER]: Refusing update from remote peer 1157c91cc6f54a68a1d99017ac8444aa: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch) I0703 01:44:09.068085 5351 consensus_queue.cc:962] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s I0703 01:44:09.070983 5351 raft_consensus.cc:2684] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 213596374e4142dca6ecfc701ba1c566 (127.5.240.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { replace: true } } peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: true } } } I0703 01:44:09.072329 5412 ts_tablet_manager.cc:669] T eea4e9534e0d4d25a827fae00bbeae4d P 6932f1cd716a48fe8d6c5573acfc48bf: Initiating tablet copy from peer 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746) I0703 01:44:09.072706 5412 tablet_copy_client.cc:239] T eea4e9534e0d4d25a827fae00bbeae4d P 6932f1cd716a48fe8d6c5573acfc48bf: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.194:59746 I0703 01:44:09.075947 3445 catalog_manager.cc:3798] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa reported cstate change: config changed from index -1 to 3, NON_VOTER 213596374e4142dca6ecfc701ba1c566 (127.5.240.196) added. New cstate: current_term: 2 leader_uuid: "1157c91cc6f54a68a1d99017ac8444aa" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I0703 01:44:09.081190 3815 raft_consensus.cc:2684] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 213596374e4142dca6ecfc701ba1c566 (127.5.240.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { replace: true } } peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: true } } } W0703 01:44:09.099421 5164 consensus_peers.cc:433] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa -> Peer 213596374e4142dca6ecfc701ba1c566 (127.5.240.196:51288): Couldn't send request to peer 213596374e4142dca6ecfc701ba1c566 for tablet dc02a796568c4677a2eebfacae32fc38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dc02a796568c4677a2eebfacae32fc38. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:44:09.103042 5416 ts_tablet_manager.cc:669] T 150033054dcf498d8d71fd996af55e88 P aa8b9c58b466436faa5086e85b5224a7: Initiating tablet copy from peer 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064) I0703 01:44:09.103356 5416 tablet_copy_client.cc:239] T 150033054dcf498d8d71fd996af55e88 P aa8b9c58b466436faa5086e85b5224a7: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.195:51064 I0703 01:44:09.135758 3716 tablet_copy_service.cc:135] P 57bea6dc182c41a78304417a3cbfc4a7: Received BeginTabletCopySession request for tablet eea4e9534e0d4d25a827fae00bbeae4d from peer 6932f1cd716a48fe8d6c5573acfc48bf ({username='jenkins'} at 127.5.240.198:49512) I0703 01:44:09.136097 3716 tablet_copy_service.cc:156] P 57bea6dc182c41a78304417a3cbfc4a7: Beginning new tablet copy session on tablet eea4e9534e0d4d25a827fae00bbeae4d from peer 6932f1cd716a48fe8d6c5573acfc48bf at {username='jenkins'} at 127.5.240.198:49512: session id = 6932f1cd716a48fe8d6c5573acfc48bf-eea4e9534e0d4d25a827fae00bbeae4d I0703 01:44:09.139415 3716 tablet_copy_source_session.cc:202] T eea4e9534e0d4d25a827fae00bbeae4d P 57bea6dc182c41a78304417a3cbfc4a7: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:44:09.143450 5412 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet eea4e9534e0d4d25a827fae00bbeae4d. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:44:09.144219 3838 tablet_copy_service.cc:135] P 8c1e8943f5654f71b030ec7c0b733742: Received BeginTabletCopySession request for tablet 150033054dcf498d8d71fd996af55e88 from peer aa8b9c58b466436faa5086e85b5224a7 ({username='jenkins'} at 127.5.240.197:45440) I0703 01:44:09.144527 3838 tablet_copy_service.cc:156] P 8c1e8943f5654f71b030ec7c0b733742: Beginning new tablet copy session on tablet 150033054dcf498d8d71fd996af55e88 from peer aa8b9c58b466436faa5086e85b5224a7 at {username='jenkins'} at 127.5.240.197:45440: session id = aa8b9c58b466436faa5086e85b5224a7-150033054dcf498d8d71fd996af55e88 I0703 01:44:09.147370 3838 tablet_copy_source_session.cc:202] T 150033054dcf498d8d71fd996af55e88 P 8c1e8943f5654f71b030ec7c0b733742: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:44:09.153234 5416 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 150033054dcf498d8d71fd996af55e88. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:44:09.164422 5412 ts_tablet_manager.cc:1173] T eea4e9534e0d4d25a827fae00bbeae4d P 6932f1cd716a48fe8d6c5573acfc48bf: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:44:09.173866 5412 tablet_copy_client.cc:566] T eea4e9534e0d4d25a827fae00bbeae4d P 6932f1cd716a48fe8d6c5573acfc48bf: tablet copy: Starting download of 0 data blocks... I0703 01:44:09.174060 5412 tablet_copy_client.cc:529] T eea4e9534e0d4d25a827fae00bbeae4d P 6932f1cd716a48fe8d6c5573acfc48bf: tablet copy: Starting download of 1 WAL segments... I0703 01:44:09.180636 5412 tablet_copy_client.cc:405] T eea4e9534e0d4d25a827fae00bbeae4d P 6932f1cd716a48fe8d6c5573acfc48bf: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:44:09.186386 3573 consensus_peers.cc:455] T 6c8253ac9a9a405885f1f57c59a84fd5 P 55223945fe3d4dd8ac03ba9dab346f54 -> Peer 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746): Closing peer: 57bea6dc182c41a78304417a3cbfc4a7 I0703 01:44:09.186763 3573 consensus_queue.cc:204] T 6c8253ac9a9a405885f1f57c59a84fd5 P 55223945fe3d4dd8ac03ba9dab346f54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 231, Committed index: 231, Last appended: 1.231, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: NON_VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: true } } I0703 01:44:09.187934 5412 ts_tablet_manager.cc:1028] T eea4e9534e0d4d25a827fae00bbeae4d P 6932f1cd716a48fe8d6c5573acfc48bf: Bootstrapping tablet I0703 01:44:09.188886 5412 tablet_bootstrap.cc:427] T eea4e9534e0d4d25a827fae00bbeae4d P 6932f1cd716a48fe8d6c5573acfc48bf: Bootstrap starting. I0703 01:44:09.195292 3694 raft_consensus.cc:1081] T 6c8253ac9a9a405885f1f57c59a84fd5 P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 FOLLOWER]: Refusing update from remote peer 55223945fe3d4dd8ac03ba9dab346f54: Log matching property violated. Preceding OpId in replica: term: 1 index: 231. Preceding OpId from leader: term: 1 index: 232. (index mismatch) I0703 01:44:09.199553 4618 consensus_queue.cc:962] T 6c8253ac9a9a405885f1f57c59a84fd5 P 55223945fe3d4dd8ac03ba9dab346f54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 232, Last known committed idx: 231, Time since last communication: 0.001s I0703 01:44:09.207042 5416 ts_tablet_manager.cc:1173] T 150033054dcf498d8d71fd996af55e88 P aa8b9c58b466436faa5086e85b5224a7: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:44:09.212568 5416 tablet_copy_client.cc:566] T 150033054dcf498d8d71fd996af55e88 P aa8b9c58b466436faa5086e85b5224a7: tablet copy: Starting download of 0 data blocks... I0703 01:44:09.212131 4750 raft_consensus.cc:2684] T 6c8253ac9a9a405885f1f57c59a84fd5 P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 LEADER]: Committing config change with OpId 1.232: config changed from index -1 to 232, NON_VOTER aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197) added. New config: { opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: NON_VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: true } } } I0703 01:44:09.212834 5416 tablet_copy_client.cc:529] T 150033054dcf498d8d71fd996af55e88 P aa8b9c58b466436faa5086e85b5224a7: tablet copy: Starting download of 1 WAL segments... I0703 01:44:09.216886 3445 catalog_manager.cc:3798] T 6c8253ac9a9a405885f1f57c59a84fd5 P 55223945fe3d4dd8ac03ba9dab346f54 reported cstate change: config changed from index -1 to 232, NON_VOTER aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197) added. New cstate: current_term: 1 leader_uuid: "55223945fe3d4dd8ac03ba9dab346f54" committed_config { opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: NON_VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I0703 01:44:09.223290 3696 raft_consensus.cc:2684] T 6c8253ac9a9a405885f1f57c59a84fd5 P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 FOLLOWER]: Committing config change with OpId 1.232: config changed from index -1 to 232, NON_VOTER aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197) added. New config: { opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: NON_VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: true } } } I0703 01:44:09.230749 5416 tablet_copy_client.cc:405] T 150033054dcf498d8d71fd996af55e88 P aa8b9c58b466436faa5086e85b5224a7: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:44:09.248721 5416 ts_tablet_manager.cc:1028] T 150033054dcf498d8d71fd996af55e88 P aa8b9c58b466436faa5086e85b5224a7: Bootstrapping tablet I0703 01:44:09.255218 5429 ts_tablet_manager.cc:669] T 25b33ca4b32a4a2493d03564bfe1bf7e P 1157c91cc6f54a68a1d99017ac8444aa: Initiating tablet copy from peer 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064) I0703 01:44:09.255548 5429 tablet_copy_client.cc:239] T 25b33ca4b32a4a2493d03564bfe1bf7e P 1157c91cc6f54a68a1d99017ac8444aa: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.195:51064 I0703 01:44:09.257426 5416 tablet_bootstrap.cc:427] T 150033054dcf498d8d71fd996af55e88 P aa8b9c58b466436faa5086e85b5224a7: Bootstrap starting. W0703 01:44:09.259807 3504 consensus_peers.cc:433] T 6c8253ac9a9a405885f1f57c59a84fd5 P 55223945fe3d4dd8ac03ba9dab346f54 -> Peer aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197:51329): Couldn't send request to peer aa8b9c58b466436faa5086e85b5224a7 for tablet 6c8253ac9a9a405885f1f57c59a84fd5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6c8253ac9a9a405885f1f57c59a84fd5. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:44:09.271716 3838 tablet_copy_service.cc:135] P 8c1e8943f5654f71b030ec7c0b733742: Received BeginTabletCopySession request for tablet 25b33ca4b32a4a2493d03564bfe1bf7e from peer 1157c91cc6f54a68a1d99017ac8444aa ({username='jenkins'} at 127.5.240.199:55739) I0703 01:44:09.271922 3838 tablet_copy_service.cc:156] P 8c1e8943f5654f71b030ec7c0b733742: Beginning new tablet copy session on tablet 25b33ca4b32a4a2493d03564bfe1bf7e from peer 1157c91cc6f54a68a1d99017ac8444aa at {username='jenkins'} at 127.5.240.199:55739: session id = 1157c91cc6f54a68a1d99017ac8444aa-25b33ca4b32a4a2493d03564bfe1bf7e I0703 01:44:09.275053 3838 tablet_copy_source_session.cc:202] T 25b33ca4b32a4a2493d03564bfe1bf7e P 8c1e8943f5654f71b030ec7c0b733742: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:44:09.281904 5429 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 25b33ca4b32a4a2493d03564bfe1bf7e. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:44:09.292479 5433 ts_tablet_manager.cc:669] T 75f9ea8ba93f442aa064d4623d2c36a1 P 213596374e4142dca6ecfc701ba1c566: Initiating tablet copy from peer 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746) I0703 01:44:09.292832 5433 tablet_copy_client.cc:239] T 75f9ea8ba93f442aa064d4623d2c36a1 P 213596374e4142dca6ecfc701ba1c566: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.194:59746 I0703 01:44:09.298275 5429 ts_tablet_manager.cc:1173] T 25b33ca4b32a4a2493d03564bfe1bf7e P 1157c91cc6f54a68a1d99017ac8444aa: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:44:09.301852 5429 tablet_copy_client.cc:566] T 25b33ca4b32a4a2493d03564bfe1bf7e P 1157c91cc6f54a68a1d99017ac8444aa: tablet copy: Starting download of 0 data blocks... I0703 01:44:09.302386 5429 tablet_copy_client.cc:529] T 25b33ca4b32a4a2493d03564bfe1bf7e P 1157c91cc6f54a68a1d99017ac8444aa: tablet copy: Starting download of 1 WAL segments... I0703 01:44:09.312479 5429 tablet_copy_client.cc:405] T 25b33ca4b32a4a2493d03564bfe1bf7e P 1157c91cc6f54a68a1d99017ac8444aa: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:44:09.318430 3816 consensus_peers.cc:455] T 674ed5a88db04825a4eaec2fa996a852 P 8c1e8943f5654f71b030ec7c0b733742 -> Peer 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193:54919): Closing peer: 55223945fe3d4dd8ac03ba9dab346f54 I0703 01:44:09.318734 5429 ts_tablet_manager.cc:1028] T 25b33ca4b32a4a2493d03564bfe1bf7e P 1157c91cc6f54a68a1d99017ac8444aa: Bootstrapping tablet I0703 01:44:09.318742 3816 consensus_queue.cc:204] T 674ed5a88db04825a4eaec2fa996a852 P 8c1e8943f5654f71b030ec7c0b733742 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 279, Committed index: 279, Last appended: 1.279, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: true } } I0703 01:44:09.318919 5429 tablet_bootstrap.cc:427] T 25b33ca4b32a4a2493d03564bfe1bf7e P 1157c91cc6f54a68a1d99017ac8444aa: Bootstrap starting. I0703 01:44:09.322175 3572 raft_consensus.cc:1081] T 674ed5a88db04825a4eaec2fa996a852 P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 FOLLOWER]: Refusing update from remote peer 8c1e8943f5654f71b030ec7c0b733742: Log matching property violated. Preceding OpId in replica: term: 1 index: 279. Preceding OpId from leader: term: 1 index: 280. (index mismatch) I0703 01:44:09.323523 4744 consensus_queue.cc:962] T 674ed5a88db04825a4eaec2fa996a852 P 8c1e8943f5654f71b030ec7c0b733742 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 280, Last known committed idx: 279, Time since last communication: 0.000s W0703 01:44:09.327065 3749 consensus_peers.cc:433] T 674ed5a88db04825a4eaec2fa996a852 P 8c1e8943f5654f71b030ec7c0b733742 -> Peer 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377): Couldn't send request to peer 1157c91cc6f54a68a1d99017ac8444aa for tablet 674ed5a88db04825a4eaec2fa996a852. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 674ed5a88db04825a4eaec2fa996a852. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:44:09.327527 3716 tablet_copy_service.cc:135] P 57bea6dc182c41a78304417a3cbfc4a7: Received BeginTabletCopySession request for tablet 75f9ea8ba93f442aa064d4623d2c36a1 from peer 213596374e4142dca6ecfc701ba1c566 ({username='jenkins'} at 127.5.240.196:36312) I0703 01:44:09.327672 3716 tablet_copy_service.cc:156] P 57bea6dc182c41a78304417a3cbfc4a7: Beginning new tablet copy session on tablet 75f9ea8ba93f442aa064d4623d2c36a1 from peer 213596374e4142dca6ecfc701ba1c566 at {username='jenkins'} at 127.5.240.196:36312: session id = 213596374e4142dca6ecfc701ba1c566-75f9ea8ba93f442aa064d4623d2c36a1 I0703 01:44:09.330070 3716 tablet_copy_source_session.cc:202] T 75f9ea8ba93f442aa064d4623d2c36a1 P 57bea6dc182c41a78304417a3cbfc4a7: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:44:09.329610 4744 raft_consensus.cc:2684] T 674ed5a88db04825a4eaec2fa996a852 P 8c1e8943f5654f71b030ec7c0b733742 [term 1 LEADER]: Committing config change with OpId 1.280: config changed from index -1 to 280, NON_VOTER 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199) added. New config: { opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: true } } } I0703 01:44:09.330480 3573 raft_consensus.cc:2684] T 674ed5a88db04825a4eaec2fa996a852 P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 FOLLOWER]: Committing config change with OpId 1.280: config changed from index -1 to 280, NON_VOTER 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199) added. New config: { opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: true } } } I0703 01:44:09.335517 3445 catalog_manager.cc:3798] T 674ed5a88db04825a4eaec2fa996a852 P 55223945fe3d4dd8ac03ba9dab346f54 reported cstate change: config changed from index -1 to 280, NON_VOTER 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199) added. New cstate: current_term: 1 leader_uuid: "8c1e8943f5654f71b030ec7c0b733742" committed_config { opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: true } } } I0703 01:44:09.343118 5433 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 75f9ea8ba93f442aa064d4623d2c36a1. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:44:09.375104 5433 ts_tablet_manager.cc:1173] T 75f9ea8ba93f442aa064d4623d2c36a1 P 213596374e4142dca6ecfc701ba1c566: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:44:09.376610 5433 tablet_copy_client.cc:566] T 75f9ea8ba93f442aa064d4623d2c36a1 P 213596374e4142dca6ecfc701ba1c566: tablet copy: Starting download of 0 data blocks... I0703 01:44:09.377743 5433 tablet_copy_client.cc:529] T 75f9ea8ba93f442aa064d4623d2c36a1 P 213596374e4142dca6ecfc701ba1c566: tablet copy: Starting download of 1 WAL segments... I0703 01:44:09.391518 5412 log.cc:525] T eea4e9534e0d4d25a827fae00bbeae4d P 6932f1cd716a48fe8d6c5573acfc48bf: Log is configured to *not* fsync() on all Append() calls I0703 01:44:09.421953 5433 tablet_copy_client.cc:405] T 75f9ea8ba93f442aa064d4623d2c36a1 P 213596374e4142dca6ecfc701ba1c566: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:44:09.424890 5433 ts_tablet_manager.cc:1028] T 75f9ea8ba93f442aa064d4623d2c36a1 P 213596374e4142dca6ecfc701ba1c566: Bootstrapping tablet I0703 01:44:09.425191 5433 tablet_bootstrap.cc:427] T 75f9ea8ba93f442aa064d4623d2c36a1 P 213596374e4142dca6ecfc701ba1c566: Bootstrap starting. I0703 01:44:09.501986 3694 consensus_peers.cc:455] T 930bbe4ac3234ed49a2207c4902541f1 P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064): Closing peer: 8c1e8943f5654f71b030ec7c0b733742 I0703 01:44:09.502259 3694 consensus_queue.cc:204] T 930bbe4ac3234ed49a2207c4902541f1 P 57bea6dc182c41a78304417a3cbfc4a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 267, Committed index: 267, Last appended: 1.267, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: true } } I0703 01:44:09.502568 5444 ts_tablet_manager.cc:669] T c352f70a51154a85a5a5b0f3d14d405f P aa8b9c58b466436faa5086e85b5224a7: Initiating tablet copy from peer 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064) I0703 01:44:09.502729 5444 tablet_copy_client.cc:239] T c352f70a51154a85a5a5b0f3d14d405f P aa8b9c58b466436faa5086e85b5224a7: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.195:51064 I0703 01:44:09.504127 3838 tablet_copy_service.cc:135] P 8c1e8943f5654f71b030ec7c0b733742: Received BeginTabletCopySession request for tablet c352f70a51154a85a5a5b0f3d14d405f from peer aa8b9c58b466436faa5086e85b5224a7 ({username='jenkins'} at 127.5.240.197:45440) I0703 01:44:09.504262 3838 tablet_copy_service.cc:156] P 8c1e8943f5654f71b030ec7c0b733742: Beginning new tablet copy session on tablet c352f70a51154a85a5a5b0f3d14d405f from peer aa8b9c58b466436faa5086e85b5224a7 at {username='jenkins'} at 127.5.240.197:45440: session id = aa8b9c58b466436faa5086e85b5224a7-c352f70a51154a85a5a5b0f3d14d405f I0703 01:44:09.510821 3838 tablet_copy_source_session.cc:202] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:44:09.511741 3815 raft_consensus.cc:1081] T 930bbe4ac3234ed49a2207c4902541f1 P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Refusing update from remote peer 57bea6dc182c41a78304417a3cbfc4a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 267. Preceding OpId from leader: term: 1 index: 268. (index mismatch) I0703 01:44:09.522320 5247 consensus_queue.cc:962] T 930bbe4ac3234ed49a2207c4902541f1 P 57bea6dc182c41a78304417a3cbfc4a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 268, Last known committed idx: 267, Time since last communication: 0.000s W0703 01:44:09.525566 3630 consensus_peers.cc:433] T 930bbe4ac3234ed49a2207c4902541f1 P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer 213596374e4142dca6ecfc701ba1c566 (127.5.240.196:51288): Couldn't send request to peer 213596374e4142dca6ecfc701ba1c566 for tablet 930bbe4ac3234ed49a2207c4902541f1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 930bbe4ac3234ed49a2207c4902541f1. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:44:09.527966 5444 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet c352f70a51154a85a5a5b0f3d14d405f. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:44:09.537252 5105 raft_consensus.cc:2684] T 930bbe4ac3234ed49a2207c4902541f1 P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 LEADER]: Committing config change with OpId 1.268: config changed from index -1 to 268, NON_VOTER 213596374e4142dca6ecfc701ba1c566 (127.5.240.196) added. New config: { opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: true } } } I0703 01:44:09.538898 5444 ts_tablet_manager.cc:1173] T c352f70a51154a85a5a5b0f3d14d405f P aa8b9c58b466436faa5086e85b5224a7: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:44:09.539355 5444 tablet_copy_client.cc:566] T c352f70a51154a85a5a5b0f3d14d405f P aa8b9c58b466436faa5086e85b5224a7: tablet copy: Starting download of 0 data blocks... I0703 01:44:09.539572 5444 tablet_copy_client.cc:529] T c352f70a51154a85a5a5b0f3d14d405f P aa8b9c58b466436faa5086e85b5224a7: tablet copy: Starting download of 1 WAL segments... I0703 01:44:09.547967 3445 catalog_manager.cc:3798] T 930bbe4ac3234ed49a2207c4902541f1 P 57bea6dc182c41a78304417a3cbfc4a7 reported cstate change: config changed from index -1 to 268, NON_VOTER 213596374e4142dca6ecfc701ba1c566 (127.5.240.196) added. New cstate: current_term: 1 leader_uuid: "57bea6dc182c41a78304417a3cbfc4a7" committed_config { opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I0703 01:44:09.553349 3815 raft_consensus.cc:2684] T 930bbe4ac3234ed49a2207c4902541f1 P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Committing config change with OpId 1.268: config changed from index -1 to 268, NON_VOTER 213596374e4142dca6ecfc701ba1c566 (127.5.240.196) added. New config: { opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: true } } } I0703 01:44:09.571776 5444 tablet_copy_client.cc:405] T c352f70a51154a85a5a5b0f3d14d405f P aa8b9c58b466436faa5086e85b5224a7: tablet copy: Tablet Copy complete. Replacing tablet superblock. W0703 01:44:09.574439 3630 consensus_peers.cc:433] T eea4e9534e0d4d25a827fae00bbeae4d P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198:51033): Couldn't send request to peer 6932f1cd716a48fe8d6c5573acfc48bf for tablet eea4e9534e0d4d25a827fae00bbeae4d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 2 times. I0703 01:44:09.579335 5444 ts_tablet_manager.cc:1028] T c352f70a51154a85a5a5b0f3d14d405f P aa8b9c58b466436faa5086e85b5224a7: Bootstrapping tablet I0703 01:44:09.582602 5444 tablet_bootstrap.cc:427] T c352f70a51154a85a5a5b0f3d14d405f P aa8b9c58b466436faa5086e85b5224a7: Bootstrap starting. W0703 01:44:09.602316 3749 consensus_peers.cc:433] T 150033054dcf498d8d71fd996af55e88 P 8c1e8943f5654f71b030ec7c0b733742 -> Peer aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197:51329): Couldn't send request to peer aa8b9c58b466436faa5086e85b5224a7 for tablet 150033054dcf498d8d71fd996af55e88. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 2 times. I0703 01:44:09.608441 5447 ts_tablet_manager.cc:669] T 4a613c1be55c4888a4df36d3f1bfc54c P 6932f1cd716a48fe8d6c5573acfc48bf: Initiating tablet copy from peer 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746) I0703 01:44:09.608711 5447 tablet_copy_client.cc:239] T 4a613c1be55c4888a4df36d3f1bfc54c P 6932f1cd716a48fe8d6c5573acfc48bf: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.194:59746 I0703 01:44:09.609577 3716 tablet_copy_service.cc:135] P 57bea6dc182c41a78304417a3cbfc4a7: Received BeginTabletCopySession request for tablet 4a613c1be55c4888a4df36d3f1bfc54c from peer 6932f1cd716a48fe8d6c5573acfc48bf ({username='jenkins'} at 127.5.240.198:49512) I0703 01:44:09.609704 3716 tablet_copy_service.cc:156] P 57bea6dc182c41a78304417a3cbfc4a7: Beginning new tablet copy session on tablet 4a613c1be55c4888a4df36d3f1bfc54c from peer 6932f1cd716a48fe8d6c5573acfc48bf at {username='jenkins'} at 127.5.240.198:49512: session id = 6932f1cd716a48fe8d6c5573acfc48bf-4a613c1be55c4888a4df36d3f1bfc54c I0703 01:44:09.612222 3716 tablet_copy_source_session.cc:202] T 4a613c1be55c4888a4df36d3f1bfc54c P 57bea6dc182c41a78304417a3cbfc4a7: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:44:09.615319 5416 log.cc:525] T 150033054dcf498d8d71fd996af55e88 P aa8b9c58b466436faa5086e85b5224a7: Log is configured to *not* fsync() on all Append() calls I0703 01:44:09.615628 5447 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 4a613c1be55c4888a4df36d3f1bfc54c. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:44:09.621179 5452 ts_tablet_manager.cc:669] T dc02a796568c4677a2eebfacae32fc38 P 213596374e4142dca6ecfc701ba1c566: Initiating tablet copy from peer 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377) I0703 01:44:09.630463 5452 tablet_copy_client.cc:239] T dc02a796568c4677a2eebfacae32fc38 P 213596374e4142dca6ecfc701ba1c566: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.199:55377 I0703 01:44:09.643581 5447 ts_tablet_manager.cc:1173] T 4a613c1be55c4888a4df36d3f1bfc54c P 6932f1cd716a48fe8d6c5573acfc48bf: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:44:09.645481 5433 log.cc:525] T 75f9ea8ba93f442aa064d4623d2c36a1 P 213596374e4142dca6ecfc701ba1c566: Log is configured to *not* fsync() on all Append() calls I0703 01:44:09.648429 5447 tablet_copy_client.cc:566] T 4a613c1be55c4888a4df36d3f1bfc54c P 6932f1cd716a48fe8d6c5573acfc48bf: tablet copy: Starting download of 0 data blocks... I0703 01:44:09.648612 5447 tablet_copy_client.cc:529] T 4a613c1be55c4888a4df36d3f1bfc54c P 6932f1cd716a48fe8d6c5573acfc48bf: tablet copy: Starting download of 1 WAL segments... I0703 01:44:09.651370 3572 consensus_peers.cc:455] T eada0e8b8608424594cabd35cf9891e3 P 55223945fe3d4dd8ac03ba9dab346f54 -> Peer 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746): Closing peer: 57bea6dc182c41a78304417a3cbfc4a7 I0703 01:44:09.651579 3572 consensus_queue.cc:204] T eada0e8b8608424594cabd35cf9891e3 P 55223945fe3d4dd8ac03ba9dab346f54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 237, Committed index: 237, Last appended: 1.238, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: NON_VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: true } } I0703 01:44:09.657441 3694 raft_consensus.cc:1081] T eada0e8b8608424594cabd35cf9891e3 P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 FOLLOWER]: Refusing update from remote peer 55223945fe3d4dd8ac03ba9dab346f54: Log matching property violated. Preceding OpId in replica: term: 1 index: 238. Preceding OpId from leader: term: 1 index: 239. (index mismatch) I0703 01:44:09.664204 5447 tablet_copy_client.cc:405] T 4a613c1be55c4888a4df36d3f1bfc54c P 6932f1cd716a48fe8d6c5573acfc48bf: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:44:09.671391 5447 ts_tablet_manager.cc:1028] T 4a613c1be55c4888a4df36d3f1bfc54c P 6932f1cd716a48fe8d6c5573acfc48bf: Bootstrapping tablet I0703 01:44:09.683177 5447 tablet_bootstrap.cc:427] T 4a613c1be55c4888a4df36d3f1bfc54c P 6932f1cd716a48fe8d6c5573acfc48bf: Bootstrap starting. I0703 01:44:09.683609 5258 tablet_copy_service.cc:135] P 1157c91cc6f54a68a1d99017ac8444aa: Received BeginTabletCopySession request for tablet dc02a796568c4677a2eebfacae32fc38 from peer 213596374e4142dca6ecfc701ba1c566 ({username='jenkins'} at 127.5.240.196:60844) I0703 01:44:09.683884 5258 tablet_copy_service.cc:156] P 1157c91cc6f54a68a1d99017ac8444aa: Beginning new tablet copy session on tablet dc02a796568c4677a2eebfacae32fc38 from peer 213596374e4142dca6ecfc701ba1c566 at {username='jenkins'} at 127.5.240.196:60844: session id = 213596374e4142dca6ecfc701ba1c566-dc02a796568c4677a2eebfacae32fc38 I0703 01:44:09.687153 5258 tablet_copy_source_session.cc:202] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:44:09.688189 5133 consensus_queue.cc:962] T eada0e8b8608424594cabd35cf9891e3 P 55223945fe3d4dd8ac03ba9dab346f54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 239, Last known committed idx: 237, Time since last communication: 0.000s I0703 01:44:09.688558 5452 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet dc02a796568c4677a2eebfacae32fc38. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:44:09.693974 5452 ts_tablet_manager.cc:1173] T dc02a796568c4677a2eebfacae32fc38 P 213596374e4142dca6ecfc701ba1c566: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:44:09.694350 5452 tablet_copy_client.cc:566] T dc02a796568c4677a2eebfacae32fc38 P 213596374e4142dca6ecfc701ba1c566: tablet copy: Starting download of 0 data blocks... I0703 01:44:09.694494 5452 tablet_copy_client.cc:529] T dc02a796568c4677a2eebfacae32fc38 P 213596374e4142dca6ecfc701ba1c566: tablet copy: Starting download of 1 WAL segments... I0703 01:44:09.696223 5452 tablet_copy_client.cc:405] T dc02a796568c4677a2eebfacae32fc38 P 213596374e4142dca6ecfc701ba1c566: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:44:09.698408 5452 ts_tablet_manager.cc:1028] T dc02a796568c4677a2eebfacae32fc38 P 213596374e4142dca6ecfc701ba1c566: Bootstrapping tablet I0703 01:44:09.698576 5452 tablet_bootstrap.cc:427] T dc02a796568c4677a2eebfacae32fc38 P 213596374e4142dca6ecfc701ba1c566: Bootstrap starting. I0703 01:44:09.698931 5442 raft_consensus.cc:2684] T eada0e8b8608424594cabd35cf9891e3 P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 LEADER]: Committing config change with OpId 1.239: config changed from index -1 to 239, NON_VOTER 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198) added. New config: { opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: NON_VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: true } } } W0703 01:44:09.708675 3507 consensus_peers.cc:433] T eada0e8b8608424594cabd35cf9891e3 P 55223945fe3d4dd8ac03ba9dab346f54 -> Peer 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198:51033): Couldn't send request to peer 6932f1cd716a48fe8d6c5573acfc48bf for tablet eada0e8b8608424594cabd35cf9891e3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eada0e8b8608424594cabd35cf9891e3. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:44:09.709854 3445 catalog_manager.cc:3798] T eada0e8b8608424594cabd35cf9891e3 P 55223945fe3d4dd8ac03ba9dab346f54 reported cstate change: config changed from index -1 to 239, NON_VOTER 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198) added. New cstate: current_term: 1 leader_uuid: "55223945fe3d4dd8ac03ba9dab346f54" committed_config { opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: NON_VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I0703 01:44:09.711761 3694 raft_consensus.cc:2684] T eada0e8b8608424594cabd35cf9891e3 P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 FOLLOWER]: Committing config change with OpId 1.239: config changed from index -1 to 239, NON_VOTER 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198) added. New config: { opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: NON_VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: true } } } I0703 01:44:09.748775 5452 tablet_bootstrap.cc:427] T dc02a796568c4677a2eebfacae32fc38 P 213596374e4142dca6ecfc701ba1c566: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:44:09.749228 5452 tablet_bootstrap.cc:427] T dc02a796568c4677a2eebfacae32fc38 P 213596374e4142dca6ecfc701ba1c566: Bootstrap complete. I0703 01:44:09.749526 5452 ts_tablet_manager.cc:1045] T dc02a796568c4677a2eebfacae32fc38 P 213596374e4142dca6ecfc701ba1c566: Time spent bootstrapping tablet: real 0.051s user 0.005s sys 0.002s I0703 01:44:09.768502 5464 ts_tablet_manager.cc:669] T 6c8253ac9a9a405885f1f57c59a84fd5 P aa8b9c58b466436faa5086e85b5224a7: Initiating tablet copy from peer 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193:54919) I0703 01:44:09.769728 5464 tablet_copy_client.cc:239] T 6c8253ac9a9a405885f1f57c59a84fd5 P aa8b9c58b466436faa5086e85b5224a7: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.193:54919 W0703 01:44:09.786695 3749 consensus_peers.cc:433] T 25b33ca4b32a4a2493d03564bfe1bf7e P 8c1e8943f5654f71b030ec7c0b733742 -> Peer 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377): Couldn't send request to peer 1157c91cc6f54a68a1d99017ac8444aa for tablet 25b33ca4b32a4a2493d03564bfe1bf7e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 2 times. I0703 01:44:09.795377 5466 ts_tablet_manager.cc:669] T 674ed5a88db04825a4eaec2fa996a852 P 1157c91cc6f54a68a1d99017ac8444aa: Initiating tablet copy from peer 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064) I0703 01:44:09.795563 5466 tablet_copy_client.cc:239] T 674ed5a88db04825a4eaec2fa996a852 P 1157c91cc6f54a68a1d99017ac8444aa: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.195:51064 I0703 01:44:09.796340 3838 tablet_copy_service.cc:135] P 8c1e8943f5654f71b030ec7c0b733742: Received BeginTabletCopySession request for tablet 674ed5a88db04825a4eaec2fa996a852 from peer 1157c91cc6f54a68a1d99017ac8444aa ({username='jenkins'} at 127.5.240.199:55739) I0703 01:44:09.796447 3838 tablet_copy_service.cc:156] P 8c1e8943f5654f71b030ec7c0b733742: Beginning new tablet copy session on tablet 674ed5a88db04825a4eaec2fa996a852 from peer 1157c91cc6f54a68a1d99017ac8444aa at {username='jenkins'} at 127.5.240.199:55739: session id = 1157c91cc6f54a68a1d99017ac8444aa-674ed5a88db04825a4eaec2fa996a852 I0703 01:44:09.797492 5452 raft_consensus.cc:325] T dc02a796568c4677a2eebfacae32fc38 P 213596374e4142dca6ecfc701ba1c566 [term 2 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { replace: true } } peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: true } } I0703 01:44:09.797998 5452 raft_consensus.cc:628] T dc02a796568c4677a2eebfacae32fc38 P 213596374e4142dca6ecfc701ba1c566 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 213596374e4142dca6ecfc701ba1c566, State: Initialized, Role: LEARNER I0703 01:44:09.799453 3838 tablet_copy_source_session.cc:202] T 674ed5a88db04825a4eaec2fa996a852 P 8c1e8943f5654f71b030ec7c0b733742: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:44:09.803107 3693 consensus_peers.cc:455] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193:54919): Closing peer: 55223945fe3d4dd8ac03ba9dab346f54 I0703 01:44:09.803400 3693 consensus_queue.cc:204] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 227, Committed index: 227, Last appended: 1.227, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: NON_VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: true } } I0703 01:44:09.812063 5466 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 674ed5a88db04825a4eaec2fa996a852. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:44:09.815577 5452 consensus_queue.cc:227] T dc02a796568c4677a2eebfacae32fc38 P 213596374e4142dca6ecfc701ba1c566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { replace: true } } peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: true } } I0703 01:44:09.818099 5466 ts_tablet_manager.cc:1173] T 674ed5a88db04825a4eaec2fa996a852 P 1157c91cc6f54a68a1d99017ac8444aa: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:44:09.820065 3572 raft_consensus.cc:1081] T 049bd2ef267f47fca634858420ac35b9 P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 FOLLOWER]: Refusing update from remote peer 57bea6dc182c41a78304417a3cbfc4a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 227. Preceding OpId from leader: term: 1 index: 228. (index mismatch) I0703 01:44:09.821055 5466 tablet_copy_client.cc:566] T 674ed5a88db04825a4eaec2fa996a852 P 1157c91cc6f54a68a1d99017ac8444aa: tablet copy: Starting download of 0 data blocks... I0703 01:44:09.822604 5247 consensus_queue.cc:962] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 228, Last known committed idx: 227, Time since last communication: 0.000s I0703 01:44:09.830224 5452 ts_tablet_manager.cc:1073] T dc02a796568c4677a2eebfacae32fc38 P 213596374e4142dca6ecfc701ba1c566: Time spent starting tablet: real 0.081s user 0.007s sys 0.002s I0703 01:44:09.831763 5258 tablet_copy_service.cc:325] P 1157c91cc6f54a68a1d99017ac8444aa: Request end of tablet copy session 213596374e4142dca6ecfc701ba1c566-dc02a796568c4677a2eebfacae32fc38 received from {username='jenkins'} at 127.5.240.196:60844 I0703 01:44:09.831887 5258 tablet_copy_service.cc:413] P 1157c91cc6f54a68a1d99017ac8444aa: ending tablet copy session 213596374e4142dca6ecfc701ba1c566-dc02a796568c4677a2eebfacae32fc38 on tablet dc02a796568c4677a2eebfacae32fc38 with peer 213596374e4142dca6ecfc701ba1c566 W0703 01:44:09.835216 3630 consensus_peers.cc:433] T 75f9ea8ba93f442aa064d4623d2c36a1 P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer 213596374e4142dca6ecfc701ba1c566 (127.5.240.196:51288): Couldn't send request to peer 213596374e4142dca6ecfc701ba1c566 for tablet 75f9ea8ba93f442aa064d4623d2c36a1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 2 times. I0703 01:44:09.837029 4556 raft_consensus.cc:2684] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 LEADER]: Committing config change with OpId 1.228: config changed from index -1 to 228, NON_VOTER aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197) added. New config: { opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: NON_VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: true } } } I0703 01:44:09.844763 3572 raft_consensus.cc:2684] T 049bd2ef267f47fca634858420ac35b9 P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 FOLLOWER]: Committing config change with OpId 1.228: config changed from index -1 to 228, NON_VOTER aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197) added. New config: { opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: NON_VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: true } } } I0703 01:44:09.845013 3445 catalog_manager.cc:3798] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 reported cstate change: config changed from index -1 to 228, NON_VOTER aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197) added. New cstate: current_term: 1 leader_uuid: "57bea6dc182c41a78304417a3cbfc4a7" committed_config { opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: NON_VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I0703 01:44:09.852038 5466 tablet_copy_client.cc:529] T 674ed5a88db04825a4eaec2fa996a852 P 1157c91cc6f54a68a1d99017ac8444aa: tablet copy: Starting download of 1 WAL segments... I0703 01:44:09.867048 3593 tablet_copy_service.cc:135] P 55223945fe3d4dd8ac03ba9dab346f54: Received BeginTabletCopySession request for tablet 6c8253ac9a9a405885f1f57c59a84fd5 from peer aa8b9c58b466436faa5086e85b5224a7 ({username='jenkins'} at 127.5.240.197:34012) I0703 01:44:09.867286 3593 tablet_copy_service.cc:156] P 55223945fe3d4dd8ac03ba9dab346f54: Beginning new tablet copy session on tablet 6c8253ac9a9a405885f1f57c59a84fd5 from peer aa8b9c58b466436faa5086e85b5224a7 at {username='jenkins'} at 127.5.240.197:34012: session id = aa8b9c58b466436faa5086e85b5224a7-6c8253ac9a9a405885f1f57c59a84fd5 I0703 01:44:09.870702 5466 tablet_copy_client.cc:405] T 674ed5a88db04825a4eaec2fa996a852 P 1157c91cc6f54a68a1d99017ac8444aa: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:44:09.874228 5466 ts_tablet_manager.cc:1028] T 674ed5a88db04825a4eaec2fa996a852 P 1157c91cc6f54a68a1d99017ac8444aa: Bootstrapping tablet I0703 01:44:09.874666 5466 tablet_bootstrap.cc:427] T 674ed5a88db04825a4eaec2fa996a852 P 1157c91cc6f54a68a1d99017ac8444aa: Bootstrap starting. I0703 01:44:09.880039 3593 tablet_copy_source_session.cc:202] T 6c8253ac9a9a405885f1f57c59a84fd5 P 55223945fe3d4dd8ac03ba9dab346f54: Tablet Copy: opened 0 blocks and 1 log segments W0703 01:44:09.881270 4879 tablet.cc:1678] T dc02a796568c4677a2eebfacae32fc38 P 213596374e4142dca6ecfc701ba1c566: Can't schedule compaction. Clean time has not been advanced past its initial value. I0703 01:44:09.887310 5464 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 6c8253ac9a9a405885f1f57c59a84fd5. 1 dirs total, 0 dirs full, 0 dirs failed W0703 01:44:09.891785 3627 consensus_peers.cc:433] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197:51329): Couldn't send request to peer aa8b9c58b466436faa5086e85b5224a7 for tablet 049bd2ef267f47fca634858420ac35b9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 049bd2ef267f47fca634858420ac35b9. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:44:09.902956 5464 ts_tablet_manager.cc:1173] T 6c8253ac9a9a405885f1f57c59a84fd5 P aa8b9c58b466436faa5086e85b5224a7: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:44:09.906730 5464 tablet_copy_client.cc:566] T 6c8253ac9a9a405885f1f57c59a84fd5 P aa8b9c58b466436faa5086e85b5224a7: tablet copy: Starting download of 0 data blocks... I0703 01:44:09.906863 5464 tablet_copy_client.cc:529] T 6c8253ac9a9a405885f1f57c59a84fd5 P aa8b9c58b466436faa5086e85b5224a7: tablet copy: Starting download of 1 WAL segments... I0703 01:44:09.920833 5464 tablet_copy_client.cc:405] T 6c8253ac9a9a405885f1f57c59a84fd5 P aa8b9c58b466436faa5086e85b5224a7: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:44:09.924533 5464 ts_tablet_manager.cc:1028] T 6c8253ac9a9a405885f1f57c59a84fd5 P aa8b9c58b466436faa5086e85b5224a7: Bootstrapping tablet I0703 01:44:09.924729 5464 tablet_bootstrap.cc:427] T 6c8253ac9a9a405885f1f57c59a84fd5 P aa8b9c58b466436faa5086e85b5224a7: Bootstrap starting. I0703 01:44:09.993314 5452 ts_tablet_manager.cc:669] T 930bbe4ac3234ed49a2207c4902541f1 P 213596374e4142dca6ecfc701ba1c566: Initiating tablet copy from peer 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746) I0703 01:44:09.993499 5452 tablet_copy_client.cc:239] T 930bbe4ac3234ed49a2207c4902541f1 P 213596374e4142dca6ecfc701ba1c566: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.194:59746 I0703 01:44:10.002095 3716 tablet_copy_service.cc:135] P 57bea6dc182c41a78304417a3cbfc4a7: Received BeginTabletCopySession request for tablet 930bbe4ac3234ed49a2207c4902541f1 from peer 213596374e4142dca6ecfc701ba1c566 ({username='jenkins'} at 127.5.240.196:36312) I0703 01:44:10.002715 3573 consensus_peers.cc:455] T 2051955b4b644a03a9adad816d350596 P 55223945fe3d4dd8ac03ba9dab346f54 -> Peer 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746): Closing peer: 57bea6dc182c41a78304417a3cbfc4a7 I0703 01:44:10.002960 3573 consensus_queue.cc:204] T 2051955b4b644a03a9adad816d350596 P 55223945fe3d4dd8ac03ba9dab346f54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 256, Committed index: 256, Last appended: 1.256, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: true } } I0703 01:44:10.017307 3716 tablet_copy_service.cc:156] P 57bea6dc182c41a78304417a3cbfc4a7: Beginning new tablet copy session on tablet 930bbe4ac3234ed49a2207c4902541f1 from peer 213596374e4142dca6ecfc701ba1c566 at {username='jenkins'} at 127.5.240.196:36312: session id = 213596374e4142dca6ecfc701ba1c566-930bbe4ac3234ed49a2207c4902541f1 I0703 01:44:10.020370 3695 raft_consensus.cc:1081] T 2051955b4b644a03a9adad816d350596 P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 FOLLOWER]: Refusing update from remote peer 55223945fe3d4dd8ac03ba9dab346f54: Log matching property violated. Preceding OpId in replica: term: 1 index: 256. Preceding OpId from leader: term: 1 index: 257. (index mismatch) W0703 01:44:10.021208 3504 consensus_peers.cc:433] T 2051955b4b644a03a9adad816d350596 P 55223945fe3d4dd8ac03ba9dab346f54 -> Peer 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377): Couldn't send request to peer 1157c91cc6f54a68a1d99017ac8444aa for tablet 2051955b4b644a03a9adad816d350596. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2051955b4b644a03a9adad816d350596. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:44:10.035315 5133 consensus_queue.cc:962] T 2051955b4b644a03a9adad816d350596 P 55223945fe3d4dd8ac03ba9dab346f54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 257, Last known committed idx: 256, Time since last communication: 0.000s I0703 01:44:10.036584 3716 tablet_copy_source_session.cc:202] T 930bbe4ac3234ed49a2207c4902541f1 P 57bea6dc182c41a78304417a3cbfc4a7: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:44:10.045686 5133 raft_consensus.cc:2684] T 2051955b4b644a03a9adad816d350596 P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 LEADER]: Committing config change with OpId 1.257: config changed from index -1 to 257, NON_VOTER 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199) added. New config: { opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: true } } } I0703 01:44:10.046764 5452 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 930bbe4ac3234ed49a2207c4902541f1. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:44:10.055649 3445 catalog_manager.cc:3798] T 2051955b4b644a03a9adad816d350596 P 55223945fe3d4dd8ac03ba9dab346f54 reported cstate change: config changed from index -1 to 257, NON_VOTER 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199) added. New cstate: current_term: 1 leader_uuid: "55223945fe3d4dd8ac03ba9dab346f54" committed_config { opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I0703 01:44:10.064288 5452 ts_tablet_manager.cc:1173] T 930bbe4ac3234ed49a2207c4902541f1 P 213596374e4142dca6ecfc701ba1c566: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:44:10.068620 5452 tablet_copy_client.cc:566] T 930bbe4ac3234ed49a2207c4902541f1 P 213596374e4142dca6ecfc701ba1c566: tablet copy: Starting download of 0 data blocks... I0703 01:44:10.069216 5452 tablet_copy_client.cc:529] T 930bbe4ac3234ed49a2207c4902541f1 P 213596374e4142dca6ecfc701ba1c566: tablet copy: Starting download of 1 WAL segments... I0703 01:44:10.070858 3692 raft_consensus.cc:2684] T 2051955b4b644a03a9adad816d350596 P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 FOLLOWER]: Committing config change with OpId 1.257: config changed from index -1 to 257, NON_VOTER 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199) added. New config: { opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: true } } } W0703 01:44:10.084468 3749 consensus_peers.cc:433] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 -> Peer aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197:51329): Couldn't send request to peer aa8b9c58b466436faa5086e85b5224a7 for tablet c352f70a51154a85a5a5b0f3d14d405f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 2 times. I0703 01:44:10.113803 4835 raft_consensus.cc:1026] T dc02a796568c4677a2eebfacae32fc38 P 213596374e4142dca6ecfc701ba1c566 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[] I0703 01:44:10.147950 5452 tablet_copy_client.cc:405] T 930bbe4ac3234ed49a2207c4902541f1 P 213596374e4142dca6ecfc701ba1c566: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:44:10.150504 5452 ts_tablet_manager.cc:1028] T 930bbe4ac3234ed49a2207c4902541f1 P 213596374e4142dca6ecfc701ba1c566: Bootstrapping tablet I0703 01:44:10.150696 5452 tablet_bootstrap.cc:427] T 930bbe4ac3234ed49a2207c4902541f1 P 213596374e4142dca6ecfc701ba1c566: Bootstrap starting. W0703 01:44:10.157891 3504 consensus_peers.cc:433] T 6c8253ac9a9a405885f1f57c59a84fd5 P 55223945fe3d4dd8ac03ba9dab346f54 -> Peer aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197:51329): Couldn't send request to peer aa8b9c58b466436faa5086e85b5224a7 for tablet 6c8253ac9a9a405885f1f57c59a84fd5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 2 times. W0703 01:44:10.169872 3630 consensus_peers.cc:433] T eea4e9534e0d4d25a827fae00bbeae4d P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198:51033): Couldn't send request to peer 6932f1cd716a48fe8d6c5573acfc48bf for tablet eea4e9534e0d4d25a827fae00bbeae4d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 3 times. W0703 01:44:10.184618 3749 consensus_peers.cc:433] T 150033054dcf498d8d71fd996af55e88 P 8c1e8943f5654f71b030ec7c0b733742 -> Peer aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197:51329): Couldn't send request to peer aa8b9c58b466436faa5086e85b5224a7 for tablet 150033054dcf498d8d71fd996af55e88. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 3 times. W0703 01:44:10.194321 3630 consensus_peers.cc:433] T 4a613c1be55c4888a4df36d3f1bfc54c P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198:51033): Couldn't send request to peer 6932f1cd716a48fe8d6c5573acfc48bf for tablet 4a613c1be55c4888a4df36d3f1bfc54c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 2 times. I0703 01:44:10.216276 5488 ts_tablet_manager.cc:669] T eada0e8b8608424594cabd35cf9891e3 P 6932f1cd716a48fe8d6c5573acfc48bf: Initiating tablet copy from peer 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193:54919) I0703 01:44:10.216514 5488 tablet_copy_client.cc:239] T eada0e8b8608424594cabd35cf9891e3 P 6932f1cd716a48fe8d6c5573acfc48bf: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.193:54919 I0703 01:44:10.219513 3572 consensus_peers.cc:455] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 -> Peer 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064): Closing peer: 8c1e8943f5654f71b030ec7c0b733742 I0703 01:44:10.219753 3572 consensus_queue.cc:204] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 292, Committed index: 292, Last appended: 1.292, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: true } } I0703 01:44:10.222386 3815 raft_consensus.cc:1081] T 60d3228294e945c5b343ae62e76dc2f0 P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Refusing update from remote peer 55223945fe3d4dd8ac03ba9dab346f54: Log matching property violated. Preceding OpId in replica: term: 1 index: 292. Preceding OpId from leader: term: 1 index: 293. (index mismatch) I0703 01:44:10.223186 5133 consensus_queue.cc:962] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 293, Last known committed idx: 292, Time since last communication: 0.000s I0703 01:44:10.226114 5442 raft_consensus.cc:2684] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 LEADER]: Committing config change with OpId 1.293: config changed from index -1 to 293, NON_VOTER 213596374e4142dca6ecfc701ba1c566 (127.5.240.196) added. New config: { opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: true } } } W0703 01:44:10.244693 3507 consensus_peers.cc:433] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 -> Peer 213596374e4142dca6ecfc701ba1c566 (127.5.240.196:51288): Couldn't send request to peer 213596374e4142dca6ecfc701ba1c566 for tablet 60d3228294e945c5b343ae62e76dc2f0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 60d3228294e945c5b343ae62e76dc2f0. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:44:10.256836 3815 raft_consensus.cc:2684] T 60d3228294e945c5b343ae62e76dc2f0 P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Committing config change with OpId 1.293: config changed from index -1 to 293, NON_VOTER 213596374e4142dca6ecfc701ba1c566 (127.5.240.196) added. New config: { opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: true } } } W0703 01:44:10.261461 3749 consensus_peers.cc:433] T 674ed5a88db04825a4eaec2fa996a852 P 8c1e8943f5654f71b030ec7c0b733742 -> Peer 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377): Couldn't send request to peer 1157c91cc6f54a68a1d99017ac8444aa for tablet 674ed5a88db04825a4eaec2fa996a852. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 2 times. I0703 01:44:10.262190 3445 catalog_manager.cc:3798] T 60d3228294e945c5b343ae62e76dc2f0 P 8c1e8943f5654f71b030ec7c0b733742 reported cstate change: config changed from index -1 to 293, NON_VOTER 213596374e4142dca6ecfc701ba1c566 (127.5.240.196) added. New cstate: current_term: 1 leader_uuid: "55223945fe3d4dd8ac03ba9dab346f54" committed_config { opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: true } } } I0703 01:44:10.276793 3593 tablet_copy_service.cc:135] P 55223945fe3d4dd8ac03ba9dab346f54: Received BeginTabletCopySession request for tablet eada0e8b8608424594cabd35cf9891e3 from peer 6932f1cd716a48fe8d6c5573acfc48bf ({username='jenkins'} at 127.5.240.198:54009) I0703 01:44:10.276914 3593 tablet_copy_service.cc:156] P 55223945fe3d4dd8ac03ba9dab346f54: Beginning new tablet copy session on tablet eada0e8b8608424594cabd35cf9891e3 from peer 6932f1cd716a48fe8d6c5573acfc48bf at {username='jenkins'} at 127.5.240.198:54009: session id = 6932f1cd716a48fe8d6c5573acfc48bf-eada0e8b8608424594cabd35cf9891e3 I0703 01:44:10.279351 3593 tablet_copy_source_session.cc:202] T eada0e8b8608424594cabd35cf9891e3 P 55223945fe3d4dd8ac03ba9dab346f54: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:44:10.283463 5488 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet eada0e8b8608424594cabd35cf9891e3. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:44:10.288091 5488 ts_tablet_manager.cc:1173] T eada0e8b8608424594cabd35cf9891e3 P 6932f1cd716a48fe8d6c5573acfc48bf: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:44:10.288265 5488 tablet_copy_client.cc:566] T eada0e8b8608424594cabd35cf9891e3 P 6932f1cd716a48fe8d6c5573acfc48bf: tablet copy: Starting download of 0 data blocks... I0703 01:44:10.288408 5488 tablet_copy_client.cc:529] T eada0e8b8608424594cabd35cf9891e3 P 6932f1cd716a48fe8d6c5573acfc48bf: tablet copy: Starting download of 1 WAL segments... I0703 01:44:10.296571 5488 tablet_copy_client.cc:405] T eada0e8b8608424594cabd35cf9891e3 P 6932f1cd716a48fe8d6c5573acfc48bf: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:44:10.301654 5488 ts_tablet_manager.cc:1028] T eada0e8b8608424594cabd35cf9891e3 P 6932f1cd716a48fe8d6c5573acfc48bf: Bootstrapping tablet I0703 01:44:10.303002 5488 tablet_bootstrap.cc:427] T eada0e8b8608424594cabd35cf9891e3 P 6932f1cd716a48fe8d6c5573acfc48bf: Bootstrap starting. W0703 01:44:10.329797 3749 consensus_peers.cc:433] T 25b33ca4b32a4a2493d03564bfe1bf7e P 8c1e8943f5654f71b030ec7c0b733742 -> Peer 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377): Couldn't send request to peer 1157c91cc6f54a68a1d99017ac8444aa for tablet 25b33ca4b32a4a2493d03564bfe1bf7e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 3 times. W0703 01:44:10.360008 3630 consensus_peers.cc:433] T 75f9ea8ba93f442aa064d4623d2c36a1 P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer 213596374e4142dca6ecfc701ba1c566 (127.5.240.196:51288): Couldn't send request to peer 213596374e4142dca6ecfc701ba1c566 for tablet 75f9ea8ba93f442aa064d4623d2c36a1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 3 times. I0703 01:44:10.463016 5506 ts_tablet_manager.cc:669] T 049bd2ef267f47fca634858420ac35b9 P aa8b9c58b466436faa5086e85b5224a7: Initiating tablet copy from peer 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746) I0703 01:44:10.463260 5506 tablet_copy_client.cc:239] T 049bd2ef267f47fca634858420ac35b9 P aa8b9c58b466436faa5086e85b5224a7: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.194:59746 I0703 01:44:10.505347 3716 tablet_copy_service.cc:135] P 57bea6dc182c41a78304417a3cbfc4a7: Received BeginTabletCopySession request for tablet 049bd2ef267f47fca634858420ac35b9 from peer aa8b9c58b466436faa5086e85b5224a7 ({username='jenkins'} at 127.5.240.197:44868) I0703 01:44:10.505450 3716 tablet_copy_service.cc:156] P 57bea6dc182c41a78304417a3cbfc4a7: Beginning new tablet copy session on tablet 049bd2ef267f47fca634858420ac35b9 from peer aa8b9c58b466436faa5086e85b5224a7 at {username='jenkins'} at 127.5.240.197:44868: session id = aa8b9c58b466436faa5086e85b5224a7-049bd2ef267f47fca634858420ac35b9 I0703 01:44:10.507820 3716 tablet_copy_source_session.cc:202] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:44:10.522794 5414 raft_consensus.cc:890] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa: attempting to promote NON_VOTER 213596374e4142dca6ecfc701ba1c566 to VOTER I0703 01:44:10.523341 5414 consensus_peers.cc:455] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa -> Peer 213596374e4142dca6ecfc701ba1c566 (127.5.240.196:51288): Closing peer: 213596374e4142dca6ecfc701ba1c566 I0703 01:44:10.523442 5414 consensus_peers.cc:455] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa -> Peer 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064): Closing peer: 8c1e8943f5654f71b030ec7c0b733742 I0703 01:44:10.523756 5414 consensus_queue.cc:204] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { replace: true } } peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } } I0703 01:44:10.546242 5506 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 049bd2ef267f47fca634858420ac35b9. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:44:10.550577 4835 raft_consensus.cc:1081] T dc02a796568c4677a2eebfacae32fc38 P 213596374e4142dca6ecfc701ba1c566 [term 2 LEARNER]: Refusing update from remote peer 1157c91cc6f54a68a1d99017ac8444aa: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch) I0703 01:44:10.551380 3815 raft_consensus.cc:1081] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [term 2 FOLLOWER]: Refusing update from remote peer 1157c91cc6f54a68a1d99017ac8444aa: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch) I0703 01:44:10.551390 5414 consensus_queue.cc:962] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I0703 01:44:10.552116 5506 ts_tablet_manager.cc:1173] T 049bd2ef267f47fca634858420ac35b9 P aa8b9c58b466436faa5086e85b5224a7: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:44:10.552357 5506 tablet_copy_client.cc:566] T 049bd2ef267f47fca634858420ac35b9 P aa8b9c58b466436faa5086e85b5224a7: tablet copy: Starting download of 0 data blocks... I0703 01:44:10.552508 5506 tablet_copy_client.cc:529] T 049bd2ef267f47fca634858420ac35b9 P aa8b9c58b466436faa5086e85b5224a7: tablet copy: Starting download of 1 WAL segments... I0703 01:44:10.553740 5414 consensus_queue.cc:962] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s I0703 01:44:10.568092 5482 raft_consensus.cc:2684] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 213596374e4142dca6ecfc701ba1c566 (127.5.240.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { replace: true } } peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } } } I0703 01:44:10.572294 3445 catalog_manager.cc:3798] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa reported cstate change: config changed from index 3 to 4, 213596374e4142dca6ecfc701ba1c566 (127.5.240.196) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "1157c91cc6f54a68a1d99017ac8444aa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } health_report { overall_health: UNKNOWN } } } I0703 01:44:10.581620 3815 raft_consensus.cc:2684] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 213596374e4142dca6ecfc701ba1c566 (127.5.240.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { replace: true } } peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } } } W0703 01:44:10.592139 3504 consensus_peers.cc:433] T 6c8253ac9a9a405885f1f57c59a84fd5 P 55223945fe3d4dd8ac03ba9dab346f54 -> Peer aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197:51329): Couldn't send request to peer aa8b9c58b466436faa5086e85b5224a7 for tablet 6c8253ac9a9a405885f1f57c59a84fd5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 3 times. W0703 01:44:10.623534 3630 consensus_peers.cc:433] T 4a613c1be55c4888a4df36d3f1bfc54c P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198:51033): Couldn't send request to peer 6932f1cd716a48fe8d6c5573acfc48bf for tablet 4a613c1be55c4888a4df36d3f1bfc54c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 3 times. W0703 01:44:10.623792 3630 consensus_peers.cc:433] T 930bbe4ac3234ed49a2207c4902541f1 P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer 213596374e4142dca6ecfc701ba1c566 (127.5.240.196:51288): Couldn't send request to peer 213596374e4142dca6ecfc701ba1c566 for tablet 930bbe4ac3234ed49a2207c4902541f1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 2 times. W0703 01:44:10.634632 3507 consensus_peers.cc:433] T eada0e8b8608424594cabd35cf9891e3 P 55223945fe3d4dd8ac03ba9dab346f54 -> Peer 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198:51033): Couldn't send request to peer 6932f1cd716a48fe8d6c5573acfc48bf for tablet eada0e8b8608424594cabd35cf9891e3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 2 times. I0703 01:44:10.636116 4835 raft_consensus.cc:2684] T dc02a796568c4677a2eebfacae32fc38 P 213596374e4142dca6ecfc701ba1c566 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 213596374e4142dca6ecfc701ba1c566 (127.5.240.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { replace: true } } peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } } } I0703 01:44:10.646471 5506 tablet_copy_client.cc:405] T 049bd2ef267f47fca634858420ac35b9 P aa8b9c58b466436faa5086e85b5224a7: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:44:10.648924 5506 ts_tablet_manager.cc:1028] T 049bd2ef267f47fca634858420ac35b9 P aa8b9c58b466436faa5086e85b5224a7: Bootstrapping tablet I0703 01:44:10.649070 5506 tablet_bootstrap.cc:427] T 049bd2ef267f47fca634858420ac35b9 P aa8b9c58b466436faa5086e85b5224a7: Bootstrap starting. I0703 01:44:10.651190 5520 ts_tablet_manager.cc:669] T 2051955b4b644a03a9adad816d350596 P 1157c91cc6f54a68a1d99017ac8444aa: Initiating tablet copy from peer 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193:54919) I0703 01:44:10.656733 5520 tablet_copy_client.cc:239] T 2051955b4b644a03a9adad816d350596 P 1157c91cc6f54a68a1d99017ac8444aa: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.193:54919 W0703 01:44:10.690654 3749 consensus_peers.cc:433] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 -> Peer aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197:51329): Couldn't send request to peer aa8b9c58b466436faa5086e85b5224a7 for tablet c352f70a51154a85a5a5b0f3d14d405f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 3 times. I0703 01:44:10.738294 3593 tablet_copy_service.cc:135] P 55223945fe3d4dd8ac03ba9dab346f54: Received BeginTabletCopySession request for tablet 2051955b4b644a03a9adad816d350596 from peer 1157c91cc6f54a68a1d99017ac8444aa ({username='jenkins'} at 127.5.240.199:60222) I0703 01:44:10.738399 3593 tablet_copy_service.cc:156] P 55223945fe3d4dd8ac03ba9dab346f54: Beginning new tablet copy session on tablet 2051955b4b644a03a9adad816d350596 from peer 1157c91cc6f54a68a1d99017ac8444aa at {username='jenkins'} at 127.5.240.199:60222: session id = 1157c91cc6f54a68a1d99017ac8444aa-2051955b4b644a03a9adad816d350596 I0703 01:44:10.740818 3593 tablet_copy_source_session.cc:202] T 2051955b4b644a03a9adad816d350596 P 55223945fe3d4dd8ac03ba9dab346f54: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:44:10.742195 5520 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 2051955b4b644a03a9adad816d350596. 1 dirs total, 0 dirs full, 0 dirs failed W0703 01:44:10.747478 3749 consensus_peers.cc:433] T 674ed5a88db04825a4eaec2fa996a852 P 8c1e8943f5654f71b030ec7c0b733742 -> Peer 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377): Couldn't send request to peer 1157c91cc6f54a68a1d99017ac8444aa for tablet 674ed5a88db04825a4eaec2fa996a852. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 3 times. I0703 01:44:10.755587 5520 ts_tablet_manager.cc:1173] T 2051955b4b644a03a9adad816d350596 P 1157c91cc6f54a68a1d99017ac8444aa: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:44:10.755818 5520 tablet_copy_client.cc:566] T 2051955b4b644a03a9adad816d350596 P 1157c91cc6f54a68a1d99017ac8444aa: tablet copy: Starting download of 0 data blocks... I0703 01:44:10.756181 5520 tablet_copy_client.cc:529] T 2051955b4b644a03a9adad816d350596 P 1157c91cc6f54a68a1d99017ac8444aa: tablet copy: Starting download of 1 WAL segments... I0703 01:44:10.771059 5520 tablet_copy_client.cc:405] T 2051955b4b644a03a9adad816d350596 P 1157c91cc6f54a68a1d99017ac8444aa: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:44:10.776664 5520 ts_tablet_manager.cc:1028] T 2051955b4b644a03a9adad816d350596 P 1157c91cc6f54a68a1d99017ac8444aa: Bootstrapping tablet I0703 01:44:10.777806 5520 tablet_bootstrap.cc:427] T 2051955b4b644a03a9adad816d350596 P 1157c91cc6f54a68a1d99017ac8444aa: Bootstrap starting. W0703 01:44:10.780704 3749 consensus_peers.cc:433] T 150033054dcf498d8d71fd996af55e88 P 8c1e8943f5654f71b030ec7c0b733742 -> Peer aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197:51329): Couldn't send request to peer aa8b9c58b466436faa5086e85b5224a7 for tablet 150033054dcf498d8d71fd996af55e88. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 4 times. W0703 01:44:10.814178 3630 consensus_peers.cc:433] T eea4e9534e0d4d25a827fae00bbeae4d P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198:51033): Couldn't send request to peer 6932f1cd716a48fe8d6c5573acfc48bf for tablet eea4e9534e0d4d25a827fae00bbeae4d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 4 times. I0703 01:44:10.835006 5528 ts_tablet_manager.cc:669] T 60d3228294e945c5b343ae62e76dc2f0 P 213596374e4142dca6ecfc701ba1c566: Initiating tablet copy from peer 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193:54919) I0703 01:44:10.835294 5528 tablet_copy_client.cc:239] T 60d3228294e945c5b343ae62e76dc2f0 P 213596374e4142dca6ecfc701ba1c566: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.193:54919 I0703 01:44:10.890717 3593 tablet_copy_service.cc:135] P 55223945fe3d4dd8ac03ba9dab346f54: Received BeginTabletCopySession request for tablet 60d3228294e945c5b343ae62e76dc2f0 from peer 213596374e4142dca6ecfc701ba1c566 ({username='jenkins'} at 127.5.240.196:57370) I0703 01:44:10.890835 3593 tablet_copy_service.cc:156] P 55223945fe3d4dd8ac03ba9dab346f54: Beginning new tablet copy session on tablet 60d3228294e945c5b343ae62e76dc2f0 from peer 213596374e4142dca6ecfc701ba1c566 at {username='jenkins'} at 127.5.240.196:57370: session id = 213596374e4142dca6ecfc701ba1c566-60d3228294e945c5b343ae62e76dc2f0 I0703 01:44:10.893220 3593 tablet_copy_source_session.cc:202] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:44:10.895367 5528 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 60d3228294e945c5b343ae62e76dc2f0. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:44:10.900297 5528 ts_tablet_manager.cc:1173] T 60d3228294e945c5b343ae62e76dc2f0 P 213596374e4142dca6ecfc701ba1c566: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:44:10.900725 5528 tablet_copy_client.cc:566] T 60d3228294e945c5b343ae62e76dc2f0 P 213596374e4142dca6ecfc701ba1c566: tablet copy: Starting download of 0 data blocks... I0703 01:44:10.900882 5528 tablet_copy_client.cc:529] T 60d3228294e945c5b343ae62e76dc2f0 P 213596374e4142dca6ecfc701ba1c566: tablet copy: Starting download of 1 WAL segments... W0703 01:44:10.911866 3627 consensus_peers.cc:433] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197:51329): Couldn't send request to peer aa8b9c58b466436faa5086e85b5224a7 for tablet 049bd2ef267f47fca634858420ac35b9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 2 times. I0703 01:44:10.915278 5528 tablet_copy_client.cc:405] T 60d3228294e945c5b343ae62e76dc2f0 P 213596374e4142dca6ecfc701ba1c566: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:44:10.918227 5528 ts_tablet_manager.cc:1028] T 60d3228294e945c5b343ae62e76dc2f0 P 213596374e4142dca6ecfc701ba1c566: Bootstrapping tablet I0703 01:44:10.918390 5528 tablet_bootstrap.cc:427] T 60d3228294e945c5b343ae62e76dc2f0 P 213596374e4142dca6ecfc701ba1c566: Bootstrap starting. W0703 01:44:10.958052 3630 consensus_peers.cc:433] T 75f9ea8ba93f442aa064d4623d2c36a1 P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer 213596374e4142dca6ecfc701ba1c566 (127.5.240.196:51288): Couldn't send request to peer 213596374e4142dca6ecfc701ba1c566 for tablet 75f9ea8ba93f442aa064d4623d2c36a1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 4 times. W0703 01:44:10.961848 3749 consensus_peers.cc:433] T 25b33ca4b32a4a2493d03564bfe1bf7e P 8c1e8943f5654f71b030ec7c0b733742 -> Peer 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377): Couldn't send request to peer 1157c91cc6f54a68a1d99017ac8444aa for tablet 25b33ca4b32a4a2493d03564bfe1bf7e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 4 times. W0703 01:44:11.022521 3630 consensus_peers.cc:433] T 4a613c1be55c4888a4df36d3f1bfc54c P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198:51033): Couldn't send request to peer 6932f1cd716a48fe8d6c5573acfc48bf for tablet 4a613c1be55c4888a4df36d3f1bfc54c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 4 times. W0703 01:44:11.101778 3630 consensus_peers.cc:433] T 930bbe4ac3234ed49a2207c4902541f1 P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer 213596374e4142dca6ecfc701ba1c566 (127.5.240.196:51288): Couldn't send request to peer 213596374e4142dca6ecfc701ba1c566 for tablet 930bbe4ac3234ed49a2207c4902541f1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 3 times. W0703 01:44:11.149111 3749 consensus_peers.cc:433] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 -> Peer aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197:51329): Couldn't send request to peer aa8b9c58b466436faa5086e85b5224a7 for tablet c352f70a51154a85a5a5b0f3d14d405f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 4 times. W0703 01:44:11.159688 3749 consensus_peers.cc:433] T 674ed5a88db04825a4eaec2fa996a852 P 8c1e8943f5654f71b030ec7c0b733742 -> Peer 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377): Couldn't send request to peer 1157c91cc6f54a68a1d99017ac8444aa for tablet 674ed5a88db04825a4eaec2fa996a852. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 4 times. W0703 01:44:11.166288 5433 log.cc:667] Time spent T 75f9ea8ba93f442aa064d4623d2c36a1 P 213596374e4142dca6ecfc701ba1c566: Append to log took a long time: real 0.068s user 0.001s sys 0.000s W0703 01:44:11.180104 3504 consensus_peers.cc:433] T 2051955b4b644a03a9adad816d350596 P 55223945fe3d4dd8ac03ba9dab346f54 -> Peer 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377): Couldn't send request to peer 1157c91cc6f54a68a1d99017ac8444aa for tablet 2051955b4b644a03a9adad816d350596. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 2 times. W0703 01:44:11.207759 3504 consensus_peers.cc:433] T 6c8253ac9a9a405885f1f57c59a84fd5 P 55223945fe3d4dd8ac03ba9dab346f54 -> Peer aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197:51329): Couldn't send request to peer aa8b9c58b466436faa5086e85b5224a7 for tablet 6c8253ac9a9a405885f1f57c59a84fd5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 4 times. W0703 01:44:11.223155 3507 consensus_peers.cc:433] T eada0e8b8608424594cabd35cf9891e3 P 55223945fe3d4dd8ac03ba9dab346f54 -> Peer 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198:51033): Couldn't send request to peer 6932f1cd716a48fe8d6c5573acfc48bf for tablet eada0e8b8608424594cabd35cf9891e3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 3 times. W0703 01:44:11.239066 3749 consensus_peers.cc:433] T 150033054dcf498d8d71fd996af55e88 P 8c1e8943f5654f71b030ec7c0b733742 -> Peer aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197:51329): Couldn't send request to peer aa8b9c58b466436faa5086e85b5224a7 for tablet 150033054dcf498d8d71fd996af55e88. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 5 times. W0703 01:44:11.243516 3507 consensus_peers.cc:433] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 -> Peer 213596374e4142dca6ecfc701ba1c566 (127.5.240.196:51288): Couldn't send request to peer 213596374e4142dca6ecfc701ba1c566 for tablet 60d3228294e945c5b343ae62e76dc2f0. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 2 times. W0703 01:44:11.294612 5412 log.cc:667] Time spent T eea4e9534e0d4d25a827fae00bbeae4d P 6932f1cd716a48fe8d6c5573acfc48bf: Append to log took a long time: real 0.061s user 0.001s sys 0.000s W0703 01:44:11.367245 3630 consensus_peers.cc:433] T 75f9ea8ba93f442aa064d4623d2c36a1 P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer 213596374e4142dca6ecfc701ba1c566 (127.5.240.196:51288): Couldn't send request to peer 213596374e4142dca6ecfc701ba1c566 for tablet 75f9ea8ba93f442aa064d4623d2c36a1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 5 times. I0703 01:44:11.372157 5416 tablet_bootstrap.cc:427] T 150033054dcf498d8d71fd996af55e88 P aa8b9c58b466436faa5086e85b5224a7: Bootstrap replayed 1/1 log segments. Stats: ops{read=211 overwritten=0 applied=211 ignored=0} inserts{seen=898 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:44:11.372658 5416 tablet_bootstrap.cc:427] T 150033054dcf498d8d71fd996af55e88 P aa8b9c58b466436faa5086e85b5224a7: Bootstrap complete. I0703 01:44:11.373569 5416 ts_tablet_manager.cc:1045] T 150033054dcf498d8d71fd996af55e88 P aa8b9c58b466436faa5086e85b5224a7: Time spent bootstrapping tablet: real 2.116s user 0.298s sys 0.021s I0703 01:44:11.414237 5235 tablet_service.cc:1140] Received LeaderStepDown RPC: tablet_id: "dc02a796568c4677a2eebfacae32fc38" dest_uuid: "1157c91cc6f54a68a1d99017ac8444aa" from {username='jenkins'} at 127.5.240.199:46104 I0703 01:44:11.414438 5235 raft_consensus.cc:552] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa [term 2 LEADER]: Received request to step down I0703 01:44:11.414494 5235 raft_consensus.cc:628] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1157c91cc6f54a68a1d99017ac8444aa, State: Running, Role: LEADER I0703 01:44:11.414727 5235 consensus_queue.cc:227] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { replace: true } } peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } } I0703 01:44:11.414940 5235 consensus_peers.cc:455] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa -> Peer 213596374e4142dca6ecfc701ba1c566 (127.5.240.196:51288): Closing peer: 213596374e4142dca6ecfc701ba1c566 I0703 01:44:11.414999 5235 consensus_peers.cc:455] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa -> Peer 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064): Closing peer: 8c1e8943f5654f71b030ec7c0b733742 I0703 01:44:11.415154 5235 raft_consensus.cc:2730] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa: Snoozing failure detection for 3.000s (explicit stepdown request) W0703 01:44:11.419240 3630 consensus_peers.cc:433] T eea4e9534e0d4d25a827fae00bbeae4d P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198:51033): Couldn't send request to peer 6932f1cd716a48fe8d6c5573acfc48bf for tablet eea4e9534e0d4d25a827fae00bbeae4d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 5 times. I0703 01:44:11.395195 5416 raft_consensus.cc:325] T 150033054dcf498d8d71fd996af55e88 P aa8b9c58b466436faa5086e85b5224a7 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 200 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: NON_VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: true } } I0703 01:44:11.440950 5416 raft_consensus.cc:628] T 150033054dcf498d8d71fd996af55e88 P aa8b9c58b466436faa5086e85b5224a7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aa8b9c58b466436faa5086e85b5224a7, State: Initialized, Role: LEARNER I0703 01:44:11.441344 5416 consensus_queue.cc:227] T 150033054dcf498d8d71fd996af55e88 P aa8b9c58b466436faa5086e85b5224a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 211, Last appended: 1.211, Last appended by leader: 211, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 200 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: NON_VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: true } } I0703 01:44:11.454177 5416 ts_tablet_manager.cc:1073] T 150033054dcf498d8d71fd996af55e88 P aa8b9c58b466436faa5086e85b5224a7: Time spent starting tablet: real 0.080s user 0.007s sys 0.002s I0703 01:44:11.455080 3838 tablet_copy_service.cc:325] P 8c1e8943f5654f71b030ec7c0b733742: Request end of tablet copy session aa8b9c58b466436faa5086e85b5224a7-150033054dcf498d8d71fd996af55e88 received from {username='jenkins'} at 127.5.240.197:45440 I0703 01:44:11.455196 3838 tablet_copy_service.cc:413] P 8c1e8943f5654f71b030ec7c0b733742: ending tablet copy session aa8b9c58b466436faa5086e85b5224a7-150033054dcf498d8d71fd996af55e88 on tablet 150033054dcf498d8d71fd996af55e88 with peer aa8b9c58b466436faa5086e85b5224a7 I0703 01:44:11.471240 5433 tablet_bootstrap.cc:427] T 75f9ea8ba93f442aa064d4623d2c36a1 P 213596374e4142dca6ecfc701ba1c566: Bootstrap replayed 1/1 log segments. Stats: ops{read=211 overwritten=0 applied=210 ignored=0} inserts{seen=897 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates I0703 01:44:11.496349 5433 tablet_bootstrap.cc:427] T 75f9ea8ba93f442aa064d4623d2c36a1 P 213596374e4142dca6ecfc701ba1c566: Bootstrap complete. I0703 01:44:11.497018 5433 ts_tablet_manager.cc:1045] T 75f9ea8ba93f442aa064d4623d2c36a1 P 213596374e4142dca6ecfc701ba1c566: Time spent bootstrapping tablet: real 2.072s user 0.300s sys 0.013s I0703 01:44:11.502076 5433 raft_consensus.cc:325] T 75f9ea8ba93f442aa064d4623d2c36a1 P 213596374e4142dca6ecfc701ba1c566 [term 1 LEARNER]: Replica starting. Triggering 1 pending transactions. Active config: opid_index: 198 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: true } } W0703 01:44:11.515985 3627 consensus_peers.cc:433] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197:51329): Couldn't send request to peer aa8b9c58b466436faa5086e85b5224a7 for tablet 049bd2ef267f47fca634858420ac35b9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 3 times. W0703 01:44:11.523250 3749 consensus_peers.cc:433] T 25b33ca4b32a4a2493d03564bfe1bf7e P 8c1e8943f5654f71b030ec7c0b733742 -> Peer 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377): Couldn't send request to peer 1157c91cc6f54a68a1d99017ac8444aa for tablet 25b33ca4b32a4a2493d03564bfe1bf7e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 5 times. W0703 01:44:11.532001 3749 consensus_peers.cc:433] T 674ed5a88db04825a4eaec2fa996a852 P 8c1e8943f5654f71b030ec7c0b733742 -> Peer 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377): Couldn't send request to peer 1157c91cc6f54a68a1d99017ac8444aa for tablet 674ed5a88db04825a4eaec2fa996a852. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 5 times. I0703 01:44:11.563043 5433 raft_consensus.cc:628] T 75f9ea8ba93f442aa064d4623d2c36a1 P 213596374e4142dca6ecfc701ba1c566 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 213596374e4142dca6ecfc701ba1c566, State: Initialized, Role: LEARNER I0703 01:44:11.563241 5433 consensus_queue.cc:227] T 75f9ea8ba93f442aa064d4623d2c36a1 P 213596374e4142dca6ecfc701ba1c566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 210, Last appended: 1.211, Last appended by leader: 211, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 198 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: true } } I0703 01:44:11.574493 5433 ts_tablet_manager.cc:1073] T 75f9ea8ba93f442aa064d4623d2c36a1 P 213596374e4142dca6ecfc701ba1c566: Time spent starting tablet: real 0.076s user 0.002s sys 0.004s I0703 01:44:11.575459 3716 tablet_copy_service.cc:325] P 57bea6dc182c41a78304417a3cbfc4a7: Request end of tablet copy session 213596374e4142dca6ecfc701ba1c566-75f9ea8ba93f442aa064d4623d2c36a1 received from {username='jenkins'} at 127.5.240.196:36312 I0703 01:44:11.575603 3716 tablet_copy_service.cc:413] P 57bea6dc182c41a78304417a3cbfc4a7: ending tablet copy session 213596374e4142dca6ecfc701ba1c566-75f9ea8ba93f442aa064d4623d2c36a1 on tablet 75f9ea8ba93f442aa064d4623d2c36a1 with peer 213596374e4142dca6ecfc701ba1c566 W0703 01:44:11.586817 3507 consensus_peers.cc:433] T eada0e8b8608424594cabd35cf9891e3 P 55223945fe3d4dd8ac03ba9dab346f54 -> Peer 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198:51033): Couldn't send request to peer 6932f1cd716a48fe8d6c5573acfc48bf for tablet eada0e8b8608424594cabd35cf9891e3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 4 times. W0703 01:44:11.594975 3504 consensus_peers.cc:433] T 6c8253ac9a9a405885f1f57c59a84fd5 P 55223945fe3d4dd8ac03ba9dab346f54 -> Peer aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197:51329): Couldn't send request to peer aa8b9c58b466436faa5086e85b5224a7 for tablet 6c8253ac9a9a405885f1f57c59a84fd5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 5 times. W0703 01:44:11.595477 3630 consensus_peers.cc:433] T 4a613c1be55c4888a4df36d3f1bfc54c P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198:51033): Couldn't send request to peer 6932f1cd716a48fe8d6c5573acfc48bf for tablet 4a613c1be55c4888a4df36d3f1bfc54c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 5 times. W0703 01:44:11.646720 3630 consensus_peers.cc:433] T 930bbe4ac3234ed49a2207c4902541f1 P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer 213596374e4142dca6ecfc701ba1c566 (127.5.240.196:51288): Couldn't send request to peer 213596374e4142dca6ecfc701ba1c566 for tablet 930bbe4ac3234ed49a2207c4902541f1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 4 times. W0703 01:44:11.674563 3507 consensus_peers.cc:433] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 -> Peer 213596374e4142dca6ecfc701ba1c566 (127.5.240.196:51288): Couldn't send request to peer 213596374e4142dca6ecfc701ba1c566 for tablet 60d3228294e945c5b343ae62e76dc2f0. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 3 times. W0703 01:44:11.692335 3749 consensus_peers.cc:433] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 -> Peer aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197:51329): Couldn't send request to peer aa8b9c58b466436faa5086e85b5224a7 for tablet c352f70a51154a85a5a5b0f3d14d405f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 5 times. W0703 01:44:11.779871 3504 consensus_peers.cc:433] T 2051955b4b644a03a9adad816d350596 P 55223945fe3d4dd8ac03ba9dab346f54 -> Peer 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377): Couldn't send request to peer 1157c91cc6f54a68a1d99017ac8444aa for tablet 2051955b4b644a03a9adad816d350596. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 3 times. I0703 01:44:11.794934 4966 raft_consensus.cc:1026] T 150033054dcf498d8d71fd996af55e88 P aa8b9c58b466436faa5086e85b5224a7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.199->[1.200-1.243] Dedup: 1.211->[1.212-1.243] W0703 01:44:11.831482 3630 consensus_peers.cc:433] T eea4e9534e0d4d25a827fae00bbeae4d P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198:51033): Couldn't send request to peer 6932f1cd716a48fe8d6c5573acfc48bf for tablet eea4e9534e0d4d25a827fae00bbeae4d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 6 times. I0703 01:44:11.859906 4835 raft_consensus.cc:1026] T 75f9ea8ba93f442aa064d4623d2c36a1 P 213596374e4142dca6ecfc701ba1c566 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.197->[1.198-1.238] Dedup: 1.211->[1.212-1.238] I0703 01:44:11.879292 5444 tablet_bootstrap.cc:427] T c352f70a51154a85a5a5b0f3d14d405f P aa8b9c58b466436faa5086e85b5224a7: Bootstrap replayed 1/1 log segments. Stats: ops{read=254 overwritten=0 applied=254 ignored=0} inserts{seen=903 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:44:11.879731 5444 tablet_bootstrap.cc:427] T c352f70a51154a85a5a5b0f3d14d405f P aa8b9c58b466436faa5086e85b5224a7: Bootstrap complete. I0703 01:44:11.880024 5444 ts_tablet_manager.cc:1045] T c352f70a51154a85a5a5b0f3d14d405f P aa8b9c58b466436faa5086e85b5224a7: Time spent bootstrapping tablet: real 2.298s user 0.329s sys 0.025s I0703 01:44:11.904007 5444 raft_consensus.cc:325] T c352f70a51154a85a5a5b0f3d14d405f P aa8b9c58b466436faa5086e85b5224a7 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: NON_VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: true } } I0703 01:44:11.904359 5444 raft_consensus.cc:628] T c352f70a51154a85a5a5b0f3d14d405f P aa8b9c58b466436faa5086e85b5224a7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aa8b9c58b466436faa5086e85b5224a7, State: Initialized, Role: LEARNER I0703 01:44:11.904554 5444 consensus_queue.cc:227] T c352f70a51154a85a5a5b0f3d14d405f P aa8b9c58b466436faa5086e85b5224a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 254, Last appended: 1.254, Last appended by leader: 254, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 238 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: NON_VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: true } } I0703 01:44:11.905642 5444 ts_tablet_manager.cc:1073] T c352f70a51154a85a5a5b0f3d14d405f P aa8b9c58b466436faa5086e85b5224a7: Time spent starting tablet: real 0.025s user 0.002s sys 0.000s W0703 01:44:11.908874 5488 log.cc:667] Time spent T eada0e8b8608424594cabd35cf9891e3 P 6932f1cd716a48fe8d6c5573acfc48bf: Append to log took a long time: real 0.051s user 0.001s sys 0.000s I0703 01:44:11.922385 3838 tablet_copy_service.cc:325] P 8c1e8943f5654f71b030ec7c0b733742: Request end of tablet copy session aa8b9c58b466436faa5086e85b5224a7-c352f70a51154a85a5a5b0f3d14d405f received from {username='jenkins'} at 127.5.240.197:45440 I0703 01:44:11.922461 3838 tablet_copy_service.cc:413] P 8c1e8943f5654f71b030ec7c0b733742: ending tablet copy session aa8b9c58b466436faa5086e85b5224a7-c352f70a51154a85a5a5b0f3d14d405f on tablet c352f70a51154a85a5a5b0f3d14d405f with peer aa8b9c58b466436faa5086e85b5224a7 W0703 01:44:11.935210 3627 consensus_peers.cc:433] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197:51329): Couldn't send request to peer aa8b9c58b466436faa5086e85b5224a7 for tablet 049bd2ef267f47fca634858420ac35b9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 4 times. I0703 01:44:11.947608 5412 tablet_bootstrap.cc:427] T eea4e9534e0d4d25a827fae00bbeae4d P 6932f1cd716a48fe8d6c5573acfc48bf: Bootstrap replayed 1/1 log segments. Stats: ops{read=209 overwritten=0 applied=209 ignored=0} inserts{seen=861 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:44:11.948088 5412 tablet_bootstrap.cc:427] T eea4e9534e0d4d25a827fae00bbeae4d P 6932f1cd716a48fe8d6c5573acfc48bf: Bootstrap complete. I0703 01:44:11.956598 5412 ts_tablet_manager.cc:1045] T eea4e9534e0d4d25a827fae00bbeae4d P 6932f1cd716a48fe8d6c5573acfc48bf: Time spent bootstrapping tablet: real 2.760s user 0.298s sys 0.021s W0703 01:44:11.958539 3749 consensus_peers.cc:433] T 674ed5a88db04825a4eaec2fa996a852 P 8c1e8943f5654f71b030ec7c0b733742 -> Peer 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377): Couldn't send request to peer 1157c91cc6f54a68a1d99017ac8444aa for tablet 674ed5a88db04825a4eaec2fa996a852. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 6 times. W0703 01:44:12.002653 5464 log.cc:667] Time spent T 6c8253ac9a9a405885f1f57c59a84fd5 P aa8b9c58b466436faa5086e85b5224a7: Append to log took a long time: real 0.077s user 0.001s sys 0.000s W0703 01:44:12.024260 3504 consensus_peers.cc:433] T 6c8253ac9a9a405885f1f57c59a84fd5 P 55223945fe3d4dd8ac03ba9dab346f54 -> Peer aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197:51329): Couldn't send request to peer aa8b9c58b466436faa5086e85b5224a7 for tablet 6c8253ac9a9a405885f1f57c59a84fd5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 6 times. I0703 01:44:12.026372 5412 raft_consensus.cc:325] T eea4e9534e0d4d25a827fae00bbeae4d P 6932f1cd716a48fe8d6c5573acfc48bf [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 199 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: NON_VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: true } } I0703 01:44:12.027663 5412 raft_consensus.cc:628] T eea4e9534e0d4d25a827fae00bbeae4d P 6932f1cd716a48fe8d6c5573acfc48bf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6932f1cd716a48fe8d6c5573acfc48bf, State: Initialized, Role: LEARNER I0703 01:44:12.028046 5412 consensus_queue.cc:227] T eea4e9534e0d4d25a827fae00bbeae4d P 6932f1cd716a48fe8d6c5573acfc48bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 209, Last appended: 1.209, Last appended by leader: 209, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 199 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: NON_VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: true } } W0703 01:44:12.029105 3630 consensus_peers.cc:433] T 4a613c1be55c4888a4df36d3f1bfc54c P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198:51033): Couldn't send request to peer 6932f1cd716a48fe8d6c5573acfc48bf for tablet 4a613c1be55c4888a4df36d3f1bfc54c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 6 times. W0703 01:44:12.044775 3749 consensus_peers.cc:433] T 25b33ca4b32a4a2493d03564bfe1bf7e P 8c1e8943f5654f71b030ec7c0b733742 -> Peer 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377): Couldn't send request to peer 1157c91cc6f54a68a1d99017ac8444aa for tablet 25b33ca4b32a4a2493d03564bfe1bf7e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 6 times. I0703 01:44:12.051990 5412 ts_tablet_manager.cc:1073] T eea4e9534e0d4d25a827fae00bbeae4d P 6932f1cd716a48fe8d6c5573acfc48bf: Time spent starting tablet: real 0.095s user 0.007s sys 0.001s I0703 01:44:12.052989 3716 tablet_copy_service.cc:325] P 57bea6dc182c41a78304417a3cbfc4a7: Request end of tablet copy session 6932f1cd716a48fe8d6c5573acfc48bf-eea4e9534e0d4d25a827fae00bbeae4d received from {username='jenkins'} at 127.5.240.198:49512 I0703 01:44:12.053099 3716 tablet_copy_service.cc:413] P 57bea6dc182c41a78304417a3cbfc4a7: ending tablet copy session 6932f1cd716a48fe8d6c5573acfc48bf-eea4e9534e0d4d25a827fae00bbeae4d on tablet eea4e9534e0d4d25a827fae00bbeae4d with peer 6932f1cd716a48fe8d6c5573acfc48bf W0703 01:44:12.060778 3507 consensus_peers.cc:433] T eada0e8b8608424594cabd35cf9891e3 P 55223945fe3d4dd8ac03ba9dab346f54 -> Peer 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198:51033): Couldn't send request to peer 6932f1cd716a48fe8d6c5573acfc48bf for tablet eada0e8b8608424594cabd35cf9891e3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 5 times. I0703 01:44:12.075130 4744 raft_consensus.cc:890] T 150033054dcf498d8d71fd996af55e88 P 8c1e8943f5654f71b030ec7c0b733742: attempting to promote NON_VOTER aa8b9c58b466436faa5086e85b5224a7 to VOTER I0703 01:44:12.075536 4744 consensus_peers.cc:455] T 150033054dcf498d8d71fd996af55e88 P 8c1e8943f5654f71b030ec7c0b733742 -> Peer aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197:51329): Closing peer: aa8b9c58b466436faa5086e85b5224a7 I0703 01:44:12.075595 4744 consensus_peers.cc:455] T 150033054dcf498d8d71fd996af55e88 P 8c1e8943f5654f71b030ec7c0b733742 -> Peer 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746): Closing peer: 57bea6dc182c41a78304417a3cbfc4a7 I0703 01:44:12.075950 4744 consensus_queue.cc:204] T 150033054dcf498d8d71fd996af55e88 P 8c1e8943f5654f71b030ec7c0b733742 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 245, Committed index: 245, Last appended: 1.245, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } I0703 01:44:12.078059 4966 raft_consensus.cc:1081] T 150033054dcf498d8d71fd996af55e88 P aa8b9c58b466436faa5086e85b5224a7 [term 1 LEARNER]: Refusing update from remote peer 8c1e8943f5654f71b030ec7c0b733742: Log matching property violated. Preceding OpId in replica: term: 1 index: 245. Preceding OpId from leader: term: 1 index: 246. (index mismatch) I0703 01:44:12.078758 3692 raft_consensus.cc:1081] T 150033054dcf498d8d71fd996af55e88 P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 FOLLOWER]: Refusing update from remote peer 8c1e8943f5654f71b030ec7c0b733742: Log matching property violated. Preceding OpId in replica: term: 1 index: 245. Preceding OpId from leader: term: 1 index: 246. (index mismatch) I0703 01:44:12.079599 4744 consensus_queue.cc:962] T 150033054dcf498d8d71fd996af55e88 P 8c1e8943f5654f71b030ec7c0b733742 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 246, Last known committed idx: 245, Time since last communication: 0.000s I0703 01:44:12.080564 5406 raft_consensus.cc:890] T 75f9ea8ba93f442aa064d4623d2c36a1 P 57bea6dc182c41a78304417a3cbfc4a7: attempting to promote NON_VOTER 213596374e4142dca6ecfc701ba1c566 to VOTER I0703 01:44:12.081485 5406 consensus_peers.cc:455] T 75f9ea8ba93f442aa064d4623d2c36a1 P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer 213596374e4142dca6ecfc701ba1c566 (127.5.240.196:51288): Closing peer: 213596374e4142dca6ecfc701ba1c566 I0703 01:44:12.081605 5406 consensus_peers.cc:455] T 75f9ea8ba93f442aa064d4623d2c36a1 P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193:54919): Closing peer: 55223945fe3d4dd8ac03ba9dab346f54 I0703 01:44:12.081846 5406 consensus_queue.cc:204] T 75f9ea8ba93f442aa064d4623d2c36a1 P 57bea6dc182c41a78304417a3cbfc4a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 240, Committed index: 240, Last appended: 1.240, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } } I0703 01:44:12.082767 5154 consensus_queue.cc:962] T 150033054dcf498d8d71fd996af55e88 P 8c1e8943f5654f71b030ec7c0b733742 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 246, Last known committed idx: 245, Time since last communication: 0.000s I0703 01:44:12.083118 4745 raft_consensus.cc:2684] T 150033054dcf498d8d71fd996af55e88 P 8c1e8943f5654f71b030ec7c0b733742 [term 1 LEADER]: Committing config change with OpId 1.246: config changed from index 200 to 246, aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197) changed from NON_VOTER to VOTER. New config: { opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } } I0703 01:44:12.084396 3692 raft_consensus.cc:2684] T 150033054dcf498d8d71fd996af55e88 P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 FOLLOWER]: Committing config change with OpId 1.246: config changed from index 200 to 246, aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197) changed from NON_VOTER to VOTER. New config: { opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } } I0703 01:44:12.085642 4966 raft_consensus.cc:2684] T 150033054dcf498d8d71fd996af55e88 P aa8b9c58b466436faa5086e85b5224a7 [term 1 FOLLOWER]: Committing config change with OpId 1.246: config changed from index 200 to 246, aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197) changed from NON_VOTER to VOTER. New config: { opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } } I0703 01:44:12.087519 3449 catalog_manager.cc:3798] T 150033054dcf498d8d71fd996af55e88 P 8c1e8943f5654f71b030ec7c0b733742 reported cstate change: config changed from index 200 to 246, aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8c1e8943f5654f71b030ec7c0b733742" committed_config { opid_index: 246 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } health_report { overall_health: UNKNOWN } } } I0703 01:44:12.100715 3573 raft_consensus.cc:1081] T 75f9ea8ba93f442aa064d4623d2c36a1 P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 FOLLOWER]: Refusing update from remote peer 57bea6dc182c41a78304417a3cbfc4a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 240. Preceding OpId from leader: term: 1 index: 241. (index mismatch) I0703 01:44:12.103863 4835 raft_consensus.cc:1081] T 75f9ea8ba93f442aa064d4623d2c36a1 P 213596374e4142dca6ecfc701ba1c566 [term 1 LEARNER]: Refusing update from remote peer 57bea6dc182c41a78304417a3cbfc4a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 240. Preceding OpId from leader: term: 1 index: 241. (index mismatch) I0703 01:44:12.110749 5470 consensus_queue.cc:962] T 75f9ea8ba93f442aa064d4623d2c36a1 P 57bea6dc182c41a78304417a3cbfc4a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 241, Last known committed idx: 240, Time since last communication: 0.000s I0703 01:44:12.118917 5470 consensus_queue.cc:962] T 75f9ea8ba93f442aa064d4623d2c36a1 P 57bea6dc182c41a78304417a3cbfc4a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 241, Last known committed idx: 240, Time since last communication: 0.000s I0703 01:44:12.126184 4965 raft_consensus.cc:1026] T c352f70a51154a85a5a5b0f3d14d405f P aa8b9c58b466436faa5086e85b5224a7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.237->[1.238-1.284] Dedup: 1.254->[1.255-1.284] I0703 01:44:12.147907 3570 raft_consensus.cc:2684] T 75f9ea8ba93f442aa064d4623d2c36a1 P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 FOLLOWER]: Committing config change with OpId 1.241: config changed from index 198 to 241, 213596374e4142dca6ecfc701ba1c566 (127.5.240.196) changed from NON_VOTER to VOTER. New config: { opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } } } I0703 01:44:12.137856 5587 raft_consensus.cc:2684] T 75f9ea8ba93f442aa064d4623d2c36a1 P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 LEADER]: Committing config change with OpId 1.241: config changed from index 198 to 241, 213596374e4142dca6ecfc701ba1c566 (127.5.240.196) changed from NON_VOTER to VOTER. New config: { opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } } } I0703 01:44:12.150821 4835 raft_consensus.cc:2684] T 75f9ea8ba93f442aa064d4623d2c36a1 P 213596374e4142dca6ecfc701ba1c566 [term 1 FOLLOWER]: Committing config change with OpId 1.241: config changed from index 198 to 241, 213596374e4142dca6ecfc701ba1c566 (127.5.240.196) changed from NON_VOTER to VOTER. New config: { opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } } } I0703 01:44:12.188380 3453 catalog_manager.cc:3798] T 75f9ea8ba93f442aa064d4623d2c36a1 P 55223945fe3d4dd8ac03ba9dab346f54 reported cstate change: config changed from index 198 to 241, 213596374e4142dca6ecfc701ba1c566 (127.5.240.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "57bea6dc182c41a78304417a3cbfc4a7" committed_config { opid_index: 241 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } } } I0703 01:44:12.191098 3445 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 150033054dcf498d8d71fd996af55e88 (attempt 1) I0703 01:44:12.192690 3815 consensus_peers.cc:455] T 150033054dcf498d8d71fd996af55e88 P 8c1e8943f5654f71b030ec7c0b733742 -> Peer aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197:51329): Closing peer: aa8b9c58b466436faa5086e85b5224a7 I0703 01:44:12.192786 3815 consensus_peers.cc:455] T 150033054dcf498d8d71fd996af55e88 P 8c1e8943f5654f71b030ec7c0b733742 -> Peer 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746): Closing peer: 57bea6dc182c41a78304417a3cbfc4a7 I0703 01:44:12.193095 3815 consensus_queue.cc:204] T 150033054dcf498d8d71fd996af55e88 P 8c1e8943f5654f71b030ec7c0b733742 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 247, Committed index: 247, Last appended: 1.247, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } I0703 01:44:12.194990 4963 raft_consensus.cc:1081] T 150033054dcf498d8d71fd996af55e88 P aa8b9c58b466436faa5086e85b5224a7 [term 1 FOLLOWER]: Refusing update from remote peer 8c1e8943f5654f71b030ec7c0b733742: Log matching property violated. Preceding OpId in replica: term: 1 index: 247. Preceding OpId from leader: term: 1 index: 248. (index mismatch) I0703 01:44:12.195904 5154 consensus_queue.cc:962] T 150033054dcf498d8d71fd996af55e88 P 8c1e8943f5654f71b030ec7c0b733742 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 248, Last known committed idx: 247, Time since last communication: 0.000s I0703 01:44:12.198750 4745 raft_consensus.cc:2684] T 150033054dcf498d8d71fd996af55e88 P 8c1e8943f5654f71b030ec7c0b733742 [term 1 LEADER]: Committing config change with OpId 1.248: config changed from index 246 to 248, VOTER 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194) evicted. New config: { opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } } I0703 01:44:12.200131 4963 raft_consensus.cc:2684] T 150033054dcf498d8d71fd996af55e88 P aa8b9c58b466436faa5086e85b5224a7 [term 1 FOLLOWER]: Committing config change with OpId 1.248: config changed from index 246 to 248, VOTER 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194) evicted. New config: { opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } } I0703 01:44:12.202672 3437 catalog_manager.cc:3352] ChangeConfig:REMOVE_PEER RPC for tablet 150033054dcf498d8d71fd996af55e88 with cas_config_opid_index 246: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0703 01:44:12.203227 3445 catalog_manager.cc:3798] T 150033054dcf498d8d71fd996af55e88 P 8c1e8943f5654f71b030ec7c0b733742 reported cstate change: config changed from index 246 to 248, VOTER 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194) evicted. New cstate: current_term: 1 leader_uuid: "8c1e8943f5654f71b030ec7c0b733742" committed_config { opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } health_report { overall_health: HEALTHY } } } W0703 01:44:12.214902 3507 consensus_peers.cc:433] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 -> Peer 213596374e4142dca6ecfc701ba1c566 (127.5.240.196:51288): Couldn't send request to peer 213596374e4142dca6ecfc701ba1c566 for tablet 60d3228294e945c5b343ae62e76dc2f0. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 4 times. I0703 01:44:12.222806 3446 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 75f9ea8ba93f442aa064d4623d2c36a1 (attempt 1) I0703 01:44:12.224817 3694 consensus_peers.cc:455] T 75f9ea8ba93f442aa064d4623d2c36a1 P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer 213596374e4142dca6ecfc701ba1c566 (127.5.240.196:51288): Closing peer: 213596374e4142dca6ecfc701ba1c566 I0703 01:44:12.224925 3694 consensus_peers.cc:455] T 75f9ea8ba93f442aa064d4623d2c36a1 P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193:54919): Closing peer: 55223945fe3d4dd8ac03ba9dab346f54 I0703 01:44:12.225133 3694 consensus_queue.cc:204] T 75f9ea8ba93f442aa064d4623d2c36a1 P 57bea6dc182c41a78304417a3cbfc4a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 242, Committed index: 242, Last appended: 1.243, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 244 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } } I0703 01:44:12.229537 3445 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 150033054dcf498d8d71fd996af55e88 on 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746) (TS 57bea6dc182c41a78304417a3cbfc4a7 not found in new config with opid_index 248) I0703 01:44:12.230188 3676 tablet_service.cc:800] Processing DeleteTablet for tablet 150033054dcf498d8d71fd996af55e88 with delete_type TABLET_DATA_TOMBSTONED (TS 57bea6dc182c41a78304417a3cbfc4a7 not found in new config with opid_index 248) from {username='jenkins'} at 127.5.240.194:39884 I0703 01:44:12.235775 4834 raft_consensus.cc:1081] T 75f9ea8ba93f442aa064d4623d2c36a1 P 213596374e4142dca6ecfc701ba1c566 [term 1 FOLLOWER]: Refusing update from remote peer 57bea6dc182c41a78304417a3cbfc4a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 242. Preceding OpId from leader: term: 1 index: 244. (index mismatch) I0703 01:44:12.238432 5623 tablet_replica.cc:261] T 150033054dcf498d8d71fd996af55e88 P 57bea6dc182c41a78304417a3cbfc4a7: stopping tablet replica I0703 01:44:12.238765 5623 maintenance_manager.cc:237] P 57bea6dc182c41a78304417a3cbfc4a7: Unregistered op CompactRowSetsOp(150033054dcf498d8d71fd996af55e88) I0703 01:44:12.238898 5623 maintenance_manager.cc:237] P 57bea6dc182c41a78304417a3cbfc4a7: Unregistered op MinorDeltaCompactionOp(150033054dcf498d8d71fd996af55e88) I0703 01:44:12.238991 5623 maintenance_manager.cc:237] P 57bea6dc182c41a78304417a3cbfc4a7: Unregistered op MajorDeltaCompactionOp(150033054dcf498d8d71fd996af55e88) I0703 01:44:12.239084 5623 maintenance_manager.cc:237] P 57bea6dc182c41a78304417a3cbfc4a7: Unregistered op UndoDeltaBlockGCOp(150033054dcf498d8d71fd996af55e88) I0703 01:44:12.239202 5623 maintenance_manager.cc:237] P 57bea6dc182c41a78304417a3cbfc4a7: Unregistered op FlushMRSOp(150033054dcf498d8d71fd996af55e88) I0703 01:44:12.239089 5098 raft_consensus.cc:1026] T eea4e9534e0d4d25a827fae00bbeae4d P 6932f1cd716a48fe8d6c5573acfc48bf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.198->[1.199-1.252] Dedup: 1.209->[1.210-1.252] I0703 01:44:12.239292 5623 maintenance_manager.cc:237] P 57bea6dc182c41a78304417a3cbfc4a7: Unregistered op FlushDeltaMemStoresOp(150033054dcf498d8d71fd996af55e88) I0703 01:44:12.239365 5623 maintenance_manager.cc:237] P 57bea6dc182c41a78304417a3cbfc4a7: Unregistered op LogGCOp(150033054dcf498d8d71fd996af55e88) I0703 01:44:12.239464 5623 raft_consensus.cc:2040] T 150033054dcf498d8d71fd996af55e88 P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 FOLLOWER]: Raft consensus shutting down. I0703 01:44:12.239732 5623 raft_consensus.cc:2067] T 150033054dcf498d8d71fd996af55e88 P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 FOLLOWER]: Raft consensus is shut down! I0703 01:44:12.246222 5587 consensus_queue.cc:962] T 75f9ea8ba93f442aa064d4623d2c36a1 P 57bea6dc182c41a78304417a3cbfc4a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 244, Last known committed idx: 242, Time since last communication: 0.000s I0703 01:44:12.249768 4834 raft_consensus.cc:1026] T 75f9ea8ba93f442aa064d4623d2c36a1 P 213596374e4142dca6ecfc701ba1c566 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.242->[1.243-1.244] Dedup: 1.243->[1.244-1.244] I0703 01:44:12.262177 5543 raft_consensus.cc:2684] T 75f9ea8ba93f442aa064d4623d2c36a1 P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 LEADER]: Committing config change with OpId 1.244: config changed from index 241 to 244, VOTER 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193) evicted. New config: { opid_index: 244 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } } } W0703 01:44:12.267477 3504 consensus_peers.cc:433] T 2051955b4b644a03a9adad816d350596 P 55223945fe3d4dd8ac03ba9dab346f54 -> Peer 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377): Couldn't send request to peer 1157c91cc6f54a68a1d99017ac8444aa for tablet 2051955b4b644a03a9adad816d350596. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 4 times. I0703 01:44:12.282335 5154 raft_consensus.cc:890] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742: attempting to promote NON_VOTER aa8b9c58b466436faa5086e85b5224a7 to VOTER I0703 01:44:12.282671 5154 consensus_peers.cc:455] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 -> Peer aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197:51329): Closing peer: aa8b9c58b466436faa5086e85b5224a7 I0703 01:44:12.282732 5154 consensus_peers.cc:455] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 -> Peer 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193:54919): Closing peer: 55223945fe3d4dd8ac03ba9dab346f54 I0703 01:44:12.283004 5154 consensus_queue.cc:204] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 289, Committed index: 289, Last appended: 1.289, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 290 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } I0703 01:44:12.287302 4965 raft_consensus.cc:1081] T c352f70a51154a85a5a5b0f3d14d405f P aa8b9c58b466436faa5086e85b5224a7 [term 1 LEARNER]: Refusing update from remote peer 8c1e8943f5654f71b030ec7c0b733742: Log matching property violated. Preceding OpId in replica: term: 1 index: 289. Preceding OpId from leader: term: 1 index: 290. (index mismatch) I0703 01:44:12.288013 4745 consensus_queue.cc:962] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 290, Last known committed idx: 289, Time since last communication: 0.000s I0703 01:44:12.290051 3434 catalog_manager.cc:3352] ChangeConfig:REMOVE_PEER RPC for tablet 75f9ea8ba93f442aa064d4623d2c36a1 with cas_config_opid_index 241: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0703 01:44:12.291072 3445 catalog_manager.cc:3798] T 75f9ea8ba93f442aa064d4623d2c36a1 P 57bea6dc182c41a78304417a3cbfc4a7 reported cstate change: config changed from index 241 to 244, VOTER 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193) evicted. New cstate: current_term: 1 leader_uuid: "57bea6dc182c41a78304417a3cbfc4a7" committed_config { opid_index: 244 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } health_report { overall_health: HEALTHY } } } W0703 01:44:12.291988 3630 consensus_peers.cc:433] T 930bbe4ac3234ed49a2207c4902541f1 P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer 213596374e4142dca6ecfc701ba1c566 (127.5.240.196:51288): Couldn't send request to peer 213596374e4142dca6ecfc701ba1c566 for tablet 930bbe4ac3234ed49a2207c4902541f1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 5 times. I0703 01:44:12.292793 4835 raft_consensus.cc:2684] T 75f9ea8ba93f442aa064d4623d2c36a1 P 213596374e4142dca6ecfc701ba1c566 [term 1 FOLLOWER]: Committing config change with OpId 1.244: config changed from index 241 to 244, VOTER 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193) evicted. New config: { opid_index: 244 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } } } I0703 01:44:12.293928 3445 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 75f9ea8ba93f442aa064d4623d2c36a1 on 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193:54919) (TS 55223945fe3d4dd8ac03ba9dab346f54 not found in new config with opid_index 244) I0703 01:44:12.294467 5623 ts_tablet_manager.cc:1369] T 150033054dcf498d8d71fd996af55e88 P 57bea6dc182c41a78304417a3cbfc4a7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:44:12.294698 3553 tablet_service.cc:800] Processing DeleteTablet for tablet 75f9ea8ba93f442aa064d4623d2c36a1 with delete_type TABLET_DATA_TOMBSTONED (TS 55223945fe3d4dd8ac03ba9dab346f54 not found in new config with opid_index 244) from {username='jenkins'} at 127.5.240.193:52692 I0703 01:44:12.299258 3570 raft_consensus.cc:1081] T c352f70a51154a85a5a5b0f3d14d405f P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 FOLLOWER]: Refusing update from remote peer 8c1e8943f5654f71b030ec7c0b733742: Log matching property violated. Preceding OpId in replica: term: 1 index: 289. Preceding OpId from leader: term: 1 index: 290. (index mismatch) I0703 01:44:12.302620 4745 consensus_queue.cc:962] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 290, Last known committed idx: 289, Time since last communication: 0.002s I0703 01:44:12.303267 4745 raft_consensus.cc:2684] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 [term 1 LEADER]: Committing config change with OpId 1.290: config changed from index 238 to 290, aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197) changed from NON_VOTER to VOTER. New config: { opid_index: 290 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } } I0703 01:44:12.305732 4966 raft_consensus.cc:2684] T c352f70a51154a85a5a5b0f3d14d405f P aa8b9c58b466436faa5086e85b5224a7 [term 1 FOLLOWER]: Committing config change with OpId 1.290: config changed from index 238 to 290, aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197) changed from NON_VOTER to VOTER. New config: { opid_index: 290 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } } I0703 01:44:12.310374 3445 catalog_manager.cc:3798] T c352f70a51154a85a5a5b0f3d14d405f P aa8b9c58b466436faa5086e85b5224a7 reported cstate change: config changed from index 238 to 290, aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8c1e8943f5654f71b030ec7c0b733742" committed_config { opid_index: 290 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } } I0703 01:44:12.313325 3448 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet c352f70a51154a85a5a5b0f3d14d405f (attempt 1) I0703 01:44:12.314079 3816 consensus_peers.cc:455] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 -> Peer aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197:51329): Closing peer: aa8b9c58b466436faa5086e85b5224a7 I0703 01:44:12.314184 3816 consensus_peers.cc:455] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 -> Peer 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193:54919): Closing peer: 55223945fe3d4dd8ac03ba9dab346f54 I0703 01:44:12.314429 3816 consensus_queue.cc:204] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 290, Committed index: 290, Last appended: 1.290, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 291 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } I0703 01:44:12.319020 4966 raft_consensus.cc:1081] T c352f70a51154a85a5a5b0f3d14d405f P aa8b9c58b466436faa5086e85b5224a7 [term 1 FOLLOWER]: Refusing update from remote peer 8c1e8943f5654f71b030ec7c0b733742: Log matching property violated. Preceding OpId in replica: term: 1 index: 290. Preceding OpId from leader: term: 1 index: 291. (index mismatch) I0703 01:44:12.319695 4744 consensus_queue.cc:962] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 291, Last known committed idx: 290, Time since last communication: 0.000s I0703 01:44:12.323245 3570 raft_consensus.cc:2684] T c352f70a51154a85a5a5b0f3d14d405f P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 FOLLOWER]: Committing config change with OpId 1.290: config changed from index 238 to 290, aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197) changed from NON_VOTER to VOTER. New config: { opid_index: 290 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } } I0703 01:44:12.327435 5623 ts_tablet_manager.cc:1382] T 150033054dcf498d8d71fd996af55e88 P 57bea6dc182c41a78304417a3cbfc4a7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.247 I0703 01:44:12.328126 5623 log.cc:986] T 150033054dcf498d8d71fd996af55e88 P 57bea6dc182c41a78304417a3cbfc4a7: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal/wals/150033054dcf498d8d71fd996af55e88 I0703 01:44:12.329812 3434 catalog_manager.cc:3103] TS 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746): tablet 150033054dcf498d8d71fd996af55e88 (table rebalance_test_table_2 [id=24e8eff3fcb34269ba893ac2456e5712]) successfully deleted I0703 01:44:12.335367 5025 raft_consensus.cc:2684] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 [term 1 LEADER]: Committing config change with OpId 1.291: config changed from index 290 to 291, VOTER 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193) evicted. New config: { opid_index: 291 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } } I0703 01:44:12.341168 4966 raft_consensus.cc:2684] T c352f70a51154a85a5a5b0f3d14d405f P aa8b9c58b466436faa5086e85b5224a7 [term 1 FOLLOWER]: Committing config change with OpId 1.291: config changed from index 290 to 291, VOTER 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193) evicted. New config: { opid_index: 291 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } } I0703 01:44:12.341874 5625 tablet_replica.cc:261] T 75f9ea8ba93f442aa064d4623d2c36a1 P 55223945fe3d4dd8ac03ba9dab346f54: stopping tablet replica I0703 01:44:12.342078 5625 maintenance_manager.cc:237] P 55223945fe3d4dd8ac03ba9dab346f54: Unregistered op CompactRowSetsOp(75f9ea8ba93f442aa064d4623d2c36a1) I0703 01:44:12.342193 5625 maintenance_manager.cc:237] P 55223945fe3d4dd8ac03ba9dab346f54: Unregistered op MinorDeltaCompactionOp(75f9ea8ba93f442aa064d4623d2c36a1) I0703 01:44:12.342273 5625 maintenance_manager.cc:237] P 55223945fe3d4dd8ac03ba9dab346f54: Unregistered op MajorDeltaCompactionOp(75f9ea8ba93f442aa064d4623d2c36a1) I0703 01:44:12.342367 5625 maintenance_manager.cc:237] P 55223945fe3d4dd8ac03ba9dab346f54: Unregistered op UndoDeltaBlockGCOp(75f9ea8ba93f442aa064d4623d2c36a1) I0703 01:44:12.342483 5625 maintenance_manager.cc:237] P 55223945fe3d4dd8ac03ba9dab346f54: Unregistered op FlushMRSOp(75f9ea8ba93f442aa064d4623d2c36a1) I0703 01:44:12.342561 5625 maintenance_manager.cc:237] P 55223945fe3d4dd8ac03ba9dab346f54: Unregistered op FlushDeltaMemStoresOp(75f9ea8ba93f442aa064d4623d2c36a1) I0703 01:44:12.342651 5625 maintenance_manager.cc:237] P 55223945fe3d4dd8ac03ba9dab346f54: Unregistered op LogGCOp(75f9ea8ba93f442aa064d4623d2c36a1) I0703 01:44:12.342743 5625 raft_consensus.cc:2040] T 75f9ea8ba93f442aa064d4623d2c36a1 P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 FOLLOWER]: Raft consensus shutting down. I0703 01:44:12.342952 5625 pending_rounds.cc:63] T 75f9ea8ba93f442aa064d4623d2c36a1 P 55223945fe3d4dd8ac03ba9dab346f54: Trying to abort 1 pending transactions. I0703 01:44:12.343019 5625 pending_rounds.cc:68] T 75f9ea8ba93f442aa064d4623d2c36a1 P 55223945fe3d4dd8ac03ba9dab346f54: Aborting transaction as it isn't in flight: id { term: 1 index: 243 } timestamp: 6269368124306460672 op_type: WRITE_OP write_request { tablet_id: "75f9ea8ba93f442aa064d4623d2c36a1" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true } } row_operations { rows: "\001\007\000vK\332n\206\227\322\323\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "9a4fcf62db70b280" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 6269368124204179456 } request_id { client_id: "956c0f4d460f4179a7ddc12c0187a34a" seq_no: 754 first_incomplete_seq_no: 754 attempt_no: 0 } I0703 01:44:12.343617 5625 raft_consensus.cc:2067] T 75f9ea8ba93f442aa064d4623d2c36a1 P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 FOLLOWER]: Raft consensus is shut down! I0703 01:44:12.345155 3448 catalog_manager.cc:3798] T c352f70a51154a85a5a5b0f3d14d405f P aa8b9c58b466436faa5086e85b5224a7 reported cstate change: config changed from index 290 to 291, VOTER 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193) evicted. New cstate: current_term: 1 leader_uuid: "8c1e8943f5654f71b030ec7c0b733742" committed_config { opid_index: 291 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } } I0703 01:44:12.345764 5625 ts_tablet_manager.cc:1369] T 75f9ea8ba93f442aa064d4623d2c36a1 P 55223945fe3d4dd8ac03ba9dab346f54: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:44:12.347484 3448 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet c352f70a51154a85a5a5b0f3d14d405f on 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193:54919) (TS 55223945fe3d4dd8ac03ba9dab346f54 not found in new config with opid_index 291) I0703 01:44:12.352443 3437 catalog_manager.cc:3352] ChangeConfig:REMOVE_PEER RPC for tablet c352f70a51154a85a5a5b0f3d14d405f with cas_config_opid_index 290: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0703 01:44:12.348170 3553 tablet_service.cc:800] Processing DeleteTablet for tablet c352f70a51154a85a5a5b0f3d14d405f with delete_type TABLET_DATA_TOMBSTONED (TS 55223945fe3d4dd8ac03ba9dab346f54 not found in new config with opid_index 291) from {username='jenkins'} at 127.5.240.193:52692 I0703 01:44:12.362468 5625 ts_tablet_manager.cc:1382] T 75f9ea8ba93f442aa064d4623d2c36a1 P 55223945fe3d4dd8ac03ba9dab346f54: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.243 I0703 01:44:12.367769 3448 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet c352f70a51154a85a5a5b0f3d14d405f on 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193:54919) (Replica with old config index 290 (current committed config index is 291)) I0703 01:44:12.368387 3553 tablet_service.cc:800] Processing DeleteTablet for tablet c352f70a51154a85a5a5b0f3d14d405f with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 290 (current committed config index is 291)) from {username='jenkins'} at 127.5.240.193:52692 I0703 01:44:12.373139 5625 log.cc:986] T 75f9ea8ba93f442aa064d4623d2c36a1 P 55223945fe3d4dd8ac03ba9dab346f54: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal/wals/75f9ea8ba93f442aa064d4623d2c36a1 I0703 01:44:12.373944 3437 catalog_manager.cc:3103] TS 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193:54919): tablet 75f9ea8ba93f442aa064d4623d2c36a1 (table rebalance_test_table_0 [id=5bc38732cb8746a7ad1c2b9fede5e7d9]) successfully deleted I0703 01:44:12.377589 5625 tablet_replica.cc:261] T c352f70a51154a85a5a5b0f3d14d405f P 55223945fe3d4dd8ac03ba9dab346f54: stopping tablet replica I0703 01:44:12.377974 5625 maintenance_manager.cc:237] P 55223945fe3d4dd8ac03ba9dab346f54: Unregistered op CompactRowSetsOp(c352f70a51154a85a5a5b0f3d14d405f) I0703 01:44:12.378078 5625 maintenance_manager.cc:237] P 55223945fe3d4dd8ac03ba9dab346f54: Unregistered op MinorDeltaCompactionOp(c352f70a51154a85a5a5b0f3d14d405f) I0703 01:44:12.378176 5625 maintenance_manager.cc:237] P 55223945fe3d4dd8ac03ba9dab346f54: Unregistered op MajorDeltaCompactionOp(c352f70a51154a85a5a5b0f3d14d405f) I0703 01:44:12.381564 5464 tablet_bootstrap.cc:427] T 6c8253ac9a9a405885f1f57c59a84fd5 P aa8b9c58b466436faa5086e85b5224a7: Bootstrap replayed 1/1 log segments. Stats: ops{read=244 overwritten=0 applied=244 ignored=0} inserts{seen=893 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:44:12.381922 5464 tablet_bootstrap.cc:427] T 6c8253ac9a9a405885f1f57c59a84fd5 P aa8b9c58b466436faa5086e85b5224a7: Bootstrap complete. I0703 01:44:12.382127 5464 ts_tablet_manager.cc:1045] T 6c8253ac9a9a405885f1f57c59a84fd5 P aa8b9c58b466436faa5086e85b5224a7: Time spent bootstrapping tablet: real 2.457s user 0.318s sys 0.016s I0703 01:44:12.385666 5625 maintenance_manager.cc:237] P 55223945fe3d4dd8ac03ba9dab346f54: Unregistered op UndoDeltaBlockGCOp(c352f70a51154a85a5a5b0f3d14d405f) I0703 01:44:12.385831 5625 maintenance_manager.cc:237] P 55223945fe3d4dd8ac03ba9dab346f54: Unregistered op FlushMRSOp(c352f70a51154a85a5a5b0f3d14d405f) I0703 01:44:12.386082 5625 maintenance_manager.cc:237] P 55223945fe3d4dd8ac03ba9dab346f54: Unregistered op FlushDeltaMemStoresOp(c352f70a51154a85a5a5b0f3d14d405f) I0703 01:44:12.386170 5625 maintenance_manager.cc:237] P 55223945fe3d4dd8ac03ba9dab346f54: Unregistered op LogGCOp(c352f70a51154a85a5a5b0f3d14d405f) I0703 01:44:12.386272 5625 raft_consensus.cc:2040] T c352f70a51154a85a5a5b0f3d14d405f P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 FOLLOWER]: Raft consensus shutting down. I0703 01:44:12.386473 5625 raft_consensus.cc:2067] T c352f70a51154a85a5a5b0f3d14d405f P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 FOLLOWER]: Raft consensus is shut down! I0703 01:44:12.388180 5625 ts_tablet_manager.cc:1369] T c352f70a51154a85a5a5b0f3d14d405f P 55223945fe3d4dd8ac03ba9dab346f54: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED W0703 01:44:12.425544 3749 consensus_peers.cc:433] T 25b33ca4b32a4a2493d03564bfe1bf7e P 8c1e8943f5654f71b030ec7c0b733742 -> Peer 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377): Couldn't send request to peer 1157c91cc6f54a68a1d99017ac8444aa for tablet 25b33ca4b32a4a2493d03564bfe1bf7e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 7 times. I0703 01:44:12.448226 5625 ts_tablet_manager.cc:1382] T c352f70a51154a85a5a5b0f3d14d405f P 55223945fe3d4dd8ac03ba9dab346f54: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.290 I0703 01:44:12.448654 5625 log.cc:986] T c352f70a51154a85a5a5b0f3d14d405f P 55223945fe3d4dd8ac03ba9dab346f54: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal/wals/c352f70a51154a85a5a5b0f3d14d405f I0703 01:44:12.449370 3437 catalog_manager.cc:3103] TS 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193:54919): tablet c352f70a51154a85a5a5b0f3d14d405f (table rebalance_test_table_1 [id=29732119416840ec89c7a3633f79c5fd]) successfully deleted I0703 01:44:12.448856 5464 raft_consensus.cc:325] T 6c8253ac9a9a405885f1f57c59a84fd5 P aa8b9c58b466436faa5086e85b5224a7 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: NON_VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: true } } I0703 01:44:12.449823 3437 catalog_manager.cc:3103] TS 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193:54919): tablet c352f70a51154a85a5a5b0f3d14d405f (table rebalance_test_table_1 [id=29732119416840ec89c7a3633f79c5fd]) successfully deleted W0703 01:44:12.486269 3749 consensus_peers.cc:433] T 674ed5a88db04825a4eaec2fa996a852 P 8c1e8943f5654f71b030ec7c0b733742 -> Peer 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377): Couldn't send request to peer 1157c91cc6f54a68a1d99017ac8444aa for tablet 674ed5a88db04825a4eaec2fa996a852. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 7 times. W0703 01:44:12.492072 3627 consensus_peers.cc:433] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197:51329): Couldn't send request to peer aa8b9c58b466436faa5086e85b5224a7 for tablet 049bd2ef267f47fca634858420ac35b9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 5 times. I0703 01:44:12.502656 5464 raft_consensus.cc:628] T 6c8253ac9a9a405885f1f57c59a84fd5 P aa8b9c58b466436faa5086e85b5224a7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aa8b9c58b466436faa5086e85b5224a7, State: Initialized, Role: LEARNER I0703 01:44:12.502821 5464 consensus_queue.cc:227] T 6c8253ac9a9a405885f1f57c59a84fd5 P aa8b9c58b466436faa5086e85b5224a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 244, Last appended: 1.244, Last appended by leader: 244, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 232 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: NON_VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: true } } I0703 01:44:12.503659 5464 ts_tablet_manager.cc:1073] T 6c8253ac9a9a405885f1f57c59a84fd5 P aa8b9c58b466436faa5086e85b5224a7: Time spent starting tablet: real 0.121s user 0.002s sys 0.000s I0703 01:44:12.506251 3593 tablet_copy_service.cc:325] P 55223945fe3d4dd8ac03ba9dab346f54: Request end of tablet copy session aa8b9c58b466436faa5086e85b5224a7-6c8253ac9a9a405885f1f57c59a84fd5 received from {username='jenkins'} at 127.5.240.197:34012 I0703 01:44:12.506335 3593 tablet_copy_service.cc:413] P 55223945fe3d4dd8ac03ba9dab346f54: ending tablet copy session aa8b9c58b466436faa5086e85b5224a7-6c8253ac9a9a405885f1f57c59a84fd5 on tablet 6c8253ac9a9a405885f1f57c59a84fd5 with peer aa8b9c58b466436faa5086e85b5224a7 W0703 01:44:12.511271 5429 log.cc:667] Time spent T 25b33ca4b32a4a2493d03564bfe1bf7e P 1157c91cc6f54a68a1d99017ac8444aa: Append to log took a long time: real 0.063s user 0.001s sys 0.000s I0703 01:44:12.511904 5543 raft_consensus.cc:890] T eea4e9534e0d4d25a827fae00bbeae4d P 57bea6dc182c41a78304417a3cbfc4a7: attempting to promote NON_VOTER 6932f1cd716a48fe8d6c5573acfc48bf to VOTER I0703 01:44:12.512331 5543 consensus_peers.cc:455] T eea4e9534e0d4d25a827fae00bbeae4d P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198:51033): Closing peer: 6932f1cd716a48fe8d6c5573acfc48bf I0703 01:44:12.512441 5543 consensus_peers.cc:455] T eea4e9534e0d4d25a827fae00bbeae4d P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064): Closing peer: 8c1e8943f5654f71b030ec7c0b733742 I0703 01:44:12.512796 5543 consensus_queue.cc:204] T eea4e9534e0d4d25a827fae00bbeae4d P 57bea6dc182c41a78304417a3cbfc4a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 253, Committed index: 253, Last appended: 1.253, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false } } I0703 01:44:12.516713 5098 raft_consensus.cc:1081] T eea4e9534e0d4d25a827fae00bbeae4d P 6932f1cd716a48fe8d6c5573acfc48bf [term 1 LEARNER]: Refusing update from remote peer 57bea6dc182c41a78304417a3cbfc4a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 253. Preceding OpId from leader: term: 1 index: 254. (index mismatch) I0703 01:44:12.517179 3816 raft_consensus.cc:1081] T eea4e9534e0d4d25a827fae00bbeae4d P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Refusing update from remote peer 57bea6dc182c41a78304417a3cbfc4a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 253. Preceding OpId from leader: term: 1 index: 254. (index mismatch) I0703 01:44:12.522735 5543 consensus_queue.cc:962] T eea4e9534e0d4d25a827fae00bbeae4d P 57bea6dc182c41a78304417a3cbfc4a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 254, Last known committed idx: 253, Time since last communication: 0.000s I0703 01:44:12.528831 4965 raft_consensus.cc:1026] T 6c8253ac9a9a405885f1f57c59a84fd5 P aa8b9c58b466436faa5086e85b5224a7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.231->[1.232-1.281] Dedup: 1.244->[1.245-1.281] I0703 01:44:12.530052 5025 raft_consensus.cc:457] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1157c91cc6f54a68a1d99017ac8444aa) I0703 01:44:12.530136 5025 raft_consensus.cc:2730] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742: Snoozing failure detection for 1.670s (starting election) I0703 01:44:12.530226 5025 raft_consensus.cc:479] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { replace: true } } peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } } I0703 01:44:12.530761 5025 leader_election.cc:243] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377) I0703 01:44:12.530912 5025 leader_election.cc:243] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 213596374e4142dca6ecfc701ba1c566 (127.5.240.196:51288) I0703 01:44:12.534570 5543 consensus_queue.cc:962] T eea4e9534e0d4d25a827fae00bbeae4d P 57bea6dc182c41a78304417a3cbfc4a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 254, Last known committed idx: 253, Time since last communication: 0.000s I0703 01:44:12.535563 5470 raft_consensus.cc:2684] T eea4e9534e0d4d25a827fae00bbeae4d P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 LEADER]: Committing config change with OpId 1.254: config changed from index 199 to 254, 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198) changed from NON_VOTER to VOTER. New config: { opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false } } } I0703 01:44:12.539831 3448 catalog_manager.cc:3798] T eea4e9534e0d4d25a827fae00bbeae4d P 57bea6dc182c41a78304417a3cbfc4a7 reported cstate change: config changed from index 199 to 254, 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "57bea6dc182c41a78304417a3cbfc4a7" committed_config { opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I0703 01:44:12.542870 5098 raft_consensus.cc:2684] T eea4e9534e0d4d25a827fae00bbeae4d P 6932f1cd716a48fe8d6c5573acfc48bf [term 1 FOLLOWER]: Committing config change with OpId 1.254: config changed from index 199 to 254, 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198) changed from NON_VOTER to VOTER. New config: { opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false } } } I0703 01:44:12.543738 3448 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet eea4e9534e0d4d25a827fae00bbeae4d (attempt 1) I0703 01:44:12.546175 3448 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet eea4e9534e0d4d25a827fae00bbeae4d (attempt 1) I0703 01:44:12.547663 3696 consensus_peers.cc:455] T eea4e9534e0d4d25a827fae00bbeae4d P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198:51033): Closing peer: 6932f1cd716a48fe8d6c5573acfc48bf I0703 01:44:12.547765 3696 consensus_peers.cc:455] T eea4e9534e0d4d25a827fae00bbeae4d P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064): Closing peer: 8c1e8943f5654f71b030ec7c0b733742 I0703 01:44:12.547926 3696 consensus_queue.cc:204] T eea4e9534e0d4d25a827fae00bbeae4d P 57bea6dc182c41a78304417a3cbfc4a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 254, Committed index: 254, Last appended: 1.254, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 255 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false } } I0703 01:44:12.555721 3434 catalog_manager.cc:3369] ChangeConfig:REMOVE_PEER RPC for tablet eea4e9534e0d4d25a827fae00bbeae4d with cas_config_opid_index 254: ChangeConfig:REMOVE_PEER failed with leader 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time. Committed config: opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false } }. Pending config: opid_index: 255 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false } } I0703 01:44:12.555824 3434 catalog_manager.cc:2903] Scheduling retry of ChangeConfig:REMOVE_PEER RPC for tablet eea4e9534e0d4d25a827fae00bbeae4d with cas_config_opid_index 254 with a delay of 32 ms (attempt = 1) I0703 01:44:12.561633 5098 raft_consensus.cc:1081] T eea4e9534e0d4d25a827fae00bbeae4d P 6932f1cd716a48fe8d6c5573acfc48bf [term 1 FOLLOWER]: Refusing update from remote peer 57bea6dc182c41a78304417a3cbfc4a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 254. Preceding OpId from leader: term: 1 index: 255. (index mismatch) I0703 01:44:12.565860 5235 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "dc02a796568c4677a2eebfacae32fc38" candidate_uuid: "8c1e8943f5654f71b030ec7c0b733742" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1157c91cc6f54a68a1d99017ac8444aa" is_pre_election: true I0703 01:44:12.566109 5235 raft_consensus.cc:2730] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa: Snoozing failure detection for 2.224s (vote granted) I0703 01:44:12.566195 5235 raft_consensus.cc:2243] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c1e8943f5654f71b030ec7c0b733742 in term 2. I0703 01:44:12.572573 3749 leader_election.cc:379] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [CANDIDATE]: Term 3 pre-election: Vote granted by peer 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377) I0703 01:44:12.572688 3749 leader_election.cc:270] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. I0703 01:44:12.573734 4745 raft_consensus.cc:2730] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742: Snoozing failure detection for 1.719s (election complete) I0703 01:44:12.573809 4745 raft_consensus.cc:2537] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [term 2 FOLLOWER]: Leader pre-election won for term 3 I0703 01:44:12.573854 4745 raft_consensus.cc:457] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1157c91cc6f54a68a1d99017ac8444aa) I0703 01:44:12.573906 4745 raft_consensus.cc:2730] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742: Snoozing failure detection for 2.206s (starting election) I0703 01:44:12.573951 4745 raft_consensus.cc:2782] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [term 2 FOLLOWER]: Advancing to term 3 I0703 01:44:12.575956 4745 raft_consensus.cc:479] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { replace: true } } peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } } I0703 01:44:12.576506 4745 leader_election.cc:243] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [CANDIDATE]: Term 3 election: Requesting vote from peer 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377) I0703 01:44:12.576710 4745 leader_election.cc:243] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [CANDIDATE]: Term 3 election: Requesting vote from peer 213596374e4142dca6ecfc701ba1c566 (127.5.240.196:51288) I0703 01:44:12.580152 5235 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "dc02a796568c4677a2eebfacae32fc38" candidate_uuid: "8c1e8943f5654f71b030ec7c0b733742" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1157c91cc6f54a68a1d99017ac8444aa" I0703 01:44:12.580307 5235 raft_consensus.cc:2782] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa [term 2 FOLLOWER]: Advancing to term 3 I0703 01:44:12.580415 5235 raft_consensus.cc:2730] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa: Snoozing failure detection for 2.018s (vote granted) I0703 01:44:12.580565 5470 consensus_queue.cc:962] T eea4e9534e0d4d25a827fae00bbeae4d P 57bea6dc182c41a78304417a3cbfc4a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 255, Last known committed idx: 254, Time since last communication: 0.000s W0703 01:44:12.582445 3630 consensus_peers.cc:433] T 4a613c1be55c4888a4df36d3f1bfc54c P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198:51033): Couldn't send request to peer 6932f1cd716a48fe8d6c5573acfc48bf for tablet 4a613c1be55c4888a4df36d3f1bfc54c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 7 times. I0703 01:44:12.582563 5235 raft_consensus.cc:2243] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c1e8943f5654f71b030ec7c0b733742 in term 3. W0703 01:44:12.587600 3507 consensus_peers.cc:433] T eada0e8b8608424594cabd35cf9891e3 P 55223945fe3d4dd8ac03ba9dab346f54 -> Peer 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198:51033): Couldn't send request to peer 6932f1cd716a48fe8d6c5573acfc48bf for tablet eada0e8b8608424594cabd35cf9891e3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 6 times. I0703 01:44:12.588456 3434 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet eea4e9534e0d4d25a827fae00bbeae4d (attempt 2) I0703 01:44:12.590921 3434 catalog_manager.cc:3369] ChangeConfig:REMOVE_PEER RPC for tablet eea4e9534e0d4d25a827fae00bbeae4d with cas_config_opid_index 254: ChangeConfig:REMOVE_PEER failed with leader 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time. Committed config: opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false } }. Pending config: opid_index: 255 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false } } I0703 01:44:12.591032 3434 catalog_manager.cc:2903] Scheduling retry of ChangeConfig:REMOVE_PEER RPC for tablet eea4e9534e0d4d25a827fae00bbeae4d with cas_config_opid_index 254 with a delay of 79 ms (attempt = 2) I0703 01:44:12.598147 4835 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "dc02a796568c4677a2eebfacae32fc38" candidate_uuid: "8c1e8943f5654f71b030ec7c0b733742" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "213596374e4142dca6ecfc701ba1c566" is_pre_election: true I0703 01:44:12.598424 4835 raft_consensus.cc:2204] T dc02a796568c4677a2eebfacae32fc38 P 213596374e4142dca6ecfc701ba1c566 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c1e8943f5654f71b030ec7c0b733742 for term 3 because replica is either leader or believes a valid leader to be alive. I0703 01:44:12.597757 4833 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "dc02a796568c4677a2eebfacae32fc38" candidate_uuid: "8c1e8943f5654f71b030ec7c0b733742" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "213596374e4142dca6ecfc701ba1c566" I0703 01:44:12.599331 4833 raft_consensus.cc:2204] T dc02a796568c4677a2eebfacae32fc38 P 213596374e4142dca6ecfc701ba1c566 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c1e8943f5654f71b030ec7c0b733742 for term 3 because replica is either leader or believes a valid leader to be alive. I0703 01:44:12.599551 5406 raft_consensus.cc:2684] T eea4e9534e0d4d25a827fae00bbeae4d P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 LEADER]: Committing config change with OpId 1.255: config changed from index 254 to 255, VOTER 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195) evicted. New config: { opid_index: 255 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false } } } I0703 01:44:12.602517 3752 leader_election.cc:393] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 213596374e4142dca6ecfc701ba1c566 (127.5.240.196:51288). Message: Invalid argument: T dc02a796568c4677a2eebfacae32fc38 P 213596374e4142dca6ecfc701ba1c566 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c1e8943f5654f71b030ec7c0b733742 for term 3 because replica is either leader or believes a valid leader to be alive. I0703 01:44:12.602629 3434 catalog_manager.cc:3352] ChangeConfig:REMOVE_PEER RPC for tablet eea4e9534e0d4d25a827fae00bbeae4d with cas_config_opid_index 254: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0703 01:44:12.603603 5097 raft_consensus.cc:2684] T eea4e9534e0d4d25a827fae00bbeae4d P 6932f1cd716a48fe8d6c5573acfc48bf [term 1 FOLLOWER]: Committing config change with OpId 1.255: config changed from index 254 to 255, VOTER 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195) evicted. New config: { opid_index: 255 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false } } } I0703 01:44:12.605038 3448 catalog_manager.cc:3798] T eea4e9534e0d4d25a827fae00bbeae4d P 57bea6dc182c41a78304417a3cbfc4a7 reported cstate change: config changed from index 254 to 255, VOTER 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195) evicted. New cstate: current_term: 1 leader_uuid: "57bea6dc182c41a78304417a3cbfc4a7" committed_config { opid_index: 255 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I0703 01:44:12.607583 3448 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet eea4e9534e0d4d25a827fae00bbeae4d on 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064) (TS 8c1e8943f5654f71b030ec7c0b733742 not found in new config with opid_index 255) I0703 01:44:12.608299 3796 tablet_service.cc:800] Processing DeleteTablet for tablet eea4e9534e0d4d25a827fae00bbeae4d with delete_type TABLET_DATA_TOMBSTONED (TS 8c1e8943f5654f71b030ec7c0b733742 not found in new config with opid_index 255) from {username='jenkins'} at 127.5.240.195:39619 I0703 01:44:12.613730 3752 leader_election.cc:393] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [CANDIDATE]: Term 3 election: Vote denied by peer 213596374e4142dca6ecfc701ba1c566 (127.5.240.196:51288). Message: Invalid argument: T dc02a796568c4677a2eebfacae32fc38 P 213596374e4142dca6ecfc701ba1c566 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c1e8943f5654f71b030ec7c0b733742 for term 3 because replica is either leader or believes a valid leader to be alive. I0703 01:44:12.618187 5641 tablet_replica.cc:261] T eea4e9534e0d4d25a827fae00bbeae4d P 8c1e8943f5654f71b030ec7c0b733742: stopping tablet replica I0703 01:44:12.623090 5641 maintenance_manager.cc:237] P 8c1e8943f5654f71b030ec7c0b733742: Unregistered op CompactRowSetsOp(eea4e9534e0d4d25a827fae00bbeae4d) I0703 01:44:12.623214 5641 maintenance_manager.cc:237] P 8c1e8943f5654f71b030ec7c0b733742: Unregistered op MinorDeltaCompactionOp(eea4e9534e0d4d25a827fae00bbeae4d) I0703 01:44:12.623301 5641 maintenance_manager.cc:237] P 8c1e8943f5654f71b030ec7c0b733742: Unregistered op MajorDeltaCompactionOp(eea4e9534e0d4d25a827fae00bbeae4d) I0703 01:44:12.623394 5641 maintenance_manager.cc:237] P 8c1e8943f5654f71b030ec7c0b733742: Unregistered op UndoDeltaBlockGCOp(eea4e9534e0d4d25a827fae00bbeae4d) I0703 01:44:12.623594 5641 maintenance_manager.cc:237] P 8c1e8943f5654f71b030ec7c0b733742: Unregistered op FlushMRSOp(eea4e9534e0d4d25a827fae00bbeae4d) I0703 01:44:12.623682 5641 maintenance_manager.cc:237] P 8c1e8943f5654f71b030ec7c0b733742: Unregistered op FlushDeltaMemStoresOp(eea4e9534e0d4d25a827fae00bbeae4d) I0703 01:44:12.623769 5641 maintenance_manager.cc:237] P 8c1e8943f5654f71b030ec7c0b733742: Unregistered op LogGCOp(eea4e9534e0d4d25a827fae00bbeae4d) I0703 01:44:12.623872 5641 raft_consensus.cc:2040] T eea4e9534e0d4d25a827fae00bbeae4d P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Raft consensus shutting down. I0703 01:44:12.624097 5641 pending_rounds.cc:63] T eea4e9534e0d4d25a827fae00bbeae4d P 8c1e8943f5654f71b030ec7c0b733742: Trying to abort 1 pending transactions. I0703 01:44:12.624169 5641 pending_rounds.cc:68] T eea4e9534e0d4d25a827fae00bbeae4d P 8c1e8943f5654f71b030ec7c0b733742: Aborting transaction as it isn't in flight: id { term: 1 index: 254 } timestamp: 6269368125489897472 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "eea4e9534e0d4d25a827fae00bbeae4d" old_config { opid_index: 199 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: NON_VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: true } } } new_config { opid_index: 254 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false } } } } I0703 01:44:12.624956 5641 raft_consensus.cc:2646] T eea4e9534e0d4d25a827fae00bbeae4d P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Aborting config change with OpId 1.254: Aborted: Transaction aborted I0703 01:44:12.626699 5641 raft_consensus.cc:2620] T eea4e9534e0d4d25a827fae00bbeae4d P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Transaction aborted I0703 01:44:12.626813 5641 raft_consensus.cc:2067] T eea4e9534e0d4d25a827fae00bbeae4d P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Raft consensus is shut down! I0703 01:44:12.629156 5641 ts_tablet_manager.cc:1369] T eea4e9534e0d4d25a827fae00bbeae4d P 8c1e8943f5654f71b030ec7c0b733742: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:44:12.632964 3749 leader_election.cc:379] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [CANDIDATE]: Term 3 election: Vote granted by peer 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377) I0703 01:44:12.633064 3749 leader_election.cc:270] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. I0703 01:44:12.635740 5429 tablet_bootstrap.cc:427] T 25b33ca4b32a4a2493d03564bfe1bf7e P 1157c91cc6f54a68a1d99017ac8444aa: Bootstrap replayed 1/1 log segments. Stats: ops{read=227 overwritten=0 applied=227 ignored=0} inserts{seen=914 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:44:12.636668 5429 tablet_bootstrap.cc:427] T 25b33ca4b32a4a2493d03564bfe1bf7e P 1157c91cc6f54a68a1d99017ac8444aa: Bootstrap complete. I0703 01:44:12.636911 5429 ts_tablet_manager.cc:1045] T 25b33ca4b32a4a2493d03564bfe1bf7e P 1157c91cc6f54a68a1d99017ac8444aa: Time spent bootstrapping tablet: real 3.318s user 0.310s sys 0.017s I0703 01:44:12.638681 4745 raft_consensus.cc:2730] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742: Snoozing failure detection for 1.693s (election complete) I0703 01:44:12.639843 4745 raft_consensus.cc:2537] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [term 3 FOLLOWER]: Leader election won for term 3 I0703 01:44:12.640008 4745 raft_consensus.cc:593] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [term 3 LEADER]: Becoming Leader. State: Replica: 8c1e8943f5654f71b030ec7c0b733742, State: Running, Role: LEADER I0703 01:44:12.640301 4745 consensus_queue.cc:204] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { replace: true } } peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } } W0703 01:44:12.645718 5452 log.cc:667] Time spent T 930bbe4ac3234ed49a2207c4902541f1 P 213596374e4142dca6ecfc701ba1c566: Append to log took a long time: real 0.086s user 0.001s sys 0.000s I0703 01:44:12.653647 5429 raft_consensus.cc:325] T 25b33ca4b32a4a2493d03564bfe1bf7e P 1157c91cc6f54a68a1d99017ac8444aa [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: true } } I0703 01:44:12.653854 5429 raft_consensus.cc:628] T 25b33ca4b32a4a2493d03564bfe1bf7e P 1157c91cc6f54a68a1d99017ac8444aa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1157c91cc6f54a68a1d99017ac8444aa, State: Initialized, Role: LEARNER I0703 01:44:12.654003 5429 consensus_queue.cc:227] T 25b33ca4b32a4a2493d03564bfe1bf7e P 1157c91cc6f54a68a1d99017ac8444aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 227, Last appended: 1.227, Last appended by leader: 227, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 221 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: true } } I0703 01:44:12.670395 3434 catalog_manager.cc:3385] ChangeConfig:REMOVE_PEER RPC for tablet eea4e9534e0d4d25a827fae00bbeae4d with cas_config_opid_index 254: aborting the task: latest config opid_index 255; task opid_index 254 I0703 01:44:12.663327 5429 ts_tablet_manager.cc:1073] T 25b33ca4b32a4a2493d03564bfe1bf7e P 1157c91cc6f54a68a1d99017ac8444aa: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s I0703 01:44:12.680368 5579 raft_consensus.cc:890] T 6c8253ac9a9a405885f1f57c59a84fd5 P 55223945fe3d4dd8ac03ba9dab346f54: attempting to promote NON_VOTER aa8b9c58b466436faa5086e85b5224a7 to VOTER I0703 01:44:12.680902 5579 consensus_peers.cc:455] T 6c8253ac9a9a405885f1f57c59a84fd5 P 55223945fe3d4dd8ac03ba9dab346f54 -> Peer aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197:51329): Closing peer: aa8b9c58b466436faa5086e85b5224a7 I0703 01:44:12.680999 5579 consensus_peers.cc:455] T 6c8253ac9a9a405885f1f57c59a84fd5 P 55223945fe3d4dd8ac03ba9dab346f54 -> Peer 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746): Closing peer: 57bea6dc182c41a78304417a3cbfc4a7 I0703 01:44:12.681322 5579 consensus_queue.cc:204] T 6c8253ac9a9a405885f1f57c59a84fd5 P 55223945fe3d4dd8ac03ba9dab346f54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 283, Committed index: 283, Last appended: 1.283, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } I0703 01:44:12.683972 4965 raft_consensus.cc:1081] T 6c8253ac9a9a405885f1f57c59a84fd5 P aa8b9c58b466436faa5086e85b5224a7 [term 1 LEARNER]: Refusing update from remote peer 55223945fe3d4dd8ac03ba9dab346f54: Log matching property violated. Preceding OpId in replica: term: 1 index: 283. Preceding OpId from leader: term: 1 index: 284. (index mismatch) I0703 01:44:12.684684 5579 consensus_queue.cc:962] T 6c8253ac9a9a405885f1f57c59a84fd5 P 55223945fe3d4dd8ac03ba9dab346f54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 284, Last known committed idx: 283, Time since last communication: 0.000s W0703 01:44:12.686064 3630 consensus_peers.cc:433] T 930bbe4ac3234ed49a2207c4902541f1 P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer 213596374e4142dca6ecfc701ba1c566 (127.5.240.196:51288): Couldn't send request to peer 213596374e4142dca6ecfc701ba1c566 for tablet 930bbe4ac3234ed49a2207c4902541f1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 6 times. I0703 01:44:12.686430 3696 raft_consensus.cc:1081] T 6c8253ac9a9a405885f1f57c59a84fd5 P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 FOLLOWER]: Refusing update from remote peer 55223945fe3d4dd8ac03ba9dab346f54: Log matching property violated. Preceding OpId in replica: term: 1 index: 283. Preceding OpId from leader: term: 1 index: 284. (index mismatch) I0703 01:44:12.687438 5577 raft_consensus.cc:457] T dc02a796568c4677a2eebfacae32fc38 P 213596374e4142dca6ecfc701ba1c566 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1157c91cc6f54a68a1d99017ac8444aa) I0703 01:44:12.687597 5577 raft_consensus.cc:2730] T dc02a796568c4677a2eebfacae32fc38 P 213596374e4142dca6ecfc701ba1c566: Snoozing failure detection for 1.618s (starting election) I0703 01:44:12.687309 5492 raft_consensus.cc:2684] T 6c8253ac9a9a405885f1f57c59a84fd5 P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 LEADER]: Committing config change with OpId 1.284: config changed from index 232 to 284, aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197) changed from NON_VOTER to VOTER. New config: { opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } } I0703 01:44:12.687710 5577 raft_consensus.cc:479] T dc02a796568c4677a2eebfacae32fc38 P 213596374e4142dca6ecfc701ba1c566 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { replace: true } } peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } } I0703 01:44:12.688716 5577 leader_election.cc:243] T dc02a796568c4677a2eebfacae32fc38 P 213596374e4142dca6ecfc701ba1c566 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377) I0703 01:44:12.689090 5577 leader_election.cc:243] T dc02a796568c4677a2eebfacae32fc38 P 213596374e4142dca6ecfc701ba1c566 [CANDIDATE]: Term 3 pre-election: Requesting pre-vote from peer 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064) I0703 01:44:12.690033 5235 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "dc02a796568c4677a2eebfacae32fc38" candidate_uuid: "213596374e4142dca6ecfc701ba1c566" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "1157c91cc6f54a68a1d99017ac8444aa" is_pre_election: true I0703 01:44:12.689970 5581 consensus_queue.cc:962] T 6c8253ac9a9a405885f1f57c59a84fd5 P 55223945fe3d4dd8ac03ba9dab346f54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 284, Last known committed idx: 283, Time since last communication: 0.000s I0703 01:44:12.690240 5235 raft_consensus.cc:2173] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 213596374e4142dca6ecfc701ba1c566 in current term 3: Already voted for candidate 8c1e8943f5654f71b030ec7c0b733742 in this term. I0703 01:44:12.691751 3696 raft_consensus.cc:2684] T 6c8253ac9a9a405885f1f57c59a84fd5 P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 FOLLOWER]: Committing config change with OpId 1.284: config changed from index 232 to 284, aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197) changed from NON_VOTER to VOTER. New config: { opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } } I0703 01:44:12.692081 3448 catalog_manager.cc:3798] T 6c8253ac9a9a405885f1f57c59a84fd5 P 55223945fe3d4dd8ac03ba9dab346f54 reported cstate change: config changed from index 232 to 284, aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "55223945fe3d4dd8ac03ba9dab346f54" committed_config { opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } health_report { overall_health: UNKNOWN } } } I0703 01:44:12.695667 4966 raft_consensus.cc:2684] T 6c8253ac9a9a405885f1f57c59a84fd5 P aa8b9c58b466436faa5086e85b5224a7 [term 1 FOLLOWER]: Committing config change with OpId 1.284: config changed from index 232 to 284, aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197) changed from NON_VOTER to VOTER. New config: { opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } } I0703 01:44:12.696717 3448 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 6c8253ac9a9a405885f1f57c59a84fd5 (attempt 1) I0703 01:44:12.697504 4762 leader_election.cc:393] T dc02a796568c4677a2eebfacae32fc38 P 213596374e4142dca6ecfc701ba1c566 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377). Message: Invalid argument: T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 213596374e4142dca6ecfc701ba1c566 in current term 3: Already voted for candidate 8c1e8943f5654f71b030ec7c0b733742 in this term. I0703 01:44:12.698150 3570 consensus_peers.cc:455] T 6c8253ac9a9a405885f1f57c59a84fd5 P 55223945fe3d4dd8ac03ba9dab346f54 -> Peer aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197:51329): Closing peer: aa8b9c58b466436faa5086e85b5224a7 I0703 01:44:12.698221 3570 consensus_peers.cc:455] T 6c8253ac9a9a405885f1f57c59a84fd5 P 55223945fe3d4dd8ac03ba9dab346f54 -> Peer 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746): Closing peer: 57bea6dc182c41a78304417a3cbfc4a7 I0703 01:44:12.698391 3570 consensus_queue.cc:204] T 6c8253ac9a9a405885f1f57c59a84fd5 P 55223945fe3d4dd8ac03ba9dab346f54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 284, Committed index: 284, Last appended: 1.284, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 285 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } I0703 01:44:12.702081 4966 raft_consensus.cc:1081] T 6c8253ac9a9a405885f1f57c59a84fd5 P aa8b9c58b466436faa5086e85b5224a7 [term 1 FOLLOWER]: Refusing update from remote peer 55223945fe3d4dd8ac03ba9dab346f54: Log matching property violated. Preceding OpId in replica: term: 1 index: 284. Preceding OpId from leader: term: 1 index: 285. (index mismatch) I0703 01:44:12.702901 5491 consensus_queue.cc:962] T 6c8253ac9a9a405885f1f57c59a84fd5 P 55223945fe3d4dd8ac03ba9dab346f54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 285, Last known committed idx: 284, Time since last communication: 0.000s I0703 01:44:12.707425 5491 raft_consensus.cc:2684] T 6c8253ac9a9a405885f1f57c59a84fd5 P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 LEADER]: Committing config change with OpId 1.285: config changed from index 284 to 285, VOTER 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194) evicted. New config: { opid_index: 285 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } } I0703 01:44:12.712740 3437 catalog_manager.cc:3352] ChangeConfig:REMOVE_PEER RPC for tablet 6c8253ac9a9a405885f1f57c59a84fd5 with cas_config_opid_index 284: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0703 01:44:12.713090 3448 catalog_manager.cc:3798] T 6c8253ac9a9a405885f1f57c59a84fd5 P 55223945fe3d4dd8ac03ba9dab346f54 reported cstate change: config changed from index 284 to 285, VOTER 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194) evicted. New cstate: current_term: 1 leader_uuid: "55223945fe3d4dd8ac03ba9dab346f54" committed_config { opid_index: 285 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I0703 01:44:12.715782 4965 raft_consensus.cc:2684] T 6c8253ac9a9a405885f1f57c59a84fd5 P aa8b9c58b466436faa5086e85b5224a7 [term 1 FOLLOWER]: Committing config change with OpId 1.285: config changed from index 284 to 285, VOTER 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194) evicted. New config: { opid_index: 285 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } } I0703 01:44:12.721004 3448 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 6c8253ac9a9a405885f1f57c59a84fd5 on 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746) (TS 57bea6dc182c41a78304417a3cbfc4a7 not found in new config with opid_index 285) I0703 01:44:12.721781 3676 tablet_service.cc:800] Processing DeleteTablet for tablet 6c8253ac9a9a405885f1f57c59a84fd5 with delete_type TABLET_DATA_TOMBSTONED (TS 57bea6dc182c41a78304417a3cbfc4a7 not found in new config with opid_index 285) from {username='jenkins'} at 127.5.240.194:39884 I0703 01:44:12.722209 5623 tablet_replica.cc:261] T 6c8253ac9a9a405885f1f57c59a84fd5 P 57bea6dc182c41a78304417a3cbfc4a7: stopping tablet replica I0703 01:44:12.722344 5623 maintenance_manager.cc:237] P 57bea6dc182c41a78304417a3cbfc4a7: Unregistered op CompactRowSetsOp(6c8253ac9a9a405885f1f57c59a84fd5) I0703 01:44:12.722472 5623 maintenance_manager.cc:237] P 57bea6dc182c41a78304417a3cbfc4a7: Unregistered op MinorDeltaCompactionOp(6c8253ac9a9a405885f1f57c59a84fd5) I0703 01:44:12.722546 5623 maintenance_manager.cc:237] P 57bea6dc182c41a78304417a3cbfc4a7: Unregistered op MajorDeltaCompactionOp(6c8253ac9a9a405885f1f57c59a84fd5) I0703 01:44:12.722635 5623 maintenance_manager.cc:237] P 57bea6dc182c41a78304417a3cbfc4a7: Unregistered op UndoDeltaBlockGCOp(6c8253ac9a9a405885f1f57c59a84fd5) I0703 01:44:12.722704 5623 maintenance_manager.cc:237] P 57bea6dc182c41a78304417a3cbfc4a7: Unregistered op FlushMRSOp(6c8253ac9a9a405885f1f57c59a84fd5) I0703 01:44:12.722750 5623 maintenance_manager.cc:237] P 57bea6dc182c41a78304417a3cbfc4a7: Unregistered op FlushDeltaMemStoresOp(6c8253ac9a9a405885f1f57c59a84fd5) I0703 01:44:12.722796 5623 maintenance_manager.cc:237] P 57bea6dc182c41a78304417a3cbfc4a7: Unregistered op LogGCOp(6c8253ac9a9a405885f1f57c59a84fd5) I0703 01:44:12.722846 5623 raft_consensus.cc:2040] T 6c8253ac9a9a405885f1f57c59a84fd5 P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 FOLLOWER]: Raft consensus shutting down. I0703 01:44:12.722952 5623 raft_consensus.cc:2067] T 6c8253ac9a9a405885f1f57c59a84fd5 P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 FOLLOWER]: Raft consensus is shut down! I0703 01:44:12.724164 5623 ts_tablet_manager.cc:1369] T 6c8253ac9a9a405885f1f57c59a84fd5 P 57bea6dc182c41a78304417a3cbfc4a7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:44:12.727538 3445 catalog_manager.cc:3798] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 reported cstate change: term changed from 2 to 3, leader changed from 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199) to 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195). New cstate: current_term: 3 leader_uuid: "8c1e8943f5654f71b030ec7c0b733742" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } health_report { overall_health: UNKNOWN } } } I0703 01:44:12.730814 5623 ts_tablet_manager.cc:1382] T 6c8253ac9a9a405885f1f57c59a84fd5 P 57bea6dc182c41a78304417a3cbfc4a7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.284 I0703 01:44:12.730887 5623 log.cc:986] T 6c8253ac9a9a405885f1f57c59a84fd5 P 57bea6dc182c41a78304417a3cbfc4a7: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal/wals/6c8253ac9a9a405885f1f57c59a84fd5 I0703 01:44:12.731580 3434 catalog_manager.cc:3103] TS 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746): tablet 6c8253ac9a9a405885f1f57c59a84fd5 (table rebalance_test_table_4 [id=d5e3657f924944ca895ef4125aaa82a1]) successfully deleted I0703 01:44:12.741291 5641 ts_tablet_manager.cc:1382] T eea4e9534e0d4d25a827fae00bbeae4d P 8c1e8943f5654f71b030ec7c0b733742: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.254 I0703 01:44:12.741578 5641 log.cc:986] T eea4e9534e0d4d25a827fae00bbeae4d P 8c1e8943f5654f71b030ec7c0b733742: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal/wals/eea4e9534e0d4d25a827fae00bbeae4d I0703 01:44:12.743870 3437 catalog_manager.cc:3103] TS 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064): tablet eea4e9534e0d4d25a827fae00bbeae4d (table rebalance_test_table_2 [id=24e8eff3fcb34269ba893ac2456e5712]) successfully deleted I0703 01:44:12.746412 3838 tablet_copy_service.cc:325] P 8c1e8943f5654f71b030ec7c0b733742: Request end of tablet copy session 1157c91cc6f54a68a1d99017ac8444aa-25b33ca4b32a4a2493d03564bfe1bf7e received from {username='jenkins'} at 127.5.240.199:55739 I0703 01:44:12.746496 3838 tablet_copy_service.cc:413] P 8c1e8943f5654f71b030ec7c0b733742: ending tablet copy session 1157c91cc6f54a68a1d99017ac8444aa-25b33ca4b32a4a2493d03564bfe1bf7e on tablet 25b33ca4b32a4a2493d03564bfe1bf7e with peer 1157c91cc6f54a68a1d99017ac8444aa W0703 01:44:12.761991 3507 consensus_peers.cc:433] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 -> Peer 213596374e4142dca6ecfc701ba1c566 (127.5.240.196:51288): Couldn't send request to peer 213596374e4142dca6ecfc701ba1c566 for tablet 60d3228294e945c5b343ae62e76dc2f0. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 5 times. I0703 01:44:12.762567 3815 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "dc02a796568c4677a2eebfacae32fc38" candidate_uuid: "213596374e4142dca6ecfc701ba1c566" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "8c1e8943f5654f71b030ec7c0b733742" is_pre_election: true I0703 01:44:12.777001 3815 raft_consensus.cc:2204] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [term 3 LEADER]: Leader pre-election vote request: Denying vote to candidate 213596374e4142dca6ecfc701ba1c566 for term 3 because replica is either leader or believes a valid leader to be alive. I0703 01:44:12.779646 4765 leader_election.cc:393] T dc02a796568c4677a2eebfacae32fc38 P 213596374e4142dca6ecfc701ba1c566 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064). Message: Invalid argument: T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [term 3 LEADER]: Leader pre-election vote request: Denying vote to candidate 213596374e4142dca6ecfc701ba1c566 for term 3 because replica is either leader or believes a valid leader to be alive. I0703 01:44:12.779736 4765 leader_election.cc:270] T dc02a796568c4677a2eebfacae32fc38 P 213596374e4142dca6ecfc701ba1c566 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. I0703 01:44:12.780274 5577 raft_consensus.cc:2730] T dc02a796568c4677a2eebfacae32fc38 P 213596374e4142dca6ecfc701ba1c566: Snoozing failure detection for 2.205s (election complete) I0703 01:44:12.780354 5577 raft_consensus.cc:2782] T dc02a796568c4677a2eebfacae32fc38 P 213596374e4142dca6ecfc701ba1c566 [term 2 FOLLOWER]: Advancing to term 3 I0703 01:44:12.782737 5577 raft_consensus.cc:2485] T dc02a796568c4677a2eebfacae32fc38 P 213596374e4142dca6ecfc701ba1c566 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority W0703 01:44:12.835870 3504 consensus_peers.cc:433] T 2051955b4b644a03a9adad816d350596 P 55223945fe3d4dd8ac03ba9dab346f54 -> Peer 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377): Couldn't send request to peer 1157c91cc6f54a68a1d99017ac8444aa for tablet 2051955b4b644a03a9adad816d350596. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 5 times. W0703 01:44:12.874732 3749 consensus_peers.cc:433] T 674ed5a88db04825a4eaec2fa996a852 P 8c1e8943f5654f71b030ec7c0b733742 -> Peer 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377): Couldn't send request to peer 1157c91cc6f54a68a1d99017ac8444aa for tablet 674ed5a88db04825a4eaec2fa996a852. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 8 times. W0703 01:44:12.949811 3627 consensus_peers.cc:433] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197:51329): Couldn't send request to peer aa8b9c58b466436faa5086e85b5224a7 for tablet 049bd2ef267f47fca634858420ac35b9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 6 times. I0703 01:44:12.990917 5235 raft_consensus.cc:1026] T 25b33ca4b32a4a2493d03564bfe1bf7e P 1157c91cc6f54a68a1d99017ac8444aa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.220->[1.221-1.300] Dedup: 1.227->[1.228-1.300] I0703 01:44:13.073156 5447 tablet_bootstrap.cc:427] T 4a613c1be55c4888a4df36d3f1bfc54c P 6932f1cd716a48fe8d6c5573acfc48bf: Bootstrap replayed 1/1 log segments. Stats: ops{read=235 overwritten=0 applied=235 ignored=0} inserts{seen=887 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:44:13.077020 5447 tablet_bootstrap.cc:427] T 4a613c1be55c4888a4df36d3f1bfc54c P 6932f1cd716a48fe8d6c5573acfc48bf: Bootstrap complete. I0703 01:44:13.081614 5447 ts_tablet_manager.cc:1045] T 4a613c1be55c4888a4df36d3f1bfc54c P 6932f1cd716a48fe8d6c5573acfc48bf: Time spent bootstrapping tablet: real 3.400s user 0.317s sys 0.017s I0703 01:44:13.082505 5447 raft_consensus.cc:325] T 4a613c1be55c4888a4df36d3f1bfc54c P 6932f1cd716a48fe8d6c5573acfc48bf [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: NON_VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: true } } I0703 01:44:13.083827 5447 raft_consensus.cc:628] T 4a613c1be55c4888a4df36d3f1bfc54c P 6932f1cd716a48fe8d6c5573acfc48bf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6932f1cd716a48fe8d6c5573acfc48bf, State: Initialized, Role: LEARNER W0703 01:44:13.091578 3630 consensus_peers.cc:433] T 4a613c1be55c4888a4df36d3f1bfc54c P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198:51033): Couldn't send request to peer 6932f1cd716a48fe8d6c5573acfc48bf for tablet 4a613c1be55c4888a4df36d3f1bfc54c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 8 times. I0703 01:44:13.116984 4833 raft_consensus.cc:1081] T dc02a796568c4677a2eebfacae32fc38 P 213596374e4142dca6ecfc701ba1c566 [term 3 FOLLOWER]: Refusing update from remote peer 8c1e8943f5654f71b030ec7c0b733742: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch) I0703 01:44:13.084127 5447 consensus_queue.cc:227] T 4a613c1be55c4888a4df36d3f1bfc54c P 6932f1cd716a48fe8d6c5573acfc48bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 235, Last appended: 1.235, Last appended by leader: 235, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 223 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: NON_VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: true } } I0703 01:44:13.118347 4744 consensus_queue.cc:962] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [LEADER]: Connected to new peer: Peer: permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s W0703 01:44:13.134074 3630 consensus_peers.cc:433] T 930bbe4ac3234ed49a2207c4902541f1 P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer 213596374e4142dca6ecfc701ba1c566 (127.5.240.196:51288): Couldn't send request to peer 213596374e4142dca6ecfc701ba1c566 for tablet 930bbe4ac3234ed49a2207c4902541f1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 7 times. I0703 01:44:13.138022 5234 raft_consensus.cc:1081] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa [term 3 FOLLOWER]: Refusing update from remote peer 8c1e8943f5654f71b030ec7c0b733742: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch) I0703 01:44:13.149080 5447 ts_tablet_manager.cc:1073] T 4a613c1be55c4888a4df36d3f1bfc54c P 6932f1cd716a48fe8d6c5573acfc48bf: Time spent starting tablet: real 0.067s user 0.001s sys 0.000s I0703 01:44:13.149931 3716 tablet_copy_service.cc:325] P 57bea6dc182c41a78304417a3cbfc4a7: Request end of tablet copy session 6932f1cd716a48fe8d6c5573acfc48bf-4a613c1be55c4888a4df36d3f1bfc54c received from {username='jenkins'} at 127.5.240.198:49512 I0703 01:44:13.150050 3716 tablet_copy_service.cc:413] P 57bea6dc182c41a78304417a3cbfc4a7: ending tablet copy session 6932f1cd716a48fe8d6c5573acfc48bf-4a613c1be55c4888a4df36d3f1bfc54c on tablet 4a613c1be55c4888a4df36d3f1bfc54c with peer 6932f1cd716a48fe8d6c5573acfc48bf I0703 01:44:13.153139 4745 consensus_queue.cc:962] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s I0703 01:44:13.153611 3445 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet dc02a796568c4677a2eebfacae32fc38 (attempt 1) I0703 01:44:13.156112 3816 consensus_peers.cc:455] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 -> Peer 213596374e4142dca6ecfc701ba1c566 (127.5.240.196:51288): Closing peer: 213596374e4142dca6ecfc701ba1c566 I0703 01:44:13.156229 3816 consensus_peers.cc:455] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 -> Peer 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377): Closing peer: 1157c91cc6f54a68a1d99017ac8444aa I0703 01:44:13.156419 3816 consensus_queue.cc:204] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } } I0703 01:44:13.159168 4833 raft_consensus.cc:1081] T dc02a796568c4677a2eebfacae32fc38 P 213596374e4142dca6ecfc701ba1c566 [term 3 FOLLOWER]: Refusing update from remote peer 8c1e8943f5654f71b030ec7c0b733742: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch) I0703 01:44:13.160509 4745 consensus_queue.cc:962] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [LEADER]: Connected to new peer: Peer: permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s I0703 01:44:13.170344 4745 raft_consensus.cc:2684] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 [term 3 LEADER]: Committing config change with OpId 3.6: config changed from index 4 to 6, VOTER 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } } } I0703 01:44:13.175417 3437 catalog_manager.cc:3352] ChangeConfig:REMOVE_PEER RPC for tablet dc02a796568c4677a2eebfacae32fc38 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0703 01:44:13.176451 3445 catalog_manager.cc:3798] T dc02a796568c4677a2eebfacae32fc38 P 8c1e8943f5654f71b030ec7c0b733742 reported cstate change: config changed from index 4 to 6, VOTER 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199) evicted. New cstate: current_term: 3 leader_uuid: "8c1e8943f5654f71b030ec7c0b733742" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I0703 01:44:13.181221 3445 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet dc02a796568c4677a2eebfacae32fc38 on 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377) (TS 1157c91cc6f54a68a1d99017ac8444aa not found in new config with opid_index 6) I0703 01:44:13.191926 4834 raft_consensus.cc:2684] T dc02a796568c4677a2eebfacae32fc38 P 213596374e4142dca6ecfc701ba1c566 [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 4 to 6, VOTER 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } } } W0703 01:44:13.201910 3507 consensus_peers.cc:433] T eada0e8b8608424594cabd35cf9891e3 P 55223945fe3d4dd8ac03ba9dab346f54 -> Peer 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198:51033): Couldn't send request to peer 6932f1cd716a48fe8d6c5573acfc48bf for tablet eada0e8b8608424594cabd35cf9891e3. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 7 times. I0703 01:44:13.229815 5214 tablet_service.cc:800] Processing DeleteTablet for tablet dc02a796568c4677a2eebfacae32fc38 with delete_type TABLET_DATA_TOMBSTONED (TS 1157c91cc6f54a68a1d99017ac8444aa not found in new config with opid_index 6) from {username='jenkins'} at 127.5.240.199:46115 I0703 01:44:13.254256 5677 tablet_replica.cc:261] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa: stopping tablet replica I0703 01:44:13.254487 5677 maintenance_manager.cc:237] P 1157c91cc6f54a68a1d99017ac8444aa: Unregistered op CompactRowSetsOp(dc02a796568c4677a2eebfacae32fc38) I0703 01:44:13.254688 5677 maintenance_manager.cc:237] P 1157c91cc6f54a68a1d99017ac8444aa: Unregistered op MinorDeltaCompactionOp(dc02a796568c4677a2eebfacae32fc38) I0703 01:44:13.267791 5677 maintenance_manager.cc:237] P 1157c91cc6f54a68a1d99017ac8444aa: Unregistered op MajorDeltaCompactionOp(dc02a796568c4677a2eebfacae32fc38) I0703 01:44:13.273293 5677 maintenance_manager.cc:237] P 1157c91cc6f54a68a1d99017ac8444aa: Unregistered op UndoDeltaBlockGCOp(dc02a796568c4677a2eebfacae32fc38) I0703 01:44:13.273423 5677 maintenance_manager.cc:237] P 1157c91cc6f54a68a1d99017ac8444aa: Unregistered op FlushMRSOp(dc02a796568c4677a2eebfacae32fc38) I0703 01:44:13.275688 5677 maintenance_manager.cc:237] P 1157c91cc6f54a68a1d99017ac8444aa: Unregistered op FlushDeltaMemStoresOp(dc02a796568c4677a2eebfacae32fc38) I0703 01:44:13.275775 5677 maintenance_manager.cc:237] P 1157c91cc6f54a68a1d99017ac8444aa: Unregistered op LogGCOp(dc02a796568c4677a2eebfacae32fc38) I0703 01:44:13.276815 5677 raft_consensus.cc:2040] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa [term 3 FOLLOWER]: Raft consensus shutting down. I0703 01:44:13.277158 5677 raft_consensus.cc:2067] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa [term 3 FOLLOWER]: Raft consensus is shut down! I0703 01:44:13.279126 5677 ts_tablet_manager.cc:1369] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:44:13.286605 5677 ts_tablet_manager.cc:1382] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.5 I0703 01:44:13.286773 5677 log.cc:986] T dc02a796568c4677a2eebfacae32fc38 P 1157c91cc6f54a68a1d99017ac8444aa: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-6/wal/wals/dc02a796568c4677a2eebfacae32fc38 I0703 01:44:13.287662 3434 catalog_manager.cc:3103] TS 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377): tablet dc02a796568c4677a2eebfacae32fc38 (table TestTable [id=35e213a216c74c5b815f1702a16a7c2e]) successfully deleted W0703 01:44:13.308503 3504 consensus_peers.cc:433] T 2051955b4b644a03a9adad816d350596 P 55223945fe3d4dd8ac03ba9dab346f54 -> Peer 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377): Couldn't send request to peer 1157c91cc6f54a68a1d99017ac8444aa for tablet 2051955b4b644a03a9adad816d350596. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 6 times. W0703 01:44:13.333921 3749 consensus_peers.cc:433] T 674ed5a88db04825a4eaec2fa996a852 P 8c1e8943f5654f71b030ec7c0b733742 -> Peer 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377): Couldn't send request to peer 1157c91cc6f54a68a1d99017ac8444aa for tablet 674ed5a88db04825a4eaec2fa996a852. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 9 times. W0703 01:44:13.363847 3507 consensus_peers.cc:433] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 -> Peer 213596374e4142dca6ecfc701ba1c566 (127.5.240.196:51288): Couldn't send request to peer 213596374e4142dca6ecfc701ba1c566 for tablet 60d3228294e945c5b343ae62e76dc2f0. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 6 times. I0703 01:44:13.400699 5466 tablet_bootstrap.cc:427] T 674ed5a88db04825a4eaec2fa996a852 P 1157c91cc6f54a68a1d99017ac8444aa: Bootstrap replayed 1/1 log segments. Stats: ops{read=288 overwritten=0 applied=288 ignored=0} inserts{seen=975 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:44:13.401012 5466 tablet_bootstrap.cc:427] T 674ed5a88db04825a4eaec2fa996a852 P 1157c91cc6f54a68a1d99017ac8444aa: Bootstrap complete. I0703 01:44:13.401242 5466 ts_tablet_manager.cc:1045] T 674ed5a88db04825a4eaec2fa996a852 P 1157c91cc6f54a68a1d99017ac8444aa: Time spent bootstrapping tablet: real 3.527s user 0.363s sys 0.033s I0703 01:44:13.401911 5466 raft_consensus.cc:325] T 674ed5a88db04825a4eaec2fa996a852 P 1157c91cc6f54a68a1d99017ac8444aa [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: true } } I0703 01:44:13.402180 5466 raft_consensus.cc:628] T 674ed5a88db04825a4eaec2fa996a852 P 1157c91cc6f54a68a1d99017ac8444aa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1157c91cc6f54a68a1d99017ac8444aa, State: Initialized, Role: LEARNER I0703 01:44:13.402334 5466 consensus_queue.cc:227] T 674ed5a88db04825a4eaec2fa996a852 P 1157c91cc6f54a68a1d99017ac8444aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 288, Last appended: 1.288, Last appended by leader: 288, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: true } } I0703 01:44:13.403163 5466 ts_tablet_manager.cc:1073] T 674ed5a88db04825a4eaec2fa996a852 P 1157c91cc6f54a68a1d99017ac8444aa: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s I0703 01:44:13.404095 3838 tablet_copy_service.cc:325] P 8c1e8943f5654f71b030ec7c0b733742: Request end of tablet copy session 1157c91cc6f54a68a1d99017ac8444aa-674ed5a88db04825a4eaec2fa996a852 received from {username='jenkins'} at 127.5.240.199:55739 I0703 01:44:13.404171 3838 tablet_copy_service.cc:413] P 8c1e8943f5654f71b030ec7c0b733742: ending tablet copy session 1157c91cc6f54a68a1d99017ac8444aa-674ed5a88db04825a4eaec2fa996a852 on tablet 674ed5a88db04825a4eaec2fa996a852 with peer 1157c91cc6f54a68a1d99017ac8444aa I0703 01:44:13.408716 5025 raft_consensus.cc:890] T 25b33ca4b32a4a2493d03564bfe1bf7e P 8c1e8943f5654f71b030ec7c0b733742: attempting to promote NON_VOTER 1157c91cc6f54a68a1d99017ac8444aa to VOTER I0703 01:44:13.409045 5025 consensus_peers.cc:455] T 25b33ca4b32a4a2493d03564bfe1bf7e P 8c1e8943f5654f71b030ec7c0b733742 -> Peer 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377): Closing peer: 1157c91cc6f54a68a1d99017ac8444aa I0703 01:44:13.409116 5025 consensus_peers.cc:455] T 25b33ca4b32a4a2493d03564bfe1bf7e P 8c1e8943f5654f71b030ec7c0b733742 -> Peer 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746): Closing peer: 57bea6dc182c41a78304417a3cbfc4a7 I0703 01:44:13.409333 5025 consensus_queue.cc:204] T 25b33ca4b32a4a2493d03564bfe1bf7e P 8c1e8943f5654f71b030ec7c0b733742 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 313, Committed index: 313, Last appended: 1.313, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 314 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: false } } I0703 01:44:13.412513 3692 raft_consensus.cc:1081] T 25b33ca4b32a4a2493d03564bfe1bf7e P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 FOLLOWER]: Refusing update from remote peer 8c1e8943f5654f71b030ec7c0b733742: Log matching property violated. Preceding OpId in replica: term: 1 index: 313. Preceding OpId from leader: term: 1 index: 314. (index mismatch) I0703 01:44:13.417506 4745 consensus_queue.cc:962] T 25b33ca4b32a4a2493d03564bfe1bf7e P 8c1e8943f5654f71b030ec7c0b733742 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 314, Last known committed idx: 313, Time since last communication: 0.000s I0703 01:44:13.417728 5234 raft_consensus.cc:1081] T 25b33ca4b32a4a2493d03564bfe1bf7e P 1157c91cc6f54a68a1d99017ac8444aa [term 1 LEARNER]: Refusing update from remote peer 8c1e8943f5654f71b030ec7c0b733742: Log matching property violated. Preceding OpId in replica: term: 1 index: 313. Preceding OpId from leader: term: 1 index: 314. (index mismatch) I0703 01:44:13.419205 4745 consensus_queue.cc:962] T 25b33ca4b32a4a2493d03564bfe1bf7e P 8c1e8943f5654f71b030ec7c0b733742 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 314, Last known committed idx: 313, Time since last communication: 0.000s I0703 01:44:13.422531 4744 raft_consensus.cc:2684] T 25b33ca4b32a4a2493d03564bfe1bf7e P 8c1e8943f5654f71b030ec7c0b733742 [term 1 LEADER]: Committing config change with OpId 1.314: config changed from index 221 to 314, 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199) changed from NON_VOTER to VOTER. New config: { opid_index: 314 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: false } } } I0703 01:44:13.424542 5234 raft_consensus.cc:2684] T 25b33ca4b32a4a2493d03564bfe1bf7e P 1157c91cc6f54a68a1d99017ac8444aa [term 1 FOLLOWER]: Committing config change with OpId 1.314: config changed from index 221 to 314, 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199) changed from NON_VOTER to VOTER. New config: { opid_index: 314 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: false } } } I0703 01:44:13.427065 3445 catalog_manager.cc:3798] T 25b33ca4b32a4a2493d03564bfe1bf7e P 8c1e8943f5654f71b030ec7c0b733742 reported cstate change: config changed from index 221 to 314, 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8c1e8943f5654f71b030ec7c0b733742" committed_config { opid_index: 314 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I0703 01:44:13.429850 3692 raft_consensus.cc:2684] T 25b33ca4b32a4a2493d03564bfe1bf7e P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 FOLLOWER]: Committing config change with OpId 1.314: config changed from index 221 to 314, 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199) changed from NON_VOTER to VOTER. New config: { opid_index: 314 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: false } } } W0703 01:44:13.431298 3627 consensus_peers.cc:433] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197:51329): Couldn't send request to peer aa8b9c58b466436faa5086e85b5224a7 for tablet 049bd2ef267f47fca634858420ac35b9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 7 times. I0703 01:44:13.432751 3445 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 25b33ca4b32a4a2493d03564bfe1bf7e (attempt 1) I0703 01:44:13.435926 3815 consensus_peers.cc:455] T 25b33ca4b32a4a2493d03564bfe1bf7e P 8c1e8943f5654f71b030ec7c0b733742 -> Peer 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377): Closing peer: 1157c91cc6f54a68a1d99017ac8444aa I0703 01:44:13.436002 3815 consensus_peers.cc:455] T 25b33ca4b32a4a2493d03564bfe1bf7e P 8c1e8943f5654f71b030ec7c0b733742 -> Peer 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746): Closing peer: 57bea6dc182c41a78304417a3cbfc4a7 I0703 01:44:13.436208 3815 consensus_queue.cc:204] T 25b33ca4b32a4a2493d03564bfe1bf7e P 8c1e8943f5654f71b030ec7c0b733742 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 315, Committed index: 315, Last appended: 1.315, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: false } } I0703 01:44:13.437912 5234 raft_consensus.cc:1081] T 25b33ca4b32a4a2493d03564bfe1bf7e P 1157c91cc6f54a68a1d99017ac8444aa [term 1 FOLLOWER]: Refusing update from remote peer 8c1e8943f5654f71b030ec7c0b733742: Log matching property violated. Preceding OpId in replica: term: 1 index: 315. Preceding OpId from leader: term: 1 index: 316. (index mismatch) I0703 01:44:13.438586 5154 consensus_queue.cc:962] T 25b33ca4b32a4a2493d03564bfe1bf7e P 8c1e8943f5654f71b030ec7c0b733742 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 316, Last known committed idx: 315, Time since last communication: 0.000s I0703 01:44:13.441534 5025 raft_consensus.cc:2684] T 25b33ca4b32a4a2493d03564bfe1bf7e P 8c1e8943f5654f71b030ec7c0b733742 [term 1 LEADER]: Committing config change with OpId 1.316: config changed from index 314 to 316, VOTER 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194) evicted. New config: { opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: false } } } I0703 01:44:13.442582 5234 raft_consensus.cc:2684] T 25b33ca4b32a4a2493d03564bfe1bf7e P 1157c91cc6f54a68a1d99017ac8444aa [term 1 FOLLOWER]: Committing config change with OpId 1.316: config changed from index 314 to 316, VOTER 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194) evicted. New config: { opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: false } } } I0703 01:44:13.444483 3437 catalog_manager.cc:3352] ChangeConfig:REMOVE_PEER RPC for tablet 25b33ca4b32a4a2493d03564bfe1bf7e with cas_config_opid_index 314: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0703 01:44:13.446506 3448 catalog_manager.cc:3798] T 25b33ca4b32a4a2493d03564bfe1bf7e P 8c1e8943f5654f71b030ec7c0b733742 reported cstate change: config changed from index 314 to 316, VOTER 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194) evicted. New cstate: current_term: 1 leader_uuid: "8c1e8943f5654f71b030ec7c0b733742" committed_config { opid_index: 316 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I0703 01:44:13.448988 3448 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 25b33ca4b32a4a2493d03564bfe1bf7e on 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746) (TS 57bea6dc182c41a78304417a3cbfc4a7 not found in new config with opid_index 316) I0703 01:44:13.449513 3676 tablet_service.cc:800] Processing DeleteTablet for tablet 25b33ca4b32a4a2493d03564bfe1bf7e with delete_type TABLET_DATA_TOMBSTONED (TS 57bea6dc182c41a78304417a3cbfc4a7 not found in new config with opid_index 316) from {username='jenkins'} at 127.5.240.194:39884 I0703 01:44:13.469199 5693 tablet_replica.cc:261] T 25b33ca4b32a4a2493d03564bfe1bf7e P 57bea6dc182c41a78304417a3cbfc4a7: stopping tablet replica I0703 01:44:13.472746 5693 maintenance_manager.cc:237] P 57bea6dc182c41a78304417a3cbfc4a7: Unregistered op CompactRowSetsOp(25b33ca4b32a4a2493d03564bfe1bf7e) I0703 01:44:13.472909 5693 maintenance_manager.cc:237] P 57bea6dc182c41a78304417a3cbfc4a7: Unregistered op MinorDeltaCompactionOp(25b33ca4b32a4a2493d03564bfe1bf7e) I0703 01:44:13.473031 5693 maintenance_manager.cc:237] P 57bea6dc182c41a78304417a3cbfc4a7: Unregistered op MajorDeltaCompactionOp(25b33ca4b32a4a2493d03564bfe1bf7e) I0703 01:44:13.476951 5693 maintenance_manager.cc:237] P 57bea6dc182c41a78304417a3cbfc4a7: Unregistered op UndoDeltaBlockGCOp(25b33ca4b32a4a2493d03564bfe1bf7e) I0703 01:44:13.477919 5693 maintenance_manager.cc:237] P 57bea6dc182c41a78304417a3cbfc4a7: Unregistered op FlushMRSOp(25b33ca4b32a4a2493d03564bfe1bf7e) I0703 01:44:13.478021 5693 maintenance_manager.cc:237] P 57bea6dc182c41a78304417a3cbfc4a7: Unregistered op FlushDeltaMemStoresOp(25b33ca4b32a4a2493d03564bfe1bf7e) I0703 01:44:13.478101 5693 maintenance_manager.cc:237] P 57bea6dc182c41a78304417a3cbfc4a7: Unregistered op LogGCOp(25b33ca4b32a4a2493d03564bfe1bf7e) I0703 01:44:13.478196 5693 raft_consensus.cc:2040] T 25b33ca4b32a4a2493d03564bfe1bf7e P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 FOLLOWER]: Raft consensus shutting down. I0703 01:44:13.479935 5693 raft_consensus.cc:2067] T 25b33ca4b32a4a2493d03564bfe1bf7e P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 FOLLOWER]: Raft consensus is shut down! I0703 01:44:13.488852 5693 ts_tablet_manager.cc:1369] T 25b33ca4b32a4a2493d03564bfe1bf7e P 57bea6dc182c41a78304417a3cbfc4a7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:44:13.512403 5693 ts_tablet_manager.cc:1382] T 25b33ca4b32a4a2493d03564bfe1bf7e P 57bea6dc182c41a78304417a3cbfc4a7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.315 I0703 01:44:13.513813 5488 tablet_bootstrap.cc:427] T eada0e8b8608424594cabd35cf9891e3 P 6932f1cd716a48fe8d6c5573acfc48bf: Bootstrap replayed 1/1 log segments. Stats: ops{read=247 overwritten=0 applied=247 ignored=0} inserts{seen=899 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:44:13.515949 5488 tablet_bootstrap.cc:427] T eada0e8b8608424594cabd35cf9891e3 P 6932f1cd716a48fe8d6c5573acfc48bf: Bootstrap complete. I0703 01:44:13.516510 5488 ts_tablet_manager.cc:1045] T eada0e8b8608424594cabd35cf9891e3 P 6932f1cd716a48fe8d6c5573acfc48bf: Time spent bootstrapping tablet: real 3.214s user 0.326s sys 0.021s I0703 01:44:13.517735 5693 log.cc:986] T 25b33ca4b32a4a2493d03564bfe1bf7e P 57bea6dc182c41a78304417a3cbfc4a7: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal/wals/25b33ca4b32a4a2493d03564bfe1bf7e I0703 01:44:13.518530 3434 catalog_manager.cc:3103] TS 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746): tablet 25b33ca4b32a4a2493d03564bfe1bf7e (table rebalance_test_table_4 [id=d5e3657f924944ca895ef4125aaa82a1]) successfully deleted I0703 01:44:13.519017 5488 raft_consensus.cc:325] T eada0e8b8608424594cabd35cf9891e3 P 6932f1cd716a48fe8d6c5573acfc48bf [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: NON_VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: true } } I0703 01:44:13.519448 5488 raft_consensus.cc:628] T eada0e8b8608424594cabd35cf9891e3 P 6932f1cd716a48fe8d6c5573acfc48bf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6932f1cd716a48fe8d6c5573acfc48bf, State: Initialized, Role: LEARNER I0703 01:44:13.519685 5488 consensus_queue.cc:227] T eada0e8b8608424594cabd35cf9891e3 P 6932f1cd716a48fe8d6c5573acfc48bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 247, Last appended: 1.247, Last appended by leader: 247, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: NON_VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: true } } I0703 01:44:13.521518 5488 ts_tablet_manager.cc:1073] T eada0e8b8608424594cabd35cf9891e3 P 6932f1cd716a48fe8d6c5573acfc48bf: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s I0703 01:44:13.523212 3593 tablet_copy_service.cc:325] P 55223945fe3d4dd8ac03ba9dab346f54: Request end of tablet copy session 6932f1cd716a48fe8d6c5573acfc48bf-eada0e8b8608424594cabd35cf9891e3 received from {username='jenkins'} at 127.5.240.198:54009 I0703 01:44:13.523289 3593 tablet_copy_service.cc:413] P 55223945fe3d4dd8ac03ba9dab346f54: ending tablet copy session 6932f1cd716a48fe8d6c5573acfc48bf-eada0e8b8608424594cabd35cf9891e3 on tablet eada0e8b8608424594cabd35cf9891e3 with peer 6932f1cd716a48fe8d6c5573acfc48bf I0703 01:44:13.544425 5506 tablet_bootstrap.cc:427] T 049bd2ef267f47fca634858420ac35b9 P aa8b9c58b466436faa5086e85b5224a7: Bootstrap replayed 1/1 log segments. Stats: ops{read=238 overwritten=0 applied=238 ignored=0} inserts{seen=887 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:44:13.544883 5506 tablet_bootstrap.cc:427] T 049bd2ef267f47fca634858420ac35b9 P aa8b9c58b466436faa5086e85b5224a7: Bootstrap complete. I0703 01:44:13.545100 5506 ts_tablet_manager.cc:1045] T 049bd2ef267f47fca634858420ac35b9 P aa8b9c58b466436faa5086e85b5224a7: Time spent bootstrapping tablet: real 2.896s user 0.321s sys 0.014s I0703 01:44:13.545822 5506 raft_consensus.cc:325] T 049bd2ef267f47fca634858420ac35b9 P aa8b9c58b466436faa5086e85b5224a7 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: NON_VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: true } } I0703 01:44:13.546066 5506 raft_consensus.cc:628] T 049bd2ef267f47fca634858420ac35b9 P aa8b9c58b466436faa5086e85b5224a7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aa8b9c58b466436faa5086e85b5224a7, State: Initialized, Role: LEARNER I0703 01:44:13.546211 5506 consensus_queue.cc:227] T 049bd2ef267f47fca634858420ac35b9 P aa8b9c58b466436faa5086e85b5224a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 238, Last appended: 1.238, Last appended by leader: 238, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: NON_VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: true } } W0703 01:44:13.554484 3630 consensus_peers.cc:433] T 930bbe4ac3234ed49a2207c4902541f1 P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer 213596374e4142dca6ecfc701ba1c566 (127.5.240.196:51288): Couldn't send request to peer 213596374e4142dca6ecfc701ba1c566 for tablet 930bbe4ac3234ed49a2207c4902541f1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 8 times. I0703 01:44:13.557037 5506 ts_tablet_manager.cc:1073] T 049bd2ef267f47fca634858420ac35b9 P aa8b9c58b466436faa5086e85b5224a7: Time spent starting tablet: real 0.012s user 0.001s sys 0.005s I0703 01:44:13.564465 3716 tablet_copy_service.cc:325] P 57bea6dc182c41a78304417a3cbfc4a7: Request end of tablet copy session aa8b9c58b466436faa5086e85b5224a7-049bd2ef267f47fca634858420ac35b9 received from {username='jenkins'} at 127.5.240.197:44868 I0703 01:44:13.564612 3716 tablet_copy_service.cc:413] P 57bea6dc182c41a78304417a3cbfc4a7: ending tablet copy session aa8b9c58b466436faa5086e85b5224a7-049bd2ef267f47fca634858420ac35b9 on tablet 049bd2ef267f47fca634858420ac35b9 with peer aa8b9c58b466436faa5086e85b5224a7 I0703 01:44:13.588054 5452 tablet_bootstrap.cc:427] T 930bbe4ac3234ed49a2207c4902541f1 P 213596374e4142dca6ecfc701ba1c566: Bootstrap replayed 1/1 log segments. Stats: ops{read=281 overwritten=0 applied=280 ignored=0} inserts{seen=932 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates I0703 01:44:13.598881 5452 tablet_bootstrap.cc:427] T 930bbe4ac3234ed49a2207c4902541f1 P 213596374e4142dca6ecfc701ba1c566: Bootstrap complete. I0703 01:44:13.599102 5452 ts_tablet_manager.cc:1045] T 930bbe4ac3234ed49a2207c4902541f1 P 213596374e4142dca6ecfc701ba1c566: Time spent bootstrapping tablet: real 3.448s user 0.345s sys 0.024s I0703 01:44:13.599884 5452 raft_consensus.cc:325] T 930bbe4ac3234ed49a2207c4902541f1 P 213596374e4142dca6ecfc701ba1c566 [term 1 LEARNER]: Replica starting. Triggering 1 pending transactions. Active config: opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: true } } I0703 01:44:13.600924 5452 raft_consensus.cc:628] T 930bbe4ac3234ed49a2207c4902541f1 P 213596374e4142dca6ecfc701ba1c566 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 213596374e4142dca6ecfc701ba1c566, State: Initialized, Role: LEARNER I0703 01:44:13.601092 5452 consensus_queue.cc:227] T 930bbe4ac3234ed49a2207c4902541f1 P 213596374e4142dca6ecfc701ba1c566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 280, Last appended: 1.281, Last appended by leader: 281, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: true } } I0703 01:44:13.602324 5452 ts_tablet_manager.cc:1073] T 930bbe4ac3234ed49a2207c4902541f1 P 213596374e4142dca6ecfc701ba1c566: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s I0703 01:44:13.604713 3716 tablet_copy_service.cc:325] P 57bea6dc182c41a78304417a3cbfc4a7: Request end of tablet copy session 213596374e4142dca6ecfc701ba1c566-930bbe4ac3234ed49a2207c4902541f1 received from {username='jenkins'} at 127.5.240.196:36312 I0703 01:44:13.604797 3716 tablet_copy_service.cc:413] P 57bea6dc182c41a78304417a3cbfc4a7: ending tablet copy session 213596374e4142dca6ecfc701ba1c566-930bbe4ac3234ed49a2207c4902541f1 on tablet 930bbe4ac3234ed49a2207c4902541f1 with peer 213596374e4142dca6ecfc701ba1c566 I0703 01:44:13.651614 5097 raft_consensus.cc:1026] T eada0e8b8608424594cabd35cf9891e3 P 6932f1cd716a48fe8d6c5573acfc48bf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.238->[1.239-1.293] Dedup: 1.247->[1.248-1.293] I0703 01:44:13.665715 5096 raft_consensus.cc:1026] T 4a613c1be55c4888a4df36d3f1bfc54c P 6932f1cd716a48fe8d6c5573acfc48bf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.222->[1.223-1.293] Dedup: 1.235->[1.236-1.293] I0703 01:44:13.760309 5492 raft_consensus.cc:890] T eada0e8b8608424594cabd35cf9891e3 P 55223945fe3d4dd8ac03ba9dab346f54: attempting to promote NON_VOTER 6932f1cd716a48fe8d6c5573acfc48bf to VOTER I0703 01:44:13.760752 5492 consensus_peers.cc:455] T eada0e8b8608424594cabd35cf9891e3 P 55223945fe3d4dd8ac03ba9dab346f54 -> Peer 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198:51033): Closing peer: 6932f1cd716a48fe8d6c5573acfc48bf I0703 01:44:13.760852 5492 consensus_peers.cc:455] T eada0e8b8608424594cabd35cf9891e3 P 55223945fe3d4dd8ac03ba9dab346f54 -> Peer 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746): Closing peer: 57bea6dc182c41a78304417a3cbfc4a7 I0703 01:44:13.761080 5492 consensus_queue.cc:204] T eada0e8b8608424594cabd35cf9891e3 P 55223945fe3d4dd8ac03ba9dab346f54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 295, Committed index: 295, Last appended: 1.295, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false } } I0703 01:44:13.762393 5528 tablet_bootstrap.cc:427] T 60d3228294e945c5b343ae62e76dc2f0 P 213596374e4142dca6ecfc701ba1c566: Bootstrap replayed 1/1 log segments. Stats: ops{read=311 overwritten=0 applied=311 ignored=0} inserts{seen=963 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:44:13.763296 5528 tablet_bootstrap.cc:427] T 60d3228294e945c5b343ae62e76dc2f0 P 213596374e4142dca6ecfc701ba1c566: Bootstrap complete. I0703 01:44:13.763361 3696 raft_consensus.cc:1081] T eada0e8b8608424594cabd35cf9891e3 P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 FOLLOWER]: Refusing update from remote peer 55223945fe3d4dd8ac03ba9dab346f54: Log matching property violated. Preceding OpId in replica: term: 1 index: 295. Preceding OpId from leader: term: 1 index: 296. (index mismatch) I0703 01:44:13.763367 5097 raft_consensus.cc:1081] T eada0e8b8608424594cabd35cf9891e3 P 6932f1cd716a48fe8d6c5573acfc48bf [term 1 LEARNER]: Refusing update from remote peer 55223945fe3d4dd8ac03ba9dab346f54: Log matching property violated. Preceding OpId in replica: term: 1 index: 295. Preceding OpId from leader: term: 1 index: 296. (index mismatch) I0703 01:44:13.764066 5579 consensus_queue.cc:962] T eada0e8b8608424594cabd35cf9891e3 P 55223945fe3d4dd8ac03ba9dab346f54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 296, Last known committed idx: 295, Time since last communication: 0.000s I0703 01:44:13.764767 5579 consensus_queue.cc:962] T eada0e8b8608424594cabd35cf9891e3 P 55223945fe3d4dd8ac03ba9dab346f54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 296, Last known committed idx: 295, Time since last communication: 0.000s I0703 01:44:13.768038 5579 raft_consensus.cc:2684] T eada0e8b8608424594cabd35cf9891e3 P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 LEADER]: Committing config change with OpId 1.296: config changed from index 239 to 296, 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198) changed from NON_VOTER to VOTER. New config: { opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false } } } I0703 01:44:13.773207 3445 catalog_manager.cc:3798] T eada0e8b8608424594cabd35cf9891e3 P 55223945fe3d4dd8ac03ba9dab346f54 reported cstate change: config changed from index 239 to 296, 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "55223945fe3d4dd8ac03ba9dab346f54" committed_config { opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I0703 01:44:13.773372 3696 raft_consensus.cc:2684] T eada0e8b8608424594cabd35cf9891e3 P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 FOLLOWER]: Committing config change with OpId 1.296: config changed from index 239 to 296, 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198) changed from NON_VOTER to VOTER. New config: { opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false } } } I0703 01:44:13.775418 5097 raft_consensus.cc:2684] T eada0e8b8608424594cabd35cf9891e3 P 6932f1cd716a48fe8d6c5573acfc48bf [term 1 FOLLOWER]: Committing config change with OpId 1.296: config changed from index 239 to 296, 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198) changed from NON_VOTER to VOTER. New config: { opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false } } } I0703 01:44:13.776125 3445 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet eada0e8b8608424594cabd35cf9891e3 (attempt 1) I0703 01:44:13.776918 3573 consensus_peers.cc:455] T eada0e8b8608424594cabd35cf9891e3 P 55223945fe3d4dd8ac03ba9dab346f54 -> Peer 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198:51033): Closing peer: 6932f1cd716a48fe8d6c5573acfc48bf I0703 01:44:13.776981 3573 consensus_peers.cc:455] T eada0e8b8608424594cabd35cf9891e3 P 55223945fe3d4dd8ac03ba9dab346f54 -> Peer 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746): Closing peer: 57bea6dc182c41a78304417a3cbfc4a7 I0703 01:44:13.777139 3573 consensus_queue.cc:204] T eada0e8b8608424594cabd35cf9891e3 P 55223945fe3d4dd8ac03ba9dab346f54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 297, Committed index: 297, Last appended: 1.298, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 299 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false } } I0703 01:44:13.777669 5528 ts_tablet_manager.cc:1045] T 60d3228294e945c5b343ae62e76dc2f0 P 213596374e4142dca6ecfc701ba1c566: Time spent bootstrapping tablet: real 2.859s user 0.386s sys 0.023s I0703 01:44:13.780550 5528 raft_consensus.cc:325] T 60d3228294e945c5b343ae62e76dc2f0 P 213596374e4142dca6ecfc701ba1c566 [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: true } } I0703 01:44:13.780817 5528 raft_consensus.cc:628] T 60d3228294e945c5b343ae62e76dc2f0 P 213596374e4142dca6ecfc701ba1c566 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 213596374e4142dca6ecfc701ba1c566, State: Initialized, Role: LEARNER I0703 01:44:13.780990 5528 consensus_queue.cc:227] T 60d3228294e945c5b343ae62e76dc2f0 P 213596374e4142dca6ecfc701ba1c566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 311, Last appended: 1.311, Last appended by leader: 311, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 293 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: NON_VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: true } } I0703 01:44:13.783681 5098 raft_consensus.cc:1081] T eada0e8b8608424594cabd35cf9891e3 P 6932f1cd716a48fe8d6c5573acfc48bf [term 1 FOLLOWER]: Refusing update from remote peer 55223945fe3d4dd8ac03ba9dab346f54: Log matching property violated. Preceding OpId in replica: term: 1 index: 297. Preceding OpId from leader: term: 1 index: 299. (index mismatch) I0703 01:44:13.786012 5528 ts_tablet_manager.cc:1073] T 60d3228294e945c5b343ae62e76dc2f0 P 213596374e4142dca6ecfc701ba1c566: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s I0703 01:44:13.787099 5234 raft_consensus.cc:1026] T 674ed5a88db04825a4eaec2fa996a852 P 1157c91cc6f54a68a1d99017ac8444aa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.279->[1.280-1.365] Dedup: 1.288->[1.289-1.365] I0703 01:44:13.787117 3593 tablet_copy_service.cc:325] P 55223945fe3d4dd8ac03ba9dab346f54: Request end of tablet copy session 213596374e4142dca6ecfc701ba1c566-60d3228294e945c5b343ae62e76dc2f0 received from {username='jenkins'} at 127.5.240.196:57370 I0703 01:44:13.787210 3593 tablet_copy_service.cc:413] P 55223945fe3d4dd8ac03ba9dab346f54: ending tablet copy session 213596374e4142dca6ecfc701ba1c566-60d3228294e945c5b343ae62e76dc2f0 on tablet 60d3228294e945c5b343ae62e76dc2f0 with peer 213596374e4142dca6ecfc701ba1c566 I0703 01:44:13.787024 5726 consensus_queue.cc:962] T eada0e8b8608424594cabd35cf9891e3 P 55223945fe3d4dd8ac03ba9dab346f54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 299, Last known committed idx: 297, Time since last communication: 0.009s I0703 01:44:13.811005 5406 raft_consensus.cc:890] T 4a613c1be55c4888a4df36d3f1bfc54c P 57bea6dc182c41a78304417a3cbfc4a7: attempting to promote NON_VOTER 6932f1cd716a48fe8d6c5573acfc48bf to VOTER I0703 01:44:13.811350 5406 consensus_peers.cc:455] T 4a613c1be55c4888a4df36d3f1bfc54c P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198:51033): Closing peer: 6932f1cd716a48fe8d6c5573acfc48bf I0703 01:44:13.811410 5406 consensus_peers.cc:455] T 4a613c1be55c4888a4df36d3f1bfc54c P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064): Closing peer: 8c1e8943f5654f71b030ec7c0b733742 I0703 01:44:13.811635 5406 consensus_queue.cc:204] T 4a613c1be55c4888a4df36d3f1bfc54c P 57bea6dc182c41a78304417a3cbfc4a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 294, Committed index: 294, Last appended: 1.295, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false } } I0703 01:44:13.814575 5098 raft_consensus.cc:1081] T 4a613c1be55c4888a4df36d3f1bfc54c P 6932f1cd716a48fe8d6c5573acfc48bf [term 1 LEARNER]: Refusing update from remote peer 57bea6dc182c41a78304417a3cbfc4a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 294. Preceding OpId from leader: term: 1 index: 296. (index mismatch) I0703 01:44:13.815279 5470 consensus_queue.cc:962] T 4a613c1be55c4888a4df36d3f1bfc54c P 57bea6dc182c41a78304417a3cbfc4a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 296, Last known committed idx: 293, Time since last communication: 0.000s I0703 01:44:13.815917 5579 raft_consensus.cc:2684] T eada0e8b8608424594cabd35cf9891e3 P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 LEADER]: Committing config change with OpId 1.299: config changed from index 296 to 299, VOTER 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194) evicted. New config: { opid_index: 299 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false } } } I0703 01:44:13.818768 3816 raft_consensus.cc:1081] T 4a613c1be55c4888a4df36d3f1bfc54c P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Refusing update from remote peer 57bea6dc182c41a78304417a3cbfc4a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 295. Preceding OpId from leader: term: 1 index: 296. (index mismatch) I0703 01:44:13.819679 3437 catalog_manager.cc:3352] ChangeConfig:REMOVE_PEER RPC for tablet eada0e8b8608424594cabd35cf9891e3 with cas_config_opid_index 296: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0703 01:44:13.820628 3445 catalog_manager.cc:3798] T eada0e8b8608424594cabd35cf9891e3 P 55223945fe3d4dd8ac03ba9dab346f54 reported cstate change: config changed from index 296 to 299, VOTER 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194) evicted. New cstate: current_term: 1 leader_uuid: "55223945fe3d4dd8ac03ba9dab346f54" committed_config { opid_index: 299 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I0703 01:44:13.820863 5470 consensus_queue.cc:962] T 4a613c1be55c4888a4df36d3f1bfc54c P 57bea6dc182c41a78304417a3cbfc4a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 296, Last known committed idx: 294, Time since last communication: 0.000s I0703 01:44:13.823053 3445 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet eada0e8b8608424594cabd35cf9891e3 on 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746) (TS 57bea6dc182c41a78304417a3cbfc4a7 not found in new config with opid_index 299) I0703 01:44:13.824365 3676 tablet_service.cc:800] Processing DeleteTablet for tablet eada0e8b8608424594cabd35cf9891e3 with delete_type TABLET_DATA_TOMBSTONED (TS 57bea6dc182c41a78304417a3cbfc4a7 not found in new config with opid_index 299) from {username='jenkins'} at 127.5.240.194:39884 I0703 01:44:13.824596 5693 tablet_replica.cc:261] T eada0e8b8608424594cabd35cf9891e3 P 57bea6dc182c41a78304417a3cbfc4a7: stopping tablet replica I0703 01:44:13.824707 5693 maintenance_manager.cc:237] P 57bea6dc182c41a78304417a3cbfc4a7: Unregistered op CompactRowSetsOp(eada0e8b8608424594cabd35cf9891e3) I0703 01:44:13.824765 5693 maintenance_manager.cc:237] P 57bea6dc182c41a78304417a3cbfc4a7: Unregistered op MinorDeltaCompactionOp(eada0e8b8608424594cabd35cf9891e3) I0703 01:44:13.824826 5693 maintenance_manager.cc:237] P 57bea6dc182c41a78304417a3cbfc4a7: Unregistered op MajorDeltaCompactionOp(eada0e8b8608424594cabd35cf9891e3) I0703 01:44:13.824890 5693 maintenance_manager.cc:237] P 57bea6dc182c41a78304417a3cbfc4a7: Unregistered op UndoDeltaBlockGCOp(eada0e8b8608424594cabd35cf9891e3) I0703 01:44:13.824959 5693 maintenance_manager.cc:237] P 57bea6dc182c41a78304417a3cbfc4a7: Unregistered op FlushMRSOp(eada0e8b8608424594cabd35cf9891e3) I0703 01:44:13.825001 5693 maintenance_manager.cc:237] P 57bea6dc182c41a78304417a3cbfc4a7: Unregistered op FlushDeltaMemStoresOp(eada0e8b8608424594cabd35cf9891e3) I0703 01:44:13.825042 5693 maintenance_manager.cc:237] P 57bea6dc182c41a78304417a3cbfc4a7: Unregistered op LogGCOp(eada0e8b8608424594cabd35cf9891e3) I0703 01:44:13.825086 5693 raft_consensus.cc:2040] T eada0e8b8608424594cabd35cf9891e3 P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 FOLLOWER]: Raft consensus shutting down. I0703 01:44:13.825203 5693 raft_consensus.cc:2067] T eada0e8b8608424594cabd35cf9891e3 P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 FOLLOWER]: Raft consensus is shut down! I0703 01:44:13.826088 5693 ts_tablet_manager.cc:1369] T eada0e8b8608424594cabd35cf9891e3 P 57bea6dc182c41a78304417a3cbfc4a7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:44:13.828045 5470 raft_consensus.cc:2684] T 4a613c1be55c4888a4df36d3f1bfc54c P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 LEADER]: Committing config change with OpId 1.296: config changed from index 223 to 296, 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198) changed from NON_VOTER to VOTER. New config: { opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false } } } I0703 01:44:13.828186 5096 raft_consensus.cc:2684] T eada0e8b8608424594cabd35cf9891e3 P 6932f1cd716a48fe8d6c5573acfc48bf [term 1 FOLLOWER]: Committing config change with OpId 1.299: config changed from index 296 to 299, VOTER 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194) evicted. New config: { opid_index: 299 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false } } } I0703 01:44:13.829313 5097 raft_consensus.cc:1026] T 4a613c1be55c4888a4df36d3f1bfc54c P 6932f1cd716a48fe8d6c5573acfc48bf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.294->[1.295-1.296] Dedup: 1.295->[1.296-1.296] I0703 01:44:13.832139 3445 catalog_manager.cc:3798] T 4a613c1be55c4888a4df36d3f1bfc54c P 57bea6dc182c41a78304417a3cbfc4a7 reported cstate change: config changed from index 223 to 296, 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "57bea6dc182c41a78304417a3cbfc4a7" committed_config { opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false } health_report { overall_health: UNKNOWN } } } I0703 01:44:13.832788 5693 ts_tablet_manager.cc:1382] T eada0e8b8608424594cabd35cf9891e3 P 57bea6dc182c41a78304417a3cbfc4a7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.297 I0703 01:44:13.832873 5693 log.cc:986] T eada0e8b8608424594cabd35cf9891e3 P 57bea6dc182c41a78304417a3cbfc4a7: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal/wals/eada0e8b8608424594cabd35cf9891e3 I0703 01:44:13.833586 3434 catalog_manager.cc:3103] TS 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746): tablet eada0e8b8608424594cabd35cf9891e3 (table rebalance_test_table_0 [id=5bc38732cb8746a7ad1c2b9fede5e7d9]) successfully deleted I0703 01:44:13.837597 3816 raft_consensus.cc:2684] T 4a613c1be55c4888a4df36d3f1bfc54c P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Committing config change with OpId 1.296: config changed from index 223 to 296, 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198) changed from NON_VOTER to VOTER. New config: { opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false } } } I0703 01:44:13.843586 5098 raft_consensus.cc:2684] T 4a613c1be55c4888a4df36d3f1bfc54c P 6932f1cd716a48fe8d6c5573acfc48bf [term 1 FOLLOWER]: Committing config change with OpId 1.296: config changed from index 223 to 296, 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198) changed from NON_VOTER to VOTER. New config: { opid_index: 296 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false } } } I0703 01:44:13.845546 3445 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 4a613c1be55c4888a4df36d3f1bfc54c (attempt 1) I0703 01:44:13.848743 3696 consensus_peers.cc:455] T 4a613c1be55c4888a4df36d3f1bfc54c P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198:51033): Closing peer: 6932f1cd716a48fe8d6c5573acfc48bf I0703 01:44:13.848810 3696 consensus_peers.cc:455] T 4a613c1be55c4888a4df36d3f1bfc54c P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064): Closing peer: 8c1e8943f5654f71b030ec7c0b733742 I0703 01:44:13.848974 3696 consensus_queue.cc:204] T 4a613c1be55c4888a4df36d3f1bfc54c P 57bea6dc182c41a78304417a3cbfc4a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 296, Committed index: 296, Last appended: 1.296, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false } } I0703 01:44:13.866194 5095 raft_consensus.cc:1081] T 4a613c1be55c4888a4df36d3f1bfc54c P 6932f1cd716a48fe8d6c5573acfc48bf [term 1 FOLLOWER]: Refusing update from remote peer 57bea6dc182c41a78304417a3cbfc4a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 296. Preceding OpId from leader: term: 1 index: 297. (index mismatch) I0703 01:44:13.882436 5731 consensus_queue.cc:962] T 4a613c1be55c4888a4df36d3f1bfc54c P 57bea6dc182c41a78304417a3cbfc4a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 297, Last known committed idx: 296, Time since last communication: 0.000s I0703 01:44:13.887732 4834 raft_consensus.cc:1026] T 60d3228294e945c5b343ae62e76dc2f0 P 213596374e4142dca6ecfc701ba1c566 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.292->[1.293-1.360] Dedup: 1.311->[1.312-1.360] W0703 01:44:13.896183 3504 consensus_peers.cc:433] T 2051955b4b644a03a9adad816d350596 P 55223945fe3d4dd8ac03ba9dab346f54 -> Peer 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377): Couldn't send request to peer 1157c91cc6f54a68a1d99017ac8444aa for tablet 2051955b4b644a03a9adad816d350596. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 7 times. I0703 01:44:13.909204 4966 raft_consensus.cc:1026] T 049bd2ef267f47fca634858420ac35b9 P aa8b9c58b466436faa5086e85b5224a7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.227->[1.228-1.283] Dedup: 1.238->[1.239-1.283] I0703 01:44:13.919353 5406 raft_consensus.cc:2684] T 4a613c1be55c4888a4df36d3f1bfc54c P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 LEADER]: Committing config change with OpId 1.297: config changed from index 296 to 297, VOTER 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195) evicted. New config: { opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false } } } I0703 01:44:13.923110 3434 catalog_manager.cc:3352] ChangeConfig:REMOVE_PEER RPC for tablet 4a613c1be55c4888a4df36d3f1bfc54c with cas_config_opid_index 296: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0703 01:44:13.924000 3445 catalog_manager.cc:3798] T 4a613c1be55c4888a4df36d3f1bfc54c P 57bea6dc182c41a78304417a3cbfc4a7 reported cstate change: config changed from index 296 to 297, VOTER 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195) evicted. New cstate: current_term: 1 leader_uuid: "57bea6dc182c41a78304417a3cbfc4a7" committed_config { opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I0703 01:44:13.927278 3445 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 4a613c1be55c4888a4df36d3f1bfc54c on 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064) (TS 8c1e8943f5654f71b030ec7c0b733742 not found in new config with opid_index 297) I0703 01:44:13.927819 3796 tablet_service.cc:800] Processing DeleteTablet for tablet 4a613c1be55c4888a4df36d3f1bfc54c with delete_type TABLET_DATA_TOMBSTONED (TS 8c1e8943f5654f71b030ec7c0b733742 not found in new config with opid_index 297) from {username='jenkins'} at 127.5.240.195:39619 I0703 01:44:13.932818 5095 raft_consensus.cc:2684] T 4a613c1be55c4888a4df36d3f1bfc54c P 6932f1cd716a48fe8d6c5573acfc48bf [term 1 FOLLOWER]: Committing config change with OpId 1.297: config changed from index 296 to 297, VOTER 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195) evicted. New config: { opid_index: 297 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false } } } I0703 01:44:13.996830 5744 tablet_replica.cc:261] T 4a613c1be55c4888a4df36d3f1bfc54c P 8c1e8943f5654f71b030ec7c0b733742: stopping tablet replica I0703 01:44:13.997241 5744 maintenance_manager.cc:237] P 8c1e8943f5654f71b030ec7c0b733742: Unregistered op CompactRowSetsOp(4a613c1be55c4888a4df36d3f1bfc54c) I0703 01:44:14.000797 5744 maintenance_manager.cc:237] P 8c1e8943f5654f71b030ec7c0b733742: Unregistered op MinorDeltaCompactionOp(4a613c1be55c4888a4df36d3f1bfc54c) I0703 01:44:13.999894 5154 raft_consensus.cc:890] T 674ed5a88db04825a4eaec2fa996a852 P 8c1e8943f5654f71b030ec7c0b733742: attempting to promote NON_VOTER 1157c91cc6f54a68a1d99017ac8444aa to VOTER I0703 01:44:14.001746 5154 consensus_peers.cc:455] T 674ed5a88db04825a4eaec2fa996a852 P 8c1e8943f5654f71b030ec7c0b733742 -> Peer 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377): Closing peer: 1157c91cc6f54a68a1d99017ac8444aa I0703 01:44:14.001817 5154 consensus_peers.cc:455] T 674ed5a88db04825a4eaec2fa996a852 P 8c1e8943f5654f71b030ec7c0b733742 -> Peer 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193:54919): Closing peer: 55223945fe3d4dd8ac03ba9dab346f54 I0703 01:44:14.002091 5154 consensus_queue.cc:204] T 674ed5a88db04825a4eaec2fa996a852 P 8c1e8943f5654f71b030ec7c0b733742 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 369, Committed index: 369, Last appended: 1.369, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 370 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: false } } I0703 01:44:14.002279 5744 maintenance_manager.cc:237] P 8c1e8943f5654f71b030ec7c0b733742: Unregistered op MajorDeltaCompactionOp(4a613c1be55c4888a4df36d3f1bfc54c) I0703 01:44:14.002411 5744 maintenance_manager.cc:237] P 8c1e8943f5654f71b030ec7c0b733742: Unregistered op UndoDeltaBlockGCOp(4a613c1be55c4888a4df36d3f1bfc54c) I0703 01:44:14.004125 3573 raft_consensus.cc:1081] T 674ed5a88db04825a4eaec2fa996a852 P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 FOLLOWER]: Refusing update from remote peer 8c1e8943f5654f71b030ec7c0b733742: Log matching property violated. Preceding OpId in replica: term: 1 index: 369. Preceding OpId from leader: term: 1 index: 370. (index mismatch) I0703 01:44:14.005334 5025 consensus_queue.cc:962] T 674ed5a88db04825a4eaec2fa996a852 P 8c1e8943f5654f71b030ec7c0b733742 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 370, Last known committed idx: 369, Time since last communication: 0.000s I0703 01:44:14.006667 5744 maintenance_manager.cc:237] P 8c1e8943f5654f71b030ec7c0b733742: Unregistered op FlushMRSOp(4a613c1be55c4888a4df36d3f1bfc54c) I0703 01:44:14.006811 5744 maintenance_manager.cc:237] P 8c1e8943f5654f71b030ec7c0b733742: Unregistered op FlushDeltaMemStoresOp(4a613c1be55c4888a4df36d3f1bfc54c) I0703 01:44:14.007242 5744 maintenance_manager.cc:237] P 8c1e8943f5654f71b030ec7c0b733742: Unregistered op LogGCOp(4a613c1be55c4888a4df36d3f1bfc54c) I0703 01:44:14.007342 5520 tablet_bootstrap.cc:427] T 2051955b4b644a03a9adad816d350596 P 1157c91cc6f54a68a1d99017ac8444aa: Bootstrap replayed 1/1 log segments. Stats: ops{read=268 overwritten=0 applied=268 ignored=0} inserts{seen=917 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:44:14.007339 5744 raft_consensus.cc:2040] T 4a613c1be55c4888a4df36d3f1bfc54c P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Raft consensus shutting down. I0703 01:44:14.007545 5744 raft_consensus.cc:2067] T 4a613c1be55c4888a4df36d3f1bfc54c P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Raft consensus is shut down! I0703 01:44:14.009820 5520 tablet_bootstrap.cc:427] T 2051955b4b644a03a9adad816d350596 P 1157c91cc6f54a68a1d99017ac8444aa: Bootstrap complete. I0703 01:44:14.009712 5025 raft_consensus.cc:2684] T 674ed5a88db04825a4eaec2fa996a852 P 8c1e8943f5654f71b030ec7c0b733742 [term 1 LEADER]: Committing config change with OpId 1.370: config changed from index 280 to 370, 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199) changed from NON_VOTER to VOTER. New config: { opid_index: 370 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: false } } } I0703 01:44:14.010059 5520 ts_tablet_manager.cc:1045] T 2051955b4b644a03a9adad816d350596 P 1157c91cc6f54a68a1d99017ac8444aa: Time spent bootstrapping tablet: real 3.232s user 0.337s sys 0.030s I0703 01:44:14.010341 3573 raft_consensus.cc:2684] T 674ed5a88db04825a4eaec2fa996a852 P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 FOLLOWER]: Committing config change with OpId 1.370: config changed from index 280 to 370, 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199) changed from NON_VOTER to VOTER. New config: { opid_index: 370 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: false } } } I0703 01:44:14.010951 5520 raft_consensus.cc:325] T 2051955b4b644a03a9adad816d350596 P 1157c91cc6f54a68a1d99017ac8444aa [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: true } } I0703 01:44:14.011231 5520 raft_consensus.cc:628] T 2051955b4b644a03a9adad816d350596 P 1157c91cc6f54a68a1d99017ac8444aa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1157c91cc6f54a68a1d99017ac8444aa, State: Initialized, Role: LEARNER I0703 01:44:14.011396 5520 consensus_queue.cc:227] T 2051955b4b644a03a9adad816d350596 P 1157c91cc6f54a68a1d99017ac8444aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 268, Last appended: 1.268, Last appended by leader: 268, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: true } } I0703 01:44:14.013187 5520 ts_tablet_manager.cc:1073] T 2051955b4b644a03a9adad816d350596 P 1157c91cc6f54a68a1d99017ac8444aa: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s I0703 01:44:14.013813 3445 catalog_manager.cc:3798] T 674ed5a88db04825a4eaec2fa996a852 P 8c1e8943f5654f71b030ec7c0b733742 reported cstate change: config changed from index 280 to 370, 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8c1e8943f5654f71b030ec7c0b733742" committed_config { opid_index: 370 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: false } health_report { overall_health: UNKNOWN } } } I0703 01:44:14.014281 3593 tablet_copy_service.cc:325] P 55223945fe3d4dd8ac03ba9dab346f54: Request end of tablet copy session 1157c91cc6f54a68a1d99017ac8444aa-2051955b4b644a03a9adad816d350596 received from {username='jenkins'} at 127.5.240.199:60222 I0703 01:44:14.014358 3593 tablet_copy_service.cc:413] P 55223945fe3d4dd8ac03ba9dab346f54: ending tablet copy session 1157c91cc6f54a68a1d99017ac8444aa-2051955b4b644a03a9adad816d350596 on tablet 2051955b4b644a03a9adad816d350596 with peer 1157c91cc6f54a68a1d99017ac8444aa I0703 01:44:14.017122 5235 raft_consensus.cc:1081] T 674ed5a88db04825a4eaec2fa996a852 P 1157c91cc6f54a68a1d99017ac8444aa [term 1 LEARNER]: Refusing update from remote peer 8c1e8943f5654f71b030ec7c0b733742: Log matching property violated. Preceding OpId in replica: term: 1 index: 369. Preceding OpId from leader: term: 1 index: 370. (index mismatch) I0703 01:44:14.018179 5744 ts_tablet_manager.cc:1369] T 4a613c1be55c4888a4df36d3f1bfc54c P 8c1e8943f5654f71b030ec7c0b733742: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:44:14.023252 5154 consensus_queue.cc:962] T 674ed5a88db04825a4eaec2fa996a852 P 8c1e8943f5654f71b030ec7c0b733742 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 370, Last known committed idx: 369, Time since last communication: 0.000s I0703 01:44:14.042448 5235 raft_consensus.cc:2684] T 674ed5a88db04825a4eaec2fa996a852 P 1157c91cc6f54a68a1d99017ac8444aa [term 1 FOLLOWER]: Committing config change with OpId 1.370: config changed from index 280 to 370, 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199) changed from NON_VOTER to VOTER. New config: { opid_index: 370 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: false } } } I0703 01:44:14.046207 5726 raft_consensus.cc:890] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54: attempting to promote NON_VOTER 213596374e4142dca6ecfc701ba1c566 to VOTER I0703 01:44:14.046569 5726 consensus_peers.cc:455] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 -> Peer 213596374e4142dca6ecfc701ba1c566 (127.5.240.196:51288): Closing peer: 213596374e4142dca6ecfc701ba1c566 I0703 01:44:14.046653 5726 consensus_peers.cc:455] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 -> Peer 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064): Closing peer: 8c1e8943f5654f71b030ec7c0b733742 I0703 01:44:14.046833 5726 consensus_queue.cc:204] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 362, Committed index: 362, Last appended: 1.362, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 363 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } } I0703 01:44:14.050830 4835 raft_consensus.cc:1081] T 60d3228294e945c5b343ae62e76dc2f0 P 213596374e4142dca6ecfc701ba1c566 [term 1 LEARNER]: Refusing update from remote peer 55223945fe3d4dd8ac03ba9dab346f54: Log matching property violated. Preceding OpId in replica: term: 1 index: 362. Preceding OpId from leader: term: 1 index: 363. (index mismatch) I0703 01:44:14.053292 5492 consensus_queue.cc:962] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 363, Last known committed idx: 362, Time since last communication: 0.000s I0703 01:44:14.054200 3446 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 674ed5a88db04825a4eaec2fa996a852 (attempt 1) I0703 01:44:14.056815 3816 consensus_peers.cc:455] T 674ed5a88db04825a4eaec2fa996a852 P 8c1e8943f5654f71b030ec7c0b733742 -> Peer 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377): Closing peer: 1157c91cc6f54a68a1d99017ac8444aa I0703 01:44:14.056934 3816 consensus_peers.cc:455] T 674ed5a88db04825a4eaec2fa996a852 P 8c1e8943f5654f71b030ec7c0b733742 -> Peer 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193:54919): Closing peer: 55223945fe3d4dd8ac03ba9dab346f54 I0703 01:44:14.057279 3816 consensus_queue.cc:204] T 674ed5a88db04825a4eaec2fa996a852 P 8c1e8943f5654f71b030ec7c0b733742 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 371, Committed index: 371, Last appended: 1.371, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 372 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: false } } I0703 01:44:14.059828 5492 raft_consensus.cc:2684] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 LEADER]: Committing config change with OpId 1.363: config changed from index 293 to 363, 213596374e4142dca6ecfc701ba1c566 (127.5.240.196) changed from NON_VOTER to VOTER. New config: { opid_index: 363 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } } } I0703 01:44:14.060288 3816 raft_consensus.cc:1081] T 60d3228294e945c5b343ae62e76dc2f0 P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Refusing update from remote peer 55223945fe3d4dd8ac03ba9dab346f54: Log matching property violated. Preceding OpId in replica: term: 1 index: 362. Preceding OpId from leader: term: 1 index: 363. (index mismatch) I0703 01:44:14.060500 5234 raft_consensus.cc:1081] T 674ed5a88db04825a4eaec2fa996a852 P 1157c91cc6f54a68a1d99017ac8444aa [term 1 FOLLOWER]: Refusing update from remote peer 8c1e8943f5654f71b030ec7c0b733742: Log matching property violated. Preceding OpId in replica: term: 1 index: 370. Preceding OpId from leader: term: 1 index: 372. (index mismatch) I0703 01:44:14.060663 5744 ts_tablet_manager.cc:1382] T 4a613c1be55c4888a4df36d3f1bfc54c P 8c1e8943f5654f71b030ec7c0b733742: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.296 I0703 01:44:14.060948 5744 log.cc:986] T 4a613c1be55c4888a4df36d3f1bfc54c P 8c1e8943f5654f71b030ec7c0b733742: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal/wals/4a613c1be55c4888a4df36d3f1bfc54c I0703 01:44:14.061807 3437 catalog_manager.cc:3103] TS 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064): tablet 4a613c1be55c4888a4df36d3f1bfc54c (table rebalance_test_table_4 [id=d5e3657f924944ca895ef4125aaa82a1]) successfully deleted I0703 01:44:14.063879 5726 consensus_queue.cc:962] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 363, Last known committed idx: 362, Time since last communication: 0.000s I0703 01:44:14.063890 5154 consensus_queue.cc:962] T 674ed5a88db04825a4eaec2fa996a852 P 8c1e8943f5654f71b030ec7c0b733742 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 372, Last known committed idx: 370, Time since last communication: 0.000s I0703 01:44:14.064105 4835 raft_consensus.cc:2684] T 60d3228294e945c5b343ae62e76dc2f0 P 213596374e4142dca6ecfc701ba1c566 [term 1 FOLLOWER]: Committing config change with OpId 1.363: config changed from index 293 to 363, 213596374e4142dca6ecfc701ba1c566 (127.5.240.196) changed from NON_VOTER to VOTER. New config: { opid_index: 363 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } } } I0703 01:44:14.065445 5235 raft_consensus.cc:1026] T 674ed5a88db04825a4eaec2fa996a852 P 1157c91cc6f54a68a1d99017ac8444aa [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.370->[1.371-1.373] Dedup: 1.371->[1.372-1.373] I0703 01:44:14.067162 3446 catalog_manager.cc:3798] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 reported cstate change: config changed from index 293 to 363, 213596374e4142dca6ecfc701ba1c566 (127.5.240.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "55223945fe3d4dd8ac03ba9dab346f54" committed_config { opid_index: 363 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I0703 01:44:14.067693 3816 raft_consensus.cc:2684] T 60d3228294e945c5b343ae62e76dc2f0 P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Committing config change with OpId 1.363: config changed from index 293 to 363, 213596374e4142dca6ecfc701ba1c566 (127.5.240.196) changed from NON_VOTER to VOTER. New config: { opid_index: 363 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } } } I0703 01:44:14.070430 3446 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 60d3228294e945c5b343ae62e76dc2f0 (attempt 1) I0703 01:44:14.072127 3573 consensus_peers.cc:455] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 -> Peer 213596374e4142dca6ecfc701ba1c566 (127.5.240.196:51288): Closing peer: 213596374e4142dca6ecfc701ba1c566 I0703 01:44:14.072201 3573 consensus_peers.cc:455] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 -> Peer 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064): Closing peer: 8c1e8943f5654f71b030ec7c0b733742 I0703 01:44:14.072412 3573 consensus_queue.cc:204] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 363, Committed index: 363, Last appended: 1.363, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 364 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } } I0703 01:44:14.074271 4834 raft_consensus.cc:1081] T 60d3228294e945c5b343ae62e76dc2f0 P 213596374e4142dca6ecfc701ba1c566 [term 1 FOLLOWER]: Refusing update from remote peer 55223945fe3d4dd8ac03ba9dab346f54: Log matching property violated. Preceding OpId in replica: term: 1 index: 363. Preceding OpId from leader: term: 1 index: 364. (index mismatch) I0703 01:44:14.075798 5154 raft_consensus.cc:2684] T 674ed5a88db04825a4eaec2fa996a852 P 8c1e8943f5654f71b030ec7c0b733742 [term 1 LEADER]: Committing config change with OpId 1.372: config changed from index 370 to 372, VOTER 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193) evicted. New config: { opid_index: 372 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: false } } } I0703 01:44:14.076126 5726 consensus_queue.cc:962] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 364, Last known committed idx: 363, Time since last communication: 0.000s I0703 01:44:14.078856 3437 catalog_manager.cc:3352] ChangeConfig:REMOVE_PEER RPC for tablet 674ed5a88db04825a4eaec2fa996a852 with cas_config_opid_index 370: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0703 01:44:14.080026 5587 raft_consensus.cc:890] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7: attempting to promote NON_VOTER aa8b9c58b466436faa5086e85b5224a7 to VOTER I0703 01:44:14.080319 5587 consensus_peers.cc:455] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197:51329): Closing peer: aa8b9c58b466436faa5086e85b5224a7 I0703 01:44:14.080416 5587 consensus_peers.cc:455] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193:54919): Closing peer: 55223945fe3d4dd8ac03ba9dab346f54 I0703 01:44:14.080189 5581 raft_consensus.cc:2684] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 LEADER]: Committing config change with OpId 1.364: config changed from index 363 to 364, VOTER 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195) evicted. New config: { opid_index: 364 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } } } I0703 01:44:14.080608 5587 consensus_queue.cc:204] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 284, Committed index: 284, Last appended: 1.285, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 286 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } I0703 01:44:14.083719 4965 raft_consensus.cc:1081] T 049bd2ef267f47fca634858420ac35b9 P aa8b9c58b466436faa5086e85b5224a7 [term 1 LEARNER]: Refusing update from remote peer 57bea6dc182c41a78304417a3cbfc4a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 285. Preceding OpId from leader: term: 1 index: 286. (index mismatch) I0703 01:44:14.083547 3446 catalog_manager.cc:3798] T 674ed5a88db04825a4eaec2fa996a852 P 8c1e8943f5654f71b030ec7c0b733742 reported cstate change: config changed from index 370 to 372, VOTER 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193) evicted. New cstate: current_term: 1 leader_uuid: "8c1e8943f5654f71b030ec7c0b733742" committed_config { opid_index: 372 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I0703 01:44:14.084246 3437 catalog_manager.cc:3352] ChangeConfig:REMOVE_PEER RPC for tablet 60d3228294e945c5b343ae62e76dc2f0 with cas_config_opid_index 363: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0703 01:44:14.084440 3445 catalog_manager.cc:3798] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 reported cstate change: config changed from index 363 to 364, VOTER 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195) evicted. New cstate: current_term: 1 leader_uuid: "55223945fe3d4dd8ac03ba9dab346f54" committed_config { opid_index: 364 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I0703 01:44:14.084691 5587 consensus_queue.cc:962] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 286, Last known committed idx: 284, Time since last communication: 0.000s I0703 01:44:14.085836 4834 raft_consensus.cc:2684] T 60d3228294e945c5b343ae62e76dc2f0 P 213596374e4142dca6ecfc701ba1c566 [term 1 FOLLOWER]: Committing config change with OpId 1.364: config changed from index 363 to 364, VOTER 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195) evicted. New config: { opid_index: 364 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } } } I0703 01:44:14.087834 3445 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 60d3228294e945c5b343ae62e76dc2f0 on 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064) (TS 8c1e8943f5654f71b030ec7c0b733742 not found in new config with opid_index 364) I0703 01:44:14.088234 3446 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 674ed5a88db04825a4eaec2fa996a852 on 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193:54919) (TS 55223945fe3d4dd8ac03ba9dab346f54 not found in new config with opid_index 372) I0703 01:44:14.089248 3553 tablet_service.cc:800] Processing DeleteTablet for tablet 674ed5a88db04825a4eaec2fa996a852 with delete_type TABLET_DATA_TOMBSTONED (TS 55223945fe3d4dd8ac03ba9dab346f54 not found in new config with opid_index 372) from {username='jenkins'} at 127.5.240.193:52692 I0703 01:44:14.089732 3796 tablet_service.cc:800] Processing DeleteTablet for tablet 60d3228294e945c5b343ae62e76dc2f0 with delete_type TABLET_DATA_TOMBSTONED (TS 8c1e8943f5654f71b030ec7c0b733742 not found in new config with opid_index 364) from {username='jenkins'} at 127.5.240.195:39619 I0703 01:44:14.091934 5406 raft_consensus.cc:2684] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 LEADER]: Committing config change with OpId 1.286: config changed from index 228 to 286, aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197) changed from NON_VOTER to VOTER. New config: { opid_index: 286 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } } I0703 01:44:14.092912 5744 tablet_replica.cc:261] T 60d3228294e945c5b343ae62e76dc2f0 P 8c1e8943f5654f71b030ec7c0b733742: stopping tablet replica I0703 01:44:14.093261 5744 maintenance_manager.cc:237] P 8c1e8943f5654f71b030ec7c0b733742: Unregistered op CompactRowSetsOp(60d3228294e945c5b343ae62e76dc2f0) I0703 01:44:14.093575 5744 maintenance_manager.cc:237] P 8c1e8943f5654f71b030ec7c0b733742: Unregistered op MinorDeltaCompactionOp(60d3228294e945c5b343ae62e76dc2f0) I0703 01:44:14.093863 5744 maintenance_manager.cc:237] P 8c1e8943f5654f71b030ec7c0b733742: Unregistered op MajorDeltaCompactionOp(60d3228294e945c5b343ae62e76dc2f0) I0703 01:44:14.094128 5744 maintenance_manager.cc:237] P 8c1e8943f5654f71b030ec7c0b733742: Unregistered op UndoDeltaBlockGCOp(60d3228294e945c5b343ae62e76dc2f0) I0703 01:44:14.094368 5744 maintenance_manager.cc:237] P 8c1e8943f5654f71b030ec7c0b733742: Unregistered op FlushMRSOp(60d3228294e945c5b343ae62e76dc2f0) I0703 01:44:14.094569 5744 maintenance_manager.cc:237] P 8c1e8943f5654f71b030ec7c0b733742: Unregistered op FlushDeltaMemStoresOp(60d3228294e945c5b343ae62e76dc2f0) I0703 01:44:14.094795 5744 maintenance_manager.cc:237] P 8c1e8943f5654f71b030ec7c0b733742: Unregistered op LogGCOp(60d3228294e945c5b343ae62e76dc2f0) I0703 01:44:14.094947 5744 raft_consensus.cc:2040] T 60d3228294e945c5b343ae62e76dc2f0 P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Raft consensus shutting down. I0703 01:44:14.095214 5744 raft_consensus.cc:2067] T 60d3228294e945c5b343ae62e76dc2f0 P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Raft consensus is shut down! I0703 01:44:14.096138 5744 ts_tablet_manager.cc:1369] T 60d3228294e945c5b343ae62e76dc2f0 P 8c1e8943f5654f71b030ec7c0b733742: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:44:14.096386 3445 catalog_manager.cc:3798] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 reported cstate change: config changed from index 228 to 286, aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "57bea6dc182c41a78304417a3cbfc4a7" committed_config { opid_index: 286 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I0703 01:44:14.097373 3570 raft_consensus.cc:1081] T 049bd2ef267f47fca634858420ac35b9 P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 FOLLOWER]: Refusing update from remote peer 57bea6dc182c41a78304417a3cbfc4a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 285. Preceding OpId from leader: term: 1 index: 286. (index mismatch) I0703 01:44:14.097194 4966 raft_consensus.cc:2684] T 049bd2ef267f47fca634858420ac35b9 P aa8b9c58b466436faa5086e85b5224a7 [term 1 FOLLOWER]: Committing config change with OpId 1.286: config changed from index 228 to 286, aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197) changed from NON_VOTER to VOTER. New config: { opid_index: 286 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } } I0703 01:44:14.097440 5235 raft_consensus.cc:2684] T 674ed5a88db04825a4eaec2fa996a852 P 1157c91cc6f54a68a1d99017ac8444aa [term 1 FOLLOWER]: Committing config change with OpId 1.372: config changed from index 370 to 372, VOTER 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193) evicted. New config: { opid_index: 372 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: false } } } I0703 01:44:14.101143 3445 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 049bd2ef267f47fca634858420ac35b9 (attempt 1) I0703 01:44:14.102645 3696 consensus_peers.cc:455] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197:51329): Closing peer: aa8b9c58b466436faa5086e85b5224a7 I0703 01:44:14.102749 3696 consensus_peers.cc:455] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193:54919): Closing peer: 55223945fe3d4dd8ac03ba9dab346f54 I0703 01:44:14.102974 3696 consensus_queue.cc:204] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 286, Committed index: 286, Last appended: 1.287, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } I0703 01:44:14.112026 4965 raft_consensus.cc:1081] T 049bd2ef267f47fca634858420ac35b9 P aa8b9c58b466436faa5086e85b5224a7 [term 1 FOLLOWER]: Refusing update from remote peer 57bea6dc182c41a78304417a3cbfc4a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 286. Preceding OpId from leader: term: 1 index: 288. (index mismatch) I0703 01:44:14.112990 5587 consensus_queue.cc:962] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 288, Last known committed idx: 286, Time since last communication: 0.000s I0703 01:44:14.117478 5470 raft_consensus.cc:2684] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 LEADER]: Committing config change with OpId 1.288: config changed from index 286 to 288, VOTER 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193) evicted. New config: { opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } } I0703 01:44:14.121042 3434 catalog_manager.cc:3352] ChangeConfig:REMOVE_PEER RPC for tablet 049bd2ef267f47fca634858420ac35b9 with cas_config_opid_index 286: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0703 01:44:14.121557 5754 tablet_replica.cc:261] T 674ed5a88db04825a4eaec2fa996a852 P 55223945fe3d4dd8ac03ba9dab346f54: stopping tablet replica I0703 01:44:14.124338 3446 catalog_manager.cc:3798] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 reported cstate change: config changed from index 286 to 288, VOTER 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193) evicted. New cstate: current_term: 1 leader_uuid: "57bea6dc182c41a78304417a3cbfc4a7" committed_config { opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I0703 01:44:14.125252 4966 raft_consensus.cc:2684] T 049bd2ef267f47fca634858420ac35b9 P aa8b9c58b466436faa5086e85b5224a7 [term 1 FOLLOWER]: Committing config change with OpId 1.288: config changed from index 286 to 288, VOTER 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193) evicted. New config: { opid_index: 288 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } } I0703 01:44:14.121686 5754 maintenance_manager.cc:237] P 55223945fe3d4dd8ac03ba9dab346f54: Unregistered op CompactRowSetsOp(674ed5a88db04825a4eaec2fa996a852) I0703 01:44:14.132016 5754 maintenance_manager.cc:237] P 55223945fe3d4dd8ac03ba9dab346f54: Unregistered op MinorDeltaCompactionOp(674ed5a88db04825a4eaec2fa996a852) I0703 01:44:14.132117 5754 maintenance_manager.cc:237] P 55223945fe3d4dd8ac03ba9dab346f54: Unregistered op MajorDeltaCompactionOp(674ed5a88db04825a4eaec2fa996a852) I0703 01:44:14.132216 5754 maintenance_manager.cc:237] P 55223945fe3d4dd8ac03ba9dab346f54: Unregistered op UndoDeltaBlockGCOp(674ed5a88db04825a4eaec2fa996a852) I0703 01:44:14.132308 5754 maintenance_manager.cc:237] P 55223945fe3d4dd8ac03ba9dab346f54: Unregistered op FlushMRSOp(674ed5a88db04825a4eaec2fa996a852) I0703 01:44:14.132349 3446 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 049bd2ef267f47fca634858420ac35b9 on 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193:54919) (TS 55223945fe3d4dd8ac03ba9dab346f54 not found in new config with opid_index 288) I0703 01:44:14.132428 5754 maintenance_manager.cc:237] P 55223945fe3d4dd8ac03ba9dab346f54: Unregistered op FlushDeltaMemStoresOp(674ed5a88db04825a4eaec2fa996a852) I0703 01:44:14.132884 3553 tablet_service.cc:800] Processing DeleteTablet for tablet 049bd2ef267f47fca634858420ac35b9 with delete_type TABLET_DATA_TOMBSTONED (TS 55223945fe3d4dd8ac03ba9dab346f54 not found in new config with opid_index 288) from {username='jenkins'} at 127.5.240.193:52692 I0703 01:44:14.133219 5754 maintenance_manager.cc:237] P 55223945fe3d4dd8ac03ba9dab346f54: Unregistered op LogGCOp(674ed5a88db04825a4eaec2fa996a852) I0703 01:44:14.133312 5754 raft_consensus.cc:2040] T 674ed5a88db04825a4eaec2fa996a852 P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 FOLLOWER]: Raft consensus shutting down. I0703 01:44:14.137130 5754 raft_consensus.cc:2067] T 674ed5a88db04825a4eaec2fa996a852 P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 FOLLOWER]: Raft consensus is shut down! I0703 01:44:14.137821 5744 ts_tablet_manager.cc:1382] T 60d3228294e945c5b343ae62e76dc2f0 P 8c1e8943f5654f71b030ec7c0b733742: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.363 I0703 01:44:14.137903 5744 log.cc:986] T 60d3228294e945c5b343ae62e76dc2f0 P 8c1e8943f5654f71b030ec7c0b733742: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal/wals/60d3228294e945c5b343ae62e76dc2f0 I0703 01:44:14.138187 5754 ts_tablet_manager.cc:1369] T 674ed5a88db04825a4eaec2fa996a852 P 55223945fe3d4dd8ac03ba9dab346f54: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:44:14.143023 3437 catalog_manager.cc:3103] TS 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064): tablet 60d3228294e945c5b343ae62e76dc2f0 (table rebalance_test_table_3 [id=2d96fe2971a440c3b073db7c0073362b]) successfully deleted I0703 01:44:14.149204 4835 raft_consensus.cc:1026] T 930bbe4ac3234ed49a2207c4902541f1 P 213596374e4142dca6ecfc701ba1c566 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.267->[1.268-1.331] Dedup: 1.281->[1.282-1.331] I0703 01:44:14.161059 5754 ts_tablet_manager.cc:1382] T 674ed5a88db04825a4eaec2fa996a852 P 55223945fe3d4dd8ac03ba9dab346f54: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.371 I0703 01:44:14.161201 5754 log.cc:986] T 674ed5a88db04825a4eaec2fa996a852 P 55223945fe3d4dd8ac03ba9dab346f54: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal/wals/674ed5a88db04825a4eaec2fa996a852 I0703 01:44:14.163882 3437 catalog_manager.cc:3103] TS 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193:54919): tablet 674ed5a88db04825a4eaec2fa996a852 (table rebalance_test_table_3 [id=2d96fe2971a440c3b073db7c0073362b]) successfully deleted I0703 01:44:14.164207 5754 tablet_replica.cc:261] T 049bd2ef267f47fca634858420ac35b9 P 55223945fe3d4dd8ac03ba9dab346f54: stopping tablet replica I0703 01:44:14.164331 5754 maintenance_manager.cc:237] P 55223945fe3d4dd8ac03ba9dab346f54: Unregistered op CompactRowSetsOp(049bd2ef267f47fca634858420ac35b9) I0703 01:44:14.164443 5754 maintenance_manager.cc:237] P 55223945fe3d4dd8ac03ba9dab346f54: Unregistered op MinorDeltaCompactionOp(049bd2ef267f47fca634858420ac35b9) I0703 01:44:14.164563 5754 maintenance_manager.cc:237] P 55223945fe3d4dd8ac03ba9dab346f54: Unregistered op MajorDeltaCompactionOp(049bd2ef267f47fca634858420ac35b9) I0703 01:44:14.164659 5754 maintenance_manager.cc:237] P 55223945fe3d4dd8ac03ba9dab346f54: Unregistered op UndoDeltaBlockGCOp(049bd2ef267f47fca634858420ac35b9) I0703 01:44:14.164757 5754 maintenance_manager.cc:237] P 55223945fe3d4dd8ac03ba9dab346f54: Unregistered op FlushMRSOp(049bd2ef267f47fca634858420ac35b9) I0703 01:44:14.164836 5754 maintenance_manager.cc:237] P 55223945fe3d4dd8ac03ba9dab346f54: Unregistered op FlushDeltaMemStoresOp(049bd2ef267f47fca634858420ac35b9) I0703 01:44:14.164921 5754 maintenance_manager.cc:237] P 55223945fe3d4dd8ac03ba9dab346f54: Unregistered op LogGCOp(049bd2ef267f47fca634858420ac35b9) I0703 01:44:14.164999 5754 raft_consensus.cc:2040] T 049bd2ef267f47fca634858420ac35b9 P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 FOLLOWER]: Raft consensus shutting down. I0703 01:44:14.165128 5754 pending_rounds.cc:63] T 049bd2ef267f47fca634858420ac35b9 P 55223945fe3d4dd8ac03ba9dab346f54: Trying to abort 1 pending transactions. I0703 01:44:14.165189 5754 pending_rounds.cc:68] T 049bd2ef267f47fca634858420ac35b9 P 55223945fe3d4dd8ac03ba9dab346f54: Aborting transaction as it isn't in flight: id { term: 1 index: 285 } timestamp: 6269368131902308352 op_type: WRITE_OP write_request { tablet_id: "049bd2ef267f47fca634858420ac35b9" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true } } row_operations { rows: "\001\007\000\351\245t\022\233r\232\020\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "8f8f0d0df143f292" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 6269368131783995392 } request_id { client_id: "956c0f4d460f4179a7ddc12c0187a34a" seq_no: 844 first_incomplete_seq_no: 844 attempt_no: 0 } I0703 01:44:14.176970 5754 raft_consensus.cc:2067] T 049bd2ef267f47fca634858420ac35b9 P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 FOLLOWER]: Raft consensus is shut down! I0703 01:44:14.180375 5754 ts_tablet_manager.cc:1369] T 049bd2ef267f47fca634858420ac35b9 P 55223945fe3d4dd8ac03ba9dab346f54: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:44:14.221844 5406 raft_consensus.cc:890] T 930bbe4ac3234ed49a2207c4902541f1 P 57bea6dc182c41a78304417a3cbfc4a7: attempting to promote NON_VOTER 213596374e4142dca6ecfc701ba1c566 to VOTER I0703 01:44:14.222184 5406 consensus_peers.cc:455] T 930bbe4ac3234ed49a2207c4902541f1 P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer 213596374e4142dca6ecfc701ba1c566 (127.5.240.196:51288): Closing peer: 213596374e4142dca6ecfc701ba1c566 I0703 01:44:14.222245 5406 consensus_peers.cc:455] T 930bbe4ac3234ed49a2207c4902541f1 P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064): Closing peer: 8c1e8943f5654f71b030ec7c0b733742 I0703 01:44:14.222506 5406 consensus_queue.cc:204] T 930bbe4ac3234ed49a2207c4902541f1 P 57bea6dc182c41a78304417a3cbfc4a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 334, Committed index: 334, Last appended: 1.334, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 335 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } } I0703 01:44:14.225839 4833 raft_consensus.cc:1081] T 930bbe4ac3234ed49a2207c4902541f1 P 213596374e4142dca6ecfc701ba1c566 [term 1 LEARNER]: Refusing update from remote peer 57bea6dc182c41a78304417a3cbfc4a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 334. Preceding OpId from leader: term: 1 index: 335. (index mismatch) I0703 01:44:14.226285 3816 raft_consensus.cc:1081] T 930bbe4ac3234ed49a2207c4902541f1 P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Refusing update from remote peer 57bea6dc182c41a78304417a3cbfc4a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 334. Preceding OpId from leader: term: 1 index: 335. (index mismatch) I0703 01:44:14.226688 5406 consensus_queue.cc:962] T 930bbe4ac3234ed49a2207c4902541f1 P 57bea6dc182c41a78304417a3cbfc4a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 335, Last known committed idx: 334, Time since last communication: 0.000s I0703 01:44:14.227807 5406 consensus_queue.cc:962] T 930bbe4ac3234ed49a2207c4902541f1 P 57bea6dc182c41a78304417a3cbfc4a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 335, Last known committed idx: 334, Time since last communication: 0.000s I0703 01:44:14.230608 5470 raft_consensus.cc:2684] T 930bbe4ac3234ed49a2207c4902541f1 P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 LEADER]: Committing config change with OpId 1.335: config changed from index 268 to 335, 213596374e4142dca6ecfc701ba1c566 (127.5.240.196) changed from NON_VOTER to VOTER. New config: { opid_index: 335 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } } } I0703 01:44:14.233814 3816 raft_consensus.cc:2684] T 930bbe4ac3234ed49a2207c4902541f1 P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Committing config change with OpId 1.335: config changed from index 268 to 335, 213596374e4142dca6ecfc701ba1c566 (127.5.240.196) changed from NON_VOTER to VOTER. New config: { opid_index: 335 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } } } I0703 01:44:14.236101 3445 catalog_manager.cc:3798] T 930bbe4ac3234ed49a2207c4902541f1 P 57bea6dc182c41a78304417a3cbfc4a7 reported cstate change: config changed from index 268 to 335, 213596374e4142dca6ecfc701ba1c566 (127.5.240.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "57bea6dc182c41a78304417a3cbfc4a7" committed_config { opid_index: 335 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } health_report { overall_health: UNKNOWN } } } I0703 01:44:14.249416 4833 raft_consensus.cc:2684] T 930bbe4ac3234ed49a2207c4902541f1 P 213596374e4142dca6ecfc701ba1c566 [term 1 FOLLOWER]: Committing config change with OpId 1.335: config changed from index 268 to 335, 213596374e4142dca6ecfc701ba1c566 (127.5.240.196) changed from NON_VOTER to VOTER. New config: { opid_index: 335 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } } peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } } } I0703 01:44:14.254163 3446 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 930bbe4ac3234ed49a2207c4902541f1 (attempt 1) I0703 01:44:14.254443 5754 ts_tablet_manager.cc:1382] T 049bd2ef267f47fca634858420ac35b9 P 55223945fe3d4dd8ac03ba9dab346f54: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.285 I0703 01:44:14.254580 5754 log.cc:986] T 049bd2ef267f47fca634858420ac35b9 P 55223945fe3d4dd8ac03ba9dab346f54: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal/wals/049bd2ef267f47fca634858420ac35b9 I0703 01:44:14.255187 3696 consensus_peers.cc:455] T 930bbe4ac3234ed49a2207c4902541f1 P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer 213596374e4142dca6ecfc701ba1c566 (127.5.240.196:51288): Closing peer: 213596374e4142dca6ecfc701ba1c566 I0703 01:44:14.255281 3696 consensus_peers.cc:455] T 930bbe4ac3234ed49a2207c4902541f1 P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064): Closing peer: 8c1e8943f5654f71b030ec7c0b733742 I0703 01:44:14.255566 3696 consensus_queue.cc:204] T 930bbe4ac3234ed49a2207c4902541f1 P 57bea6dc182c41a78304417a3cbfc4a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 335, Committed index: 335, Last appended: 1.335, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 336 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } } I0703 01:44:14.259811 4833 raft_consensus.cc:1081] T 930bbe4ac3234ed49a2207c4902541f1 P 213596374e4142dca6ecfc701ba1c566 [term 1 FOLLOWER]: Refusing update from remote peer 57bea6dc182c41a78304417a3cbfc4a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 335. Preceding OpId from leader: term: 1 index: 336. (index mismatch) I0703 01:44:14.260558 3437 catalog_manager.cc:3103] TS 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193:54919): tablet 049bd2ef267f47fca634858420ac35b9 (table rebalance_test_table_0 [id=5bc38732cb8746a7ad1c2b9fede5e7d9]) successfully deleted I0703 01:44:14.262883 5470 consensus_queue.cc:962] T 930bbe4ac3234ed49a2207c4902541f1 P 57bea6dc182c41a78304417a3cbfc4a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 336, Last known committed idx: 335, Time since last communication: 0.000s I0703 01:44:14.267053 5587 raft_consensus.cc:2684] T 930bbe4ac3234ed49a2207c4902541f1 P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 LEADER]: Committing config change with OpId 1.336: config changed from index 335 to 336, VOTER 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195) evicted. New config: { opid_index: 336 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } } } I0703 01:44:14.270299 3434 catalog_manager.cc:3352] ChangeConfig:REMOVE_PEER RPC for tablet 930bbe4ac3234ed49a2207c4902541f1 with cas_config_opid_index 335: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0703 01:44:14.271309 3446 catalog_manager.cc:3798] T 930bbe4ac3234ed49a2207c4902541f1 P 57bea6dc182c41a78304417a3cbfc4a7 reported cstate change: config changed from index 335 to 336, VOTER 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195) evicted. New cstate: current_term: 1 leader_uuid: "57bea6dc182c41a78304417a3cbfc4a7" committed_config { opid_index: 336 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } health_report { overall_health: UNKNOWN } } } I0703 01:44:14.274673 3446 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 930bbe4ac3234ed49a2207c4902541f1 on 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064) (TS 8c1e8943f5654f71b030ec7c0b733742 not found in new config with opid_index 336) I0703 01:44:14.275312 3796 tablet_service.cc:800] Processing DeleteTablet for tablet 930bbe4ac3234ed49a2207c4902541f1 with delete_type TABLET_DATA_TOMBSTONED (TS 8c1e8943f5654f71b030ec7c0b733742 not found in new config with opid_index 336) from {username='jenkins'} at 127.5.240.195:39619 I0703 01:44:14.275585 5744 tablet_replica.cc:261] T 930bbe4ac3234ed49a2207c4902541f1 P 8c1e8943f5654f71b030ec7c0b733742: stopping tablet replica I0703 01:44:14.275666 5744 maintenance_manager.cc:237] P 8c1e8943f5654f71b030ec7c0b733742: Unregistered op CompactRowSetsOp(930bbe4ac3234ed49a2207c4902541f1) I0703 01:44:14.275743 5744 maintenance_manager.cc:237] P 8c1e8943f5654f71b030ec7c0b733742: Unregistered op MinorDeltaCompactionOp(930bbe4ac3234ed49a2207c4902541f1) I0703 01:44:14.275807 5744 maintenance_manager.cc:237] P 8c1e8943f5654f71b030ec7c0b733742: Unregistered op MajorDeltaCompactionOp(930bbe4ac3234ed49a2207c4902541f1) I0703 01:44:14.275880 5744 maintenance_manager.cc:237] P 8c1e8943f5654f71b030ec7c0b733742: Unregistered op UndoDeltaBlockGCOp(930bbe4ac3234ed49a2207c4902541f1) I0703 01:44:14.275943 5744 maintenance_manager.cc:237] P 8c1e8943f5654f71b030ec7c0b733742: Unregistered op FlushMRSOp(930bbe4ac3234ed49a2207c4902541f1) I0703 01:44:14.276000 5744 maintenance_manager.cc:237] P 8c1e8943f5654f71b030ec7c0b733742: Unregistered op FlushDeltaMemStoresOp(930bbe4ac3234ed49a2207c4902541f1) I0703 01:44:14.276046 5744 maintenance_manager.cc:237] P 8c1e8943f5654f71b030ec7c0b733742: Unregistered op LogGCOp(930bbe4ac3234ed49a2207c4902541f1) I0703 01:44:14.276108 5744 raft_consensus.cc:2040] T 930bbe4ac3234ed49a2207c4902541f1 P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Raft consensus shutting down. I0703 01:44:14.276212 5744 raft_consensus.cc:2067] T 930bbe4ac3234ed49a2207c4902541f1 P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Raft consensus is shut down! I0703 01:44:14.277176 5744 ts_tablet_manager.cc:1369] T 930bbe4ac3234ed49a2207c4902541f1 P 8c1e8943f5654f71b030ec7c0b733742: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:44:14.279109 4833 raft_consensus.cc:2684] T 930bbe4ac3234ed49a2207c4902541f1 P 213596374e4142dca6ecfc701ba1c566 [term 1 FOLLOWER]: Committing config change with OpId 1.336: config changed from index 335 to 336, VOTER 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195) evicted. New config: { opid_index: 336 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } } } I0703 01:44:14.294353 5744 ts_tablet_manager.cc:1382] T 930bbe4ac3234ed49a2207c4902541f1 P 8c1e8943f5654f71b030ec7c0b733742: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.335 I0703 01:44:14.294523 5744 log.cc:986] T 930bbe4ac3234ed49a2207c4902541f1 P 8c1e8943f5654f71b030ec7c0b733742: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal/wals/930bbe4ac3234ed49a2207c4902541f1 I0703 01:44:14.295433 3437 catalog_manager.cc:3103] TS 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064): tablet 930bbe4ac3234ed49a2207c4902541f1 (table rebalance_test_table_1 [id=29732119416840ec89c7a3633f79c5fd]) successfully deleted I0703 01:44:14.349781 5234 raft_consensus.cc:1026] T 2051955b4b644a03a9adad816d350596 P 1157c91cc6f54a68a1d99017ac8444aa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.256->[1.257-1.322] Dedup: 1.268->[1.269-1.322] I0703 01:44:14.510995 5492 raft_consensus.cc:890] T 2051955b4b644a03a9adad816d350596 P 55223945fe3d4dd8ac03ba9dab346f54: attempting to promote NON_VOTER 1157c91cc6f54a68a1d99017ac8444aa to VOTER I0703 01:44:14.511391 5492 consensus_peers.cc:455] T 2051955b4b644a03a9adad816d350596 P 55223945fe3d4dd8ac03ba9dab346f54 -> Peer 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377): Closing peer: 1157c91cc6f54a68a1d99017ac8444aa I0703 01:44:14.511751 5492 consensus_peers.cc:455] T 2051955b4b644a03a9adad816d350596 P 55223945fe3d4dd8ac03ba9dab346f54 -> Peer 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746): Closing peer: 57bea6dc182c41a78304417a3cbfc4a7 I0703 01:44:14.512063 5492 consensus_queue.cc:204] T 2051955b4b644a03a9adad816d350596 P 55223945fe3d4dd8ac03ba9dab346f54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 326, Committed index: 326, Last appended: 1.326, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 327 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: false } } I0703 01:44:14.519857 5235 raft_consensus.cc:1081] T 2051955b4b644a03a9adad816d350596 P 1157c91cc6f54a68a1d99017ac8444aa [term 1 LEARNER]: Refusing update from remote peer 55223945fe3d4dd8ac03ba9dab346f54: Log matching property violated. Preceding OpId in replica: term: 1 index: 326. Preceding OpId from leader: term: 1 index: 327. (index mismatch) I0703 01:44:14.520566 5492 consensus_queue.cc:962] T 2051955b4b644a03a9adad816d350596 P 55223945fe3d4dd8ac03ba9dab346f54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 327, Last known committed idx: 326, Time since last communication: 0.000s I0703 01:44:14.521582 3696 raft_consensus.cc:1081] T 2051955b4b644a03a9adad816d350596 P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 FOLLOWER]: Refusing update from remote peer 55223945fe3d4dd8ac03ba9dab346f54: Log matching property violated. Preceding OpId in replica: term: 1 index: 326. Preceding OpId from leader: term: 1 index: 327. (index mismatch) I0703 01:44:14.527051 5492 consensus_queue.cc:962] T 2051955b4b644a03a9adad816d350596 P 55223945fe3d4dd8ac03ba9dab346f54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 327, Last known committed idx: 326, Time since last communication: 0.000s I0703 01:44:14.527798 5579 raft_consensus.cc:2684] T 2051955b4b644a03a9adad816d350596 P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 LEADER]: Committing config change with OpId 1.327: config changed from index 257 to 327, 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199) changed from NON_VOTER to VOTER. New config: { opid_index: 327 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: false } } } I0703 01:44:14.528929 5234 raft_consensus.cc:2684] T 2051955b4b644a03a9adad816d350596 P 1157c91cc6f54a68a1d99017ac8444aa [term 1 FOLLOWER]: Committing config change with OpId 1.327: config changed from index 257 to 327, 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199) changed from NON_VOTER to VOTER. New config: { opid_index: 327 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: false } } } I0703 01:44:14.532018 3445 catalog_manager.cc:3798] T 2051955b4b644a03a9adad816d350596 P 55223945fe3d4dd8ac03ba9dab346f54 reported cstate change: config changed from index 257 to 327, 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "55223945fe3d4dd8ac03ba9dab346f54" committed_config { opid_index: 327 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I0703 01:44:14.534862 3445 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 2051955b4b644a03a9adad816d350596 (attempt 1) I0703 01:44:14.536092 3570 consensus_peers.cc:455] T 2051955b4b644a03a9adad816d350596 P 55223945fe3d4dd8ac03ba9dab346f54 -> Peer 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377): Closing peer: 1157c91cc6f54a68a1d99017ac8444aa I0703 01:44:14.536166 3570 consensus_peers.cc:455] T 2051955b4b644a03a9adad816d350596 P 55223945fe3d4dd8ac03ba9dab346f54 -> Peer 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746): Closing peer: 57bea6dc182c41a78304417a3cbfc4a7 I0703 01:44:14.536321 3570 consensus_queue.cc:204] T 2051955b4b644a03a9adad816d350596 P 55223945fe3d4dd8ac03ba9dab346f54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 327, Committed index: 327, Last appended: 1.328, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 329 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: false } } I0703 01:44:14.545053 5234 raft_consensus.cc:1081] T 2051955b4b644a03a9adad816d350596 P 1157c91cc6f54a68a1d99017ac8444aa [term 1 FOLLOWER]: Refusing update from remote peer 55223945fe3d4dd8ac03ba9dab346f54: Log matching property violated. Preceding OpId in replica: term: 1 index: 328. Preceding OpId from leader: term: 1 index: 329. (index mismatch) I0703 01:44:14.547716 5492 consensus_queue.cc:962] T 2051955b4b644a03a9adad816d350596 P 55223945fe3d4dd8ac03ba9dab346f54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 329, Last known committed idx: 327, Time since last communication: 0.000s I0703 01:44:14.552227 5726 raft_consensus.cc:2684] T 2051955b4b644a03a9adad816d350596 P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 LEADER]: Committing config change with OpId 1.329: config changed from index 327 to 329, VOTER 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194) evicted. New config: { opid_index: 329 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: false } } } I0703 01:44:14.553016 5234 raft_consensus.cc:2684] T 2051955b4b644a03a9adad816d350596 P 1157c91cc6f54a68a1d99017ac8444aa [term 1 FOLLOWER]: Committing config change with OpId 1.329: config changed from index 327 to 329, VOTER 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194) evicted. New config: { opid_index: 329 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: false } } } I0703 01:44:14.557749 3446 catalog_manager.cc:3798] T 2051955b4b644a03a9adad816d350596 P 1157c91cc6f54a68a1d99017ac8444aa reported cstate change: config changed from index 327 to 329, VOTER 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194) evicted. New cstate: current_term: 1 leader_uuid: "55223945fe3d4dd8ac03ba9dab346f54" committed_config { opid_index: 329 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: false } } } I0703 01:44:14.560103 3446 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 2051955b4b644a03a9adad816d350596 on 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746) (TS 57bea6dc182c41a78304417a3cbfc4a7 not found in new config with opid_index 329) I0703 01:44:14.560822 3437 catalog_manager.cc:3352] ChangeConfig:REMOVE_PEER RPC for tablet 2051955b4b644a03a9adad816d350596 with cas_config_opid_index 327: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0703 01:44:14.561573 3676 tablet_service.cc:800] Processing DeleteTablet for tablet 2051955b4b644a03a9adad816d350596 with delete_type TABLET_DATA_TOMBSTONED (TS 57bea6dc182c41a78304417a3cbfc4a7 not found in new config with opid_index 329) from {username='jenkins'} at 127.5.240.194:39884 I0703 01:44:14.569918 5788 tablet_replica.cc:261] T 2051955b4b644a03a9adad816d350596 P 57bea6dc182c41a78304417a3cbfc4a7: stopping tablet replica I0703 01:44:14.606585 5788 maintenance_manager.cc:237] P 57bea6dc182c41a78304417a3cbfc4a7: Unregistered op CompactRowSetsOp(2051955b4b644a03a9adad816d350596) I0703 01:44:14.607064 5788 maintenance_manager.cc:237] P 57bea6dc182c41a78304417a3cbfc4a7: Unregistered op MinorDeltaCompactionOp(2051955b4b644a03a9adad816d350596) I0703 01:44:14.607233 5788 maintenance_manager.cc:237] P 57bea6dc182c41a78304417a3cbfc4a7: Unregistered op MajorDeltaCompactionOp(2051955b4b644a03a9adad816d350596) I0703 01:44:14.607391 5788 maintenance_manager.cc:237] P 57bea6dc182c41a78304417a3cbfc4a7: Unregistered op UndoDeltaBlockGCOp(2051955b4b644a03a9adad816d350596) I0703 01:44:14.607537 5788 maintenance_manager.cc:237] P 57bea6dc182c41a78304417a3cbfc4a7: Unregistered op FlushMRSOp(2051955b4b644a03a9adad816d350596) I0703 01:44:14.607661 5788 maintenance_manager.cc:237] P 57bea6dc182c41a78304417a3cbfc4a7: Unregistered op FlushDeltaMemStoresOp(2051955b4b644a03a9adad816d350596) I0703 01:44:14.611171 5788 maintenance_manager.cc:237] P 57bea6dc182c41a78304417a3cbfc4a7: Unregistered op LogGCOp(2051955b4b644a03a9adad816d350596) I0703 01:44:14.611318 5788 raft_consensus.cc:2040] T 2051955b4b644a03a9adad816d350596 P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 FOLLOWER]: Raft consensus shutting down. I0703 01:44:14.611469 5788 pending_rounds.cc:63] T 2051955b4b644a03a9adad816d350596 P 57bea6dc182c41a78304417a3cbfc4a7: Trying to abort 2 pending transactions. I0703 01:44:14.636309 5788 pending_rounds.cc:68] T 2051955b4b644a03a9adad816d350596 P 57bea6dc182c41a78304417a3cbfc4a7: Aborting transaction as it isn't in flight: id { term: 1 index: 327 } timestamp: 6269368133677948928 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "2051955b4b644a03a9adad816d350596" old_config { opid_index: 257 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: true } } } new_config { opid_index: 327 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } } peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: false } } } } I0703 01:44:14.640084 5788 raft_consensus.cc:2646] T 2051955b4b644a03a9adad816d350596 P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 FOLLOWER]: Aborting config change with OpId 1.327: Aborted: Transaction aborted I0703 01:44:14.640210 5788 raft_consensus.cc:2620] T 2051955b4b644a03a9adad816d350596 P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Transaction aborted I0703 01:44:14.640636 5788 pending_rounds.cc:68] T 2051955b4b644a03a9adad816d350596 P 57bea6dc182c41a78304417a3cbfc4a7: Aborting transaction as it isn't in flight: id { term: 1 index: 328 } timestamp: 6269368133726072832 op_type: WRITE_OP write_request { tablet_id: "2051955b4b644a03a9adad816d350596" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true } } row_operations { rows: "\001\007\000|\305-\033u7\020N\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "2ff3b29150a0cc5e" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 6269368133474959360 } request_id { client_id: "327e9f6d8c49476cbcbaddf9f0c9908e" seq_no: 884 first_incomplete_seq_no: 884 attempt_no: 0 } I0703 01:44:14.641763 5788 raft_consensus.cc:2067] T 2051955b4b644a03a9adad816d350596 P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 FOLLOWER]: Raft consensus is shut down! I0703 01:44:14.649610 5788 ts_tablet_manager.cc:1369] T 2051955b4b644a03a9adad816d350596 P 57bea6dc182c41a78304417a3cbfc4a7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:44:14.686578 5788 ts_tablet_manager.cc:1382] T 2051955b4b644a03a9adad816d350596 P 57bea6dc182c41a78304417a3cbfc4a7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.328 I0703 01:44:14.687110 5788 log.cc:986] T 2051955b4b644a03a9adad816d350596 P 57bea6dc182c41a78304417a3cbfc4a7: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal/wals/2051955b4b644a03a9adad816d350596 I0703 01:44:14.688760 3434 catalog_manager.cc:3103] TS 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746): tablet 2051955b4b644a03a9adad816d350596 (table rebalance_test_table_2 [id=24e8eff3fcb34269ba893ac2456e5712]) successfully deleted I0703 01:44:15.415745 3692 consensus_peers.cc:455] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197:51329): Closing peer: aa8b9c58b466436faa5086e85b5224a7 I0703 01:44:15.416036 3692 consensus_queue.cc:204] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 323, Committed index: 323, Last appended: 1.323, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 324 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: true } } I0703 01:44:15.422899 4966 raft_consensus.cc:1081] T 049bd2ef267f47fca634858420ac35b9 P aa8b9c58b466436faa5086e85b5224a7 [term 1 FOLLOWER]: Refusing update from remote peer 57bea6dc182c41a78304417a3cbfc4a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 323. Preceding OpId from leader: term: 1 index: 324. (index mismatch) I0703 01:44:15.426194 5808 consensus_queue.cc:962] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 324, Last known committed idx: 323, Time since last communication: 0.000s W0703 01:44:15.432989 3627 consensus_peers.cc:433] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377): Couldn't send request to peer 1157c91cc6f54a68a1d99017ac8444aa for tablet 049bd2ef267f47fca634858420ac35b9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 049bd2ef267f47fca634858420ac35b9. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:44:15.438423 5808 raft_consensus.cc:2684] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 LEADER]: Committing config change with OpId 1.324: config changed from index 288 to 324, NON_VOTER 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199) added. New config: { opid_index: 324 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: true } } } I0703 01:44:15.442886 3445 catalog_manager.cc:3798] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 reported cstate change: config changed from index 288 to 324, NON_VOTER 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199) added. New cstate: current_term: 1 leader_uuid: "57bea6dc182c41a78304417a3cbfc4a7" committed_config { opid_index: 324 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I0703 01:44:15.443953 4966 raft_consensus.cc:2684] T 049bd2ef267f47fca634858420ac35b9 P aa8b9c58b466436faa5086e85b5224a7 [term 1 FOLLOWER]: Committing config change with OpId 1.324: config changed from index 288 to 324, NON_VOTER 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199) added. New config: { opid_index: 324 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: true } } } I0703 01:44:15.486734 3816 consensus_peers.cc:455] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 -> Peer aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197:51329): Closing peer: aa8b9c58b466436faa5086e85b5224a7 I0703 01:44:15.486976 3816 consensus_queue.cc:204] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 322, Committed index: 322, Last appended: 1.322, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 323 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: NON_VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: true } } I0703 01:44:15.490386 4966 raft_consensus.cc:1081] T c352f70a51154a85a5a5b0f3d14d405f P aa8b9c58b466436faa5086e85b5224a7 [term 1 FOLLOWER]: Refusing update from remote peer 8c1e8943f5654f71b030ec7c0b733742: Log matching property violated. Preceding OpId in replica: term: 1 index: 322. Preceding OpId from leader: term: 1 index: 323. (index mismatch) I0703 01:44:15.493532 5025 consensus_queue.cc:962] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 323, Last known committed idx: 322, Time since last communication: 0.000s I0703 01:44:15.496835 5154 raft_consensus.cc:2684] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 [term 1 LEADER]: Committing config change with OpId 1.323: config changed from index 291 to 323, NON_VOTER 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198) added. New config: { opid_index: 323 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: NON_VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: true } } } I0703 01:44:15.497752 4966 raft_consensus.cc:2684] T c352f70a51154a85a5a5b0f3d14d405f P aa8b9c58b466436faa5086e85b5224a7 [term 1 FOLLOWER]: Committing config change with OpId 1.323: config changed from index 291 to 323, NON_VOTER 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198) added. New config: { opid_index: 323 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: NON_VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: true } } } I0703 01:44:15.501220 3446 catalog_manager.cc:3798] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 reported cstate change: config changed from index 291 to 323, NON_VOTER 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198) added. New cstate: current_term: 1 leader_uuid: "8c1e8943f5654f71b030ec7c0b733742" committed_config { opid_index: 323 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: NON_VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } W0703 01:44:15.527899 3752 consensus_peers.cc:433] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 -> Peer 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198:51033): Couldn't send request to peer 6932f1cd716a48fe8d6c5573acfc48bf for tablet c352f70a51154a85a5a5b0f3d14d405f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c352f70a51154a85a5a5b0f3d14d405f. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:44:15.561408 3570 consensus_peers.cc:455] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 -> Peer 213596374e4142dca6ecfc701ba1c566 (127.5.240.196:51288): Closing peer: 213596374e4142dca6ecfc701ba1c566 I0703 01:44:15.561632 3570 consensus_queue.cc:204] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 404, Committed index: 404, Last appended: 1.405, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 406 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: NON_VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: true } } W0703 01:44:15.564131 3507 consensus_peers.cc:433] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 -> Peer 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198:51033): Couldn't send request to peer 6932f1cd716a48fe8d6c5573acfc48bf for tablet 60d3228294e945c5b343ae62e76dc2f0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 60d3228294e945c5b343ae62e76dc2f0. Retrying in the next heartbeat period. Already tried 1 times. I0703 01:44:15.566691 4833 raft_consensus.cc:1081] T 60d3228294e945c5b343ae62e76dc2f0 P 213596374e4142dca6ecfc701ba1c566 [term 1 FOLLOWER]: Refusing update from remote peer 55223945fe3d4dd8ac03ba9dab346f54: Log matching property violated. Preceding OpId in replica: term: 1 index: 405. Preceding OpId from leader: term: 1 index: 406. (index mismatch) I0703 01:44:15.567617 5579 consensus_queue.cc:962] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 406, Last known committed idx: 404, Time since last communication: 0.000s I0703 01:44:15.570916 5492 raft_consensus.cc:2684] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 LEADER]: Committing config change with OpId 1.406: config changed from index 364 to 406, NON_VOTER 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198) added. New config: { opid_index: 406 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: NON_VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: true } } } I0703 01:44:15.571419 4833 raft_consensus.cc:2684] T 60d3228294e945c5b343ae62e76dc2f0 P 213596374e4142dca6ecfc701ba1c566 [term 1 FOLLOWER]: Committing config change with OpId 1.406: config changed from index 364 to 406, NON_VOTER 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198) added. New config: { opid_index: 406 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: NON_VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: true } } } I0703 01:44:15.581728 3445 catalog_manager.cc:3798] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 reported cstate change: config changed from index 364 to 406, NON_VOTER 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198) added. New cstate: current_term: 1 leader_uuid: "55223945fe3d4dd8ac03ba9dab346f54" committed_config { opid_index: 406 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: NON_VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } I0703 01:44:15.991989 5889 ts_tablet_manager.cc:669] T 049bd2ef267f47fca634858420ac35b9 P 1157c91cc6f54a68a1d99017ac8444aa: Initiating tablet copy from peer 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746) I0703 01:44:15.992149 5889 tablet_copy_client.cc:239] T 049bd2ef267f47fca634858420ac35b9 P 1157c91cc6f54a68a1d99017ac8444aa: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.194:59746 I0703 01:44:16.012835 3716 tablet_copy_service.cc:135] P 57bea6dc182c41a78304417a3cbfc4a7: Received BeginTabletCopySession request for tablet 049bd2ef267f47fca634858420ac35b9 from peer 1157c91cc6f54a68a1d99017ac8444aa ({username='jenkins'} at 127.5.240.199:49896) I0703 01:44:16.012995 3716 tablet_copy_service.cc:156] P 57bea6dc182c41a78304417a3cbfc4a7: Beginning new tablet copy session on tablet 049bd2ef267f47fca634858420ac35b9 from peer 1157c91cc6f54a68a1d99017ac8444aa at {username='jenkins'} at 127.5.240.199:49896: session id = 1157c91cc6f54a68a1d99017ac8444aa-049bd2ef267f47fca634858420ac35b9 I0703 01:44:16.015878 3716 tablet_copy_source_session.cc:202] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:44:16.017418 5889 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 049bd2ef267f47fca634858420ac35b9. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:44:16.045562 5889 ts_tablet_manager.cc:1173] T 049bd2ef267f47fca634858420ac35b9 P 1157c91cc6f54a68a1d99017ac8444aa: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:44:16.049595 5889 tablet_copy_client.cc:566] T 049bd2ef267f47fca634858420ac35b9 P 1157c91cc6f54a68a1d99017ac8444aa: tablet copy: Starting download of 0 data blocks... I0703 01:44:16.050232 5895 ts_tablet_manager.cc:669] T c352f70a51154a85a5a5b0f3d14d405f P 6932f1cd716a48fe8d6c5573acfc48bf: Initiating tablet copy from peer 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064) I0703 01:44:16.050712 5895 tablet_copy_client.cc:239] T c352f70a51154a85a5a5b0f3d14d405f P 6932f1cd716a48fe8d6c5573acfc48bf: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.195:51064 I0703 01:44:16.051429 5889 tablet_copy_client.cc:529] T 049bd2ef267f47fca634858420ac35b9 P 1157c91cc6f54a68a1d99017ac8444aa: tablet copy: Starting download of 1 WAL segments... I0703 01:44:16.060755 5889 tablet_copy_client.cc:405] T 049bd2ef267f47fca634858420ac35b9 P 1157c91cc6f54a68a1d99017ac8444aa: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:44:16.067768 5889 ts_tablet_manager.cc:1028] T 049bd2ef267f47fca634858420ac35b9 P 1157c91cc6f54a68a1d99017ac8444aa: Bootstrapping tablet I0703 01:44:16.067931 5889 tablet_bootstrap.cc:427] T 049bd2ef267f47fca634858420ac35b9 P 1157c91cc6f54a68a1d99017ac8444aa: Bootstrap starting. I0703 01:44:16.091382 3838 tablet_copy_service.cc:135] P 8c1e8943f5654f71b030ec7c0b733742: Received BeginTabletCopySession request for tablet c352f70a51154a85a5a5b0f3d14d405f from peer 6932f1cd716a48fe8d6c5573acfc48bf ({username='jenkins'} at 127.5.240.198:54194) I0703 01:44:16.091482 3838 tablet_copy_service.cc:156] P 8c1e8943f5654f71b030ec7c0b733742: Beginning new tablet copy session on tablet c352f70a51154a85a5a5b0f3d14d405f from peer 6932f1cd716a48fe8d6c5573acfc48bf at {username='jenkins'} at 127.5.240.198:54194: session id = 6932f1cd716a48fe8d6c5573acfc48bf-c352f70a51154a85a5a5b0f3d14d405f I0703 01:44:16.093921 3838 tablet_copy_source_session.cc:202] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:44:16.096444 5895 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet c352f70a51154a85a5a5b0f3d14d405f. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:44:16.101490 5895 ts_tablet_manager.cc:1173] T c352f70a51154a85a5a5b0f3d14d405f P 6932f1cd716a48fe8d6c5573acfc48bf: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:44:16.101924 5895 tablet_copy_client.cc:566] T c352f70a51154a85a5a5b0f3d14d405f P 6932f1cd716a48fe8d6c5573acfc48bf: tablet copy: Starting download of 0 data blocks... I0703 01:44:16.102149 5895 tablet_copy_client.cc:529] T c352f70a51154a85a5a5b0f3d14d405f P 6932f1cd716a48fe8d6c5573acfc48bf: tablet copy: Starting download of 1 WAL segments... I0703 01:44:16.109544 5895 tablet_copy_client.cc:405] T c352f70a51154a85a5a5b0f3d14d405f P 6932f1cd716a48fe8d6c5573acfc48bf: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:44:16.111810 5895 ts_tablet_manager.cc:1028] T c352f70a51154a85a5a5b0f3d14d405f P 6932f1cd716a48fe8d6c5573acfc48bf: Bootstrapping tablet I0703 01:44:16.111974 5895 tablet_bootstrap.cc:427] T c352f70a51154a85a5a5b0f3d14d405f P 6932f1cd716a48fe8d6c5573acfc48bf: Bootstrap starting. I0703 01:44:16.206135 5904 ts_tablet_manager.cc:669] T 60d3228294e945c5b343ae62e76dc2f0 P 6932f1cd716a48fe8d6c5573acfc48bf: Initiating tablet copy from peer 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193:54919) I0703 01:44:16.206346 5904 tablet_copy_client.cc:239] T 60d3228294e945c5b343ae62e76dc2f0 P 6932f1cd716a48fe8d6c5573acfc48bf: tablet copy: Beginning tablet copy session from remote peer at address 127.5.240.193:54919 I0703 01:44:16.207150 3593 tablet_copy_service.cc:135] P 55223945fe3d4dd8ac03ba9dab346f54: Received BeginTabletCopySession request for tablet 60d3228294e945c5b343ae62e76dc2f0 from peer 6932f1cd716a48fe8d6c5573acfc48bf ({username='jenkins'} at 127.5.240.198:54009) I0703 01:44:16.207258 3593 tablet_copy_service.cc:156] P 55223945fe3d4dd8ac03ba9dab346f54: Beginning new tablet copy session on tablet 60d3228294e945c5b343ae62e76dc2f0 from peer 6932f1cd716a48fe8d6c5573acfc48bf at {username='jenkins'} at 127.5.240.198:54009: session id = 6932f1cd716a48fe8d6c5573acfc48bf-60d3228294e945c5b343ae62e76dc2f0 I0703 01:44:16.209841 3593 tablet_copy_source_session.cc:202] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54: Tablet Copy: opened 0 blocks and 1 log segments I0703 01:44:16.225037 5904 data_dirs.cc:934] Could only allocate 1 dirs of requested 3 for tablet 60d3228294e945c5b343ae62e76dc2f0. 1 dirs total, 0 dirs full, 0 dirs failed I0703 01:44:16.243638 5904 ts_tablet_manager.cc:1173] T 60d3228294e945c5b343ae62e76dc2f0 P 6932f1cd716a48fe8d6c5573acfc48bf: Registered tablet (data state: TABLET_DATA_COPYING) I0703 01:44:16.245097 5904 tablet_copy_client.cc:566] T 60d3228294e945c5b343ae62e76dc2f0 P 6932f1cd716a48fe8d6c5573acfc48bf: tablet copy: Starting download of 0 data blocks... I0703 01:44:16.245281 5904 tablet_copy_client.cc:529] T 60d3228294e945c5b343ae62e76dc2f0 P 6932f1cd716a48fe8d6c5573acfc48bf: tablet copy: Starting download of 1 WAL segments... I0703 01:44:16.260095 5904 tablet_copy_client.cc:405] T 60d3228294e945c5b343ae62e76dc2f0 P 6932f1cd716a48fe8d6c5573acfc48bf: tablet copy: Tablet Copy complete. Replacing tablet superblock. I0703 01:44:16.265496 5904 ts_tablet_manager.cc:1028] T 60d3228294e945c5b343ae62e76dc2f0 P 6932f1cd716a48fe8d6c5573acfc48bf: Bootstrapping tablet I0703 01:44:16.271936 5904 tablet_bootstrap.cc:427] T 60d3228294e945c5b343ae62e76dc2f0 P 6932f1cd716a48fe8d6c5573acfc48bf: Bootstrap starting. W0703 01:44:16.471336 3627 consensus_peers.cc:433] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377): Couldn't send request to peer 1157c91cc6f54a68a1d99017ac8444aa for tablet 049bd2ef267f47fca634858420ac35b9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 2 times. W0703 01:44:16.613148 3752 consensus_peers.cc:433] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 -> Peer 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198:51033): Couldn't send request to peer 6932f1cd716a48fe8d6c5573acfc48bf for tablet c352f70a51154a85a5a5b0f3d14d405f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 2 times. W0703 01:44:16.654695 3507 consensus_peers.cc:433] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 -> Peer 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198:51033): Couldn't send request to peer 6932f1cd716a48fe8d6c5573acfc48bf for tablet 60d3228294e945c5b343ae62e76dc2f0. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 2 times. W0703 01:44:17.029361 3627 consensus_peers.cc:433] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377): Couldn't send request to peer 1157c91cc6f54a68a1d99017ac8444aa for tablet 049bd2ef267f47fca634858420ac35b9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 3 times. W0703 01:44:17.082715 3752 consensus_peers.cc:433] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 -> Peer 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198:51033): Couldn't send request to peer 6932f1cd716a48fe8d6c5573acfc48bf for tablet c352f70a51154a85a5a5b0f3d14d405f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 3 times. W0703 01:44:17.172303 3507 consensus_peers.cc:433] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 -> Peer 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198:51033): Couldn't send request to peer 6932f1cd716a48fe8d6c5573acfc48bf for tablet 60d3228294e945c5b343ae62e76dc2f0. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 3 times. W0703 01:44:17.407630 3627 consensus_peers.cc:433] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377): Couldn't send request to peer 1157c91cc6f54a68a1d99017ac8444aa for tablet 049bd2ef267f47fca634858420ac35b9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 4 times. I0703 01:44:17.469601 5895 tablet_bootstrap.cc:427] T c352f70a51154a85a5a5b0f3d14d405f P 6932f1cd716a48fe8d6c5573acfc48bf: Bootstrap replayed 1/1 log segments. Stats: ops{read=341 overwritten=0 applied=340 ignored=0} inserts{seen=986 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates I0703 01:44:17.470089 5895 tablet_bootstrap.cc:427] T c352f70a51154a85a5a5b0f3d14d405f P 6932f1cd716a48fe8d6c5573acfc48bf: Bootstrap complete. I0703 01:44:17.470358 5895 ts_tablet_manager.cc:1045] T c352f70a51154a85a5a5b0f3d14d405f P 6932f1cd716a48fe8d6c5573acfc48bf: Time spent bootstrapping tablet: real 1.358s user 0.406s sys 0.041s I0703 01:44:17.472820 5895 raft_consensus.cc:325] T c352f70a51154a85a5a5b0f3d14d405f P 6932f1cd716a48fe8d6c5573acfc48bf [term 1 LEARNER]: Replica starting. Triggering 1 pending transactions. Active config: opid_index: 323 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: NON_VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: true } } I0703 01:44:17.474432 5895 raft_consensus.cc:628] T c352f70a51154a85a5a5b0f3d14d405f P 6932f1cd716a48fe8d6c5573acfc48bf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6932f1cd716a48fe8d6c5573acfc48bf, State: Initialized, Role: LEARNER I0703 01:44:17.474684 5895 consensus_queue.cc:227] T c352f70a51154a85a5a5b0f3d14d405f P 6932f1cd716a48fe8d6c5573acfc48bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 340, Last appended: 1.341, Last appended by leader: 341, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 323 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: NON_VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: true } } I0703 01:44:17.484349 5895 ts_tablet_manager.cc:1073] T c352f70a51154a85a5a5b0f3d14d405f P 6932f1cd716a48fe8d6c5573acfc48bf: Time spent starting tablet: real 0.014s user 0.001s sys 0.002s I0703 01:44:17.486075 3838 tablet_copy_service.cc:325] P 8c1e8943f5654f71b030ec7c0b733742: Request end of tablet copy session 6932f1cd716a48fe8d6c5573acfc48bf-c352f70a51154a85a5a5b0f3d14d405f received from {username='jenkins'} at 127.5.240.198:54194 I0703 01:44:17.486196 3838 tablet_copy_service.cc:413] P 8c1e8943f5654f71b030ec7c0b733742: ending tablet copy session 6932f1cd716a48fe8d6c5573acfc48bf-c352f70a51154a85a5a5b0f3d14d405f on tablet c352f70a51154a85a5a5b0f3d14d405f with peer 6932f1cd716a48fe8d6c5573acfc48bf W0703 01:44:17.583808 3507 consensus_peers.cc:433] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 -> Peer 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198:51033): Couldn't send request to peer 6932f1cd716a48fe8d6c5573acfc48bf for tablet 60d3228294e945c5b343ae62e76dc2f0. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 4 times. I0703 01:44:17.621213 5889 tablet_bootstrap.cc:427] T 049bd2ef267f47fca634858420ac35b9 P 1157c91cc6f54a68a1d99017ac8444aa: Bootstrap replayed 1/1 log segments. Stats: ops{read=337 overwritten=0 applied=337 ignored=0} inserts{seen=983 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates I0703 01:44:17.622154 5889 tablet_bootstrap.cc:427] T 049bd2ef267f47fca634858420ac35b9 P 1157c91cc6f54a68a1d99017ac8444aa: Bootstrap complete. I0703 01:44:17.622404 5889 ts_tablet_manager.cc:1045] T 049bd2ef267f47fca634858420ac35b9 P 1157c91cc6f54a68a1d99017ac8444aa: Time spent bootstrapping tablet: real 1.555s user 0.402s sys 0.032s I0703 01:44:17.623395 5889 raft_consensus.cc:325] T 049bd2ef267f47fca634858420ac35b9 P 1157c91cc6f54a68a1d99017ac8444aa [term 1 LEARNER]: Replica starting. Triggering 0 pending transactions. Active config: opid_index: 324 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: true } } I0703 01:44:17.625711 5889 raft_consensus.cc:628] T 049bd2ef267f47fca634858420ac35b9 P 1157c91cc6f54a68a1d99017ac8444aa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1157c91cc6f54a68a1d99017ac8444aa, State: Initialized, Role: LEARNER I0703 01:44:17.625875 5889 consensus_queue.cc:227] T 049bd2ef267f47fca634858420ac35b9 P 1157c91cc6f54a68a1d99017ac8444aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 337, Last appended: 1.337, Last appended by leader: 337, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 324 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: NON_VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: true } } I0703 01:44:17.635398 5889 ts_tablet_manager.cc:1073] T 049bd2ef267f47fca634858420ac35b9 P 1157c91cc6f54a68a1d99017ac8444aa: Time spent starting tablet: real 0.013s user 0.001s sys 0.002s I0703 01:44:17.636564 3716 tablet_copy_service.cc:325] P 57bea6dc182c41a78304417a3cbfc4a7: Request end of tablet copy session 1157c91cc6f54a68a1d99017ac8444aa-049bd2ef267f47fca634858420ac35b9 received from {username='jenkins'} at 127.5.240.199:49896 I0703 01:44:17.636696 3716 tablet_copy_service.cc:413] P 57bea6dc182c41a78304417a3cbfc4a7: ending tablet copy session 1157c91cc6f54a68a1d99017ac8444aa-049bd2ef267f47fca634858420ac35b9 on tablet 049bd2ef267f47fca634858420ac35b9 with peer 1157c91cc6f54a68a1d99017ac8444aa I0703 01:44:17.642241 5097 raft_consensus.cc:1026] T c352f70a51154a85a5a5b0f3d14d405f P 6932f1cd716a48fe8d6c5573acfc48bf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.322->[1.323-1.375] Dedup: 1.341->[1.342-1.375] I0703 01:44:17.779495 5722 raft_consensus.cc:890] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742: attempting to promote NON_VOTER 6932f1cd716a48fe8d6c5573acfc48bf to VOTER I0703 01:44:17.779835 5722 consensus_peers.cc:455] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 -> Peer 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198:51033): Closing peer: 6932f1cd716a48fe8d6c5573acfc48bf I0703 01:44:17.779914 5722 consensus_peers.cc:455] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 -> Peer aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197:51329): Closing peer: aa8b9c58b466436faa5086e85b5224a7 I0703 01:44:17.780277 5722 consensus_queue.cc:204] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 375, Committed index: 375, Last appended: 1.376, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 377 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false } } I0703 01:44:17.782505 5097 raft_consensus.cc:1081] T c352f70a51154a85a5a5b0f3d14d405f P 6932f1cd716a48fe8d6c5573acfc48bf [term 1 LEARNER]: Refusing update from remote peer 8c1e8943f5654f71b030ec7c0b733742: Log matching property violated. Preceding OpId in replica: term: 1 index: 376. Preceding OpId from leader: term: 1 index: 377. (index mismatch) I0703 01:44:17.783186 5949 consensus_queue.cc:962] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 377, Last known committed idx: 375, Time since last communication: 0.000s I0703 01:44:17.784363 4961 raft_consensus.cc:1081] T c352f70a51154a85a5a5b0f3d14d405f P aa8b9c58b466436faa5086e85b5224a7 [term 1 FOLLOWER]: Refusing update from remote peer 8c1e8943f5654f71b030ec7c0b733742: Log matching property violated. Preceding OpId in replica: term: 1 index: 376. Preceding OpId from leader: term: 1 index: 377. (index mismatch) I0703 01:44:17.785089 5949 consensus_queue.cc:962] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 377, Last known committed idx: 375, Time since last communication: 0.000s I0703 01:44:17.785892 5870 raft_consensus.cc:2684] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 [term 1 LEADER]: Committing config change with OpId 1.377: config changed from index 323 to 377, 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198) changed from NON_VOTER to VOTER. New config: { opid_index: 377 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false } } } I0703 01:44:17.790194 5097 raft_consensus.cc:2684] T c352f70a51154a85a5a5b0f3d14d405f P 6932f1cd716a48fe8d6c5573acfc48bf [term 1 FOLLOWER]: Committing config change with OpId 1.377: config changed from index 323 to 377, 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198) changed from NON_VOTER to VOTER. New config: { opid_index: 377 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false } } } I0703 01:44:17.790997 3446 catalog_manager.cc:3798] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 reported cstate change: config changed from index 323 to 377, 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8c1e8943f5654f71b030ec7c0b733742" committed_config { opid_index: 377 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I0703 01:44:17.791308 4961 raft_consensus.cc:2684] T c352f70a51154a85a5a5b0f3d14d405f P aa8b9c58b466436faa5086e85b5224a7 [term 1 FOLLOWER]: Committing config change with OpId 1.377: config changed from index 323 to 377, 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198) changed from NON_VOTER to VOTER. New config: { opid_index: 377 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false } } } I0703 01:44:18.019991 5235 raft_consensus.cc:1026] T 049bd2ef267f47fca634858420ac35b9 P 1157c91cc6f54a68a1d99017ac8444aa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.323->[1.324-1.379] Dedup: 1.337->[1.338-1.379] W0703 01:44:18.049722 3507 consensus_peers.cc:433] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 -> Peer 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198:51033): Couldn't send request to peer 6932f1cd716a48fe8d6c5573acfc48bf for tablet 60d3228294e945c5b343ae62e76dc2f0. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. Retrying in the next heartbeat period. Already tried 5 times. I0703 01:44:18.160305 5904 tablet_bootstrap.cc:427] T 60d3228294e945c5b343ae62e76dc2f0 P 6932f1cd716a48fe8d6c5573acfc48bf: Bootstrap replayed 1/1 log segments. Stats: ops{read=428 overwritten=0 applied=427 ignored=0} inserts{seen=1076 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates I0703 01:44:18.161106 5904 tablet_bootstrap.cc:427] T 60d3228294e945c5b343ae62e76dc2f0 P 6932f1cd716a48fe8d6c5573acfc48bf: Bootstrap complete. I0703 01:44:18.161360 5904 ts_tablet_manager.cc:1045] T 60d3228294e945c5b343ae62e76dc2f0 P 6932f1cd716a48fe8d6c5573acfc48bf: Time spent bootstrapping tablet: real 1.889s user 0.488s sys 0.053s I0703 01:44:18.162856 5809 raft_consensus.cc:890] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7: attempting to promote NON_VOTER 1157c91cc6f54a68a1d99017ac8444aa to VOTER I0703 01:44:18.163203 5809 consensus_peers.cc:455] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377): Closing peer: 1157c91cc6f54a68a1d99017ac8444aa I0703 01:44:18.163283 5809 consensus_peers.cc:455] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197:51329): Closing peer: aa8b9c58b466436faa5086e85b5224a7 I0703 01:44:18.163561 5809 consensus_queue.cc:204] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 380, Committed index: 380, Last appended: 1.380, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 381 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: false } } I0703 01:44:18.163918 5904 raft_consensus.cc:325] T 60d3228294e945c5b343ae62e76dc2f0 P 6932f1cd716a48fe8d6c5573acfc48bf [term 1 LEARNER]: Replica starting. Triggering 1 pending transactions. Active config: opid_index: 406 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: NON_VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: true } } I0703 01:44:18.166033 4965 raft_consensus.cc:1081] T 049bd2ef267f47fca634858420ac35b9 P aa8b9c58b466436faa5086e85b5224a7 [term 1 FOLLOWER]: Refusing update from remote peer 57bea6dc182c41a78304417a3cbfc4a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 380. Preceding OpId from leader: term: 1 index: 381. (index mismatch) I0703 01:44:18.166098 5904 raft_consensus.cc:628] T 60d3228294e945c5b343ae62e76dc2f0 P 6932f1cd716a48fe8d6c5573acfc48bf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6932f1cd716a48fe8d6c5573acfc48bf, State: Initialized, Role: LEARNER I0703 01:44:18.166223 5904 consensus_queue.cc:227] T 60d3228294e945c5b343ae62e76dc2f0 P 6932f1cd716a48fe8d6c5573acfc48bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 427, Last appended: 1.428, Last appended by leader: 428, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 406 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: NON_VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: true } } I0703 01:44:18.167310 5233 raft_consensus.cc:1081] T 049bd2ef267f47fca634858420ac35b9 P 1157c91cc6f54a68a1d99017ac8444aa [term 1 LEARNER]: Refusing update from remote peer 57bea6dc182c41a78304417a3cbfc4a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 380. Preceding OpId from leader: term: 1 index: 381. (index mismatch) I0703 01:44:18.167477 5808 consensus_queue.cc:962] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 381, Last known committed idx: 380, Time since last communication: 0.000s I0703 01:44:18.169559 5809 consensus_queue.cc:962] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 381, Last known committed idx: 380, Time since last communication: 0.000s I0703 01:44:18.171942 5808 raft_consensus.cc:2684] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 LEADER]: Committing config change with OpId 1.381: config changed from index 324 to 381, 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199) changed from NON_VOTER to VOTER. New config: { opid_index: 381 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: false } } } I0703 01:44:18.174250 5904 ts_tablet_manager.cc:1073] T 60d3228294e945c5b343ae62e76dc2f0 P 6932f1cd716a48fe8d6c5573acfc48bf: Time spent starting tablet: real 0.012s user 0.000s sys 0.001s I0703 01:44:18.175132 3593 tablet_copy_service.cc:325] P 55223945fe3d4dd8ac03ba9dab346f54: Request end of tablet copy session 6932f1cd716a48fe8d6c5573acfc48bf-60d3228294e945c5b343ae62e76dc2f0 received from {username='jenkins'} at 127.5.240.198:54009 I0703 01:44:18.175216 3593 tablet_copy_service.cc:413] P 55223945fe3d4dd8ac03ba9dab346f54: ending tablet copy session 6932f1cd716a48fe8d6c5573acfc48bf-60d3228294e945c5b343ae62e76dc2f0 on tablet 60d3228294e945c5b343ae62e76dc2f0 with peer 6932f1cd716a48fe8d6c5573acfc48bf I0703 01:44:18.175915 3448 catalog_manager.cc:3798] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 reported cstate change: config changed from index 324 to 381, 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "57bea6dc182c41a78304417a3cbfc4a7" committed_config { opid_index: 381 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: false } health_report { overall_health: UNKNOWN } } } I0703 01:44:18.176195 4965 raft_consensus.cc:2684] T 049bd2ef267f47fca634858420ac35b9 P aa8b9c58b466436faa5086e85b5224a7 [term 1 FOLLOWER]: Committing config change with OpId 1.381: config changed from index 324 to 381, 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199) changed from NON_VOTER to VOTER. New config: { opid_index: 381 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: false } } } I0703 01:44:18.179155 5234 raft_consensus.cc:2684] T 049bd2ef267f47fca634858420ac35b9 P 1157c91cc6f54a68a1d99017ac8444aa [term 1 FOLLOWER]: Committing config change with OpId 1.381: config changed from index 324 to 381, 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199) changed from NON_VOTER to VOTER. New config: { opid_index: 381 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: false } } } I0703 01:44:18.187360 3816 tablet_service.cc:1140] Received LeaderStepDown RPC: tablet_id: "c352f70a51154a85a5a5b0f3d14d405f" dest_uuid: "8c1e8943f5654f71b030ec7c0b733742" from {username='jenkins'} at 127.5.240.195:39766 I0703 01:44:18.187578 3816 raft_consensus.cc:552] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 [term 1 LEADER]: Received request to step down I0703 01:44:18.187657 3816 raft_consensus.cc:628] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8c1e8943f5654f71b030ec7c0b733742, State: Running, Role: LEADER I0703 01:44:18.187927 3816 consensus_queue.cc:227] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 385, Committed index: 385, Last appended: 1.385, Last appended by leader: 385, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 377 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false } } I0703 01:44:18.188290 3816 consensus_peers.cc:455] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 -> Peer 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198:51033): Closing peer: 6932f1cd716a48fe8d6c5573acfc48bf I0703 01:44:18.190351 3816 consensus_peers.cc:455] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 -> Peer aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197:51329): Closing peer: aa8b9c58b466436faa5086e85b5224a7 I0703 01:44:18.190517 3816 raft_consensus.cc:2730] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742: Snoozing failure detection for 3.000s (explicit stepdown request) I0703 01:44:18.452123 3692 tablet_service.cc:1140] Received LeaderStepDown RPC: tablet_id: "049bd2ef267f47fca634858420ac35b9" dest_uuid: "57bea6dc182c41a78304417a3cbfc4a7" from {username='jenkins'} at 127.5.240.194:40028 I0703 01:44:18.452311 3692 raft_consensus.cc:552] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 LEADER]: Received request to step down I0703 01:44:18.452383 3692 raft_consensus.cc:628] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 57bea6dc182c41a78304417a3cbfc4a7, State: Running, Role: LEADER I0703 01:44:18.452704 3692 consensus_queue.cc:227] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 393, Committed index: 393, Last appended: 1.393, Last appended by leader: 393, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 381 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: false } } I0703 01:44:18.453006 3692 consensus_peers.cc:455] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377): Closing peer: 1157c91cc6f54a68a1d99017ac8444aa I0703 01:44:18.453102 3692 consensus_peers.cc:455] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 -> Peer aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197:51329): Closing peer: aa8b9c58b466436faa5086e85b5224a7 I0703 01:44:18.453194 3692 raft_consensus.cc:2730] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7: Snoozing failure detection for 3.000s (explicit stepdown request) I0703 01:44:18.542083 5097 raft_consensus.cc:1026] T 60d3228294e945c5b343ae62e76dc2f0 P 6932f1cd716a48fe8d6c5573acfc48bf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.405->[1.406-1.496] Dedup: 1.428->[1.429-1.496] I0703 01:44:18.625707 5978 raft_consensus.cc:890] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54: attempting to promote NON_VOTER 6932f1cd716a48fe8d6c5573acfc48bf to VOTER I0703 01:44:18.626078 5978 consensus_peers.cc:455] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 -> Peer 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198:51033): Closing peer: 6932f1cd716a48fe8d6c5573acfc48bf I0703 01:44:18.626147 5978 consensus_peers.cc:455] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 -> Peer 213596374e4142dca6ecfc701ba1c566 (127.5.240.196:51288): Closing peer: 213596374e4142dca6ecfc701ba1c566 I0703 01:44:18.626325 5978 consensus_queue.cc:204] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 498, Committed index: 498, Last appended: 1.499, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 500 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false } } I0703 01:44:18.629467 4835 raft_consensus.cc:1081] T 60d3228294e945c5b343ae62e76dc2f0 P 213596374e4142dca6ecfc701ba1c566 [term 1 FOLLOWER]: Refusing update from remote peer 55223945fe3d4dd8ac03ba9dab346f54: Log matching property violated. Preceding OpId in replica: term: 1 index: 499. Preceding OpId from leader: term: 1 index: 500. (index mismatch) I0703 01:44:18.630468 5928 consensus_queue.cc:962] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 500, Last known committed idx: 498, Time since last communication: 0.000s I0703 01:44:18.631970 5095 raft_consensus.cc:1081] T 60d3228294e945c5b343ae62e76dc2f0 P 6932f1cd716a48fe8d6c5573acfc48bf [term 1 LEARNER]: Refusing update from remote peer 55223945fe3d4dd8ac03ba9dab346f54: Log matching property violated. Preceding OpId in replica: term: 1 index: 499. Preceding OpId from leader: term: 1 index: 500. (index mismatch) I0703 01:44:18.634493 5928 consensus_queue.cc:962] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 500, Last known committed idx: 498, Time since last communication: 0.000s I0703 01:44:18.635051 5928 raft_consensus.cc:2684] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 LEADER]: Committing config change with OpId 1.500: config changed from index 406 to 500, 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198) changed from NON_VOTER to VOTER. New config: { opid_index: 500 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false } } } I0703 01:44:18.640111 4835 raft_consensus.cc:2684] T 60d3228294e945c5b343ae62e76dc2f0 P 213596374e4142dca6ecfc701ba1c566 [term 1 FOLLOWER]: Committing config change with OpId 1.500: config changed from index 406 to 500, 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198) changed from NON_VOTER to VOTER. New config: { opid_index: 500 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false } } } I0703 01:44:18.640501 5098 raft_consensus.cc:2684] T 60d3228294e945c5b343ae62e76dc2f0 P 6932f1cd716a48fe8d6c5573acfc48bf [term 1 FOLLOWER]: Committing config change with OpId 1.500: config changed from index 406 to 500, 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198) changed from NON_VOTER to VOTER. New config: { opid_index: 500 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false } } } I0703 01:44:18.640604 3445 catalog_manager.cc:3798] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 reported cstate change: config changed from index 406 to 500, 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "55223945fe3d4dd8ac03ba9dab346f54" committed_config { opid_index: 500 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false } health_report { overall_health: UNKNOWN } } } I0703 01:44:18.762802 3570 tablet_service.cc:1140] Received LeaderStepDown RPC: tablet_id: "60d3228294e945c5b343ae62e76dc2f0" dest_uuid: "55223945fe3d4dd8ac03ba9dab346f54" from {username='jenkins'} at 127.5.240.193:52839 I0703 01:44:18.763010 3570 raft_consensus.cc:552] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 LEADER]: Received request to step down I0703 01:44:18.763092 3570 raft_consensus.cc:628] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 55223945fe3d4dd8ac03ba9dab346f54, State: Running, Role: LEADER I0703 01:44:18.763353 3570 consensus_queue.cc:227] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 503, Committed index: 503, Last appended: 1.504, Last appended by leader: 504, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 500 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false } } I0703 01:44:18.763523 3570 consensus_peers.cc:455] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 -> Peer 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198:51033): Closing peer: 6932f1cd716a48fe8d6c5573acfc48bf I0703 01:44:18.763576 3570 consensus_peers.cc:455] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 -> Peer 213596374e4142dca6ecfc701ba1c566 (127.5.240.196:51288): Closing peer: 213596374e4142dca6ecfc701ba1c566 W0703 01:44:18.763648 5978 consensus_queue.cc:1041] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" responder_term: 1 status { last_received { term: 1 index: 503 } last_committed_idx: 503 last_received_current_leader { term: 1 index: 503 } } I0703 01:44:18.763952 3570 raft_consensus.cc:2730] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54: Snoozing failure detection for 3.000s (explicit stepdown request) I0703 01:44:19.536334 6101 raft_consensus.cc:457] T c352f70a51154a85a5a5b0f3d14d405f P 6932f1cd716a48fe8d6c5573acfc48bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c1e8943f5654f71b030ec7c0b733742) I0703 01:44:19.536880 6101 raft_consensus.cc:2730] T c352f70a51154a85a5a5b0f3d14d405f P 6932f1cd716a48fe8d6c5573acfc48bf: Snoozing failure detection for 1.805s (starting election) I0703 01:44:19.536991 6101 raft_consensus.cc:479] T c352f70a51154a85a5a5b0f3d14d405f P 6932f1cd716a48fe8d6c5573acfc48bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 377 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false } } I0703 01:44:19.537729 6101 leader_election.cc:243] T c352f70a51154a85a5a5b0f3d14d405f P 6932f1cd716a48fe8d6c5573acfc48bf [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064) I0703 01:44:19.537997 6101 leader_election.cc:243] T c352f70a51154a85a5a5b0f3d14d405f P 6932f1cd716a48fe8d6c5573acfc48bf [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197:51329) I0703 01:44:19.538656 3816 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "c352f70a51154a85a5a5b0f3d14d405f" candidate_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 385 } } ignore_live_leader: false dest_uuid: "8c1e8943f5654f71b030ec7c0b733742" is_pre_election: true I0703 01:44:19.538835 3816 raft_consensus.cc:2730] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742: Snoozing failure detection for 1.818s (vote granted) I0703 01:44:19.538908 3816 raft_consensus.cc:2243] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6932f1cd716a48fe8d6c5573acfc48bf in term 1. I0703 01:44:19.539917 5031 leader_election.cc:379] T c352f70a51154a85a5a5b0f3d14d405f P 6932f1cd716a48fe8d6c5573acfc48bf [CANDIDATE]: Term 2 pre-election: Vote granted by peer 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064) I0703 01:44:19.540086 5031 leader_election.cc:270] T c352f70a51154a85a5a5b0f3d14d405f P 6932f1cd716a48fe8d6c5573acfc48bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. I0703 01:44:19.540796 6101 raft_consensus.cc:2730] T c352f70a51154a85a5a5b0f3d14d405f P 6932f1cd716a48fe8d6c5573acfc48bf: Snoozing failure detection for 2.075s (election complete) I0703 01:44:19.540910 6101 raft_consensus.cc:2537] T c352f70a51154a85a5a5b0f3d14d405f P 6932f1cd716a48fe8d6c5573acfc48bf [term 1 FOLLOWER]: Leader pre-election won for term 2 I0703 01:44:19.541175 6101 raft_consensus.cc:457] T c352f70a51154a85a5a5b0f3d14d405f P 6932f1cd716a48fe8d6c5573acfc48bf [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8c1e8943f5654f71b030ec7c0b733742) I0703 01:44:19.541275 6101 raft_consensus.cc:2730] T c352f70a51154a85a5a5b0f3d14d405f P 6932f1cd716a48fe8d6c5573acfc48bf: Snoozing failure detection for 1.740s (starting election) I0703 01:44:19.541368 6101 raft_consensus.cc:2782] T c352f70a51154a85a5a5b0f3d14d405f P 6932f1cd716a48fe8d6c5573acfc48bf [term 1 FOLLOWER]: Advancing to term 2 I0703 01:44:19.546054 6101 raft_consensus.cc:479] T c352f70a51154a85a5a5b0f3d14d405f P 6932f1cd716a48fe8d6c5573acfc48bf [term 2 FOLLOWER]: Starting leader election with config: opid_index: 377 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false } } I0703 01:44:19.547092 6101 leader_election.cc:243] T c352f70a51154a85a5a5b0f3d14d405f P 6932f1cd716a48fe8d6c5573acfc48bf [CANDIDATE]: Term 2 election: Requesting vote from peer 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064) I0703 01:44:19.547894 6101 leader_election.cc:243] T c352f70a51154a85a5a5b0f3d14d405f P 6932f1cd716a48fe8d6c5573acfc48bf [CANDIDATE]: Term 2 election: Requesting vote from peer aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197:51329) I0703 01:44:19.548724 3816 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "c352f70a51154a85a5a5b0f3d14d405f" candidate_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 385 } } ignore_live_leader: false dest_uuid: "8c1e8943f5654f71b030ec7c0b733742" I0703 01:44:19.548864 3816 raft_consensus.cc:2782] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 [term 1 FOLLOWER]: Advancing to term 2 I0703 01:44:19.548941 3816 raft_consensus.cc:2730] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742: Snoozing failure detection for 1.639s (vote granted) I0703 01:44:19.548946 6104 raft_consensus.cc:457] T c352f70a51154a85a5a5b0f3d14d405f P aa8b9c58b466436faa5086e85b5224a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8c1e8943f5654f71b030ec7c0b733742) I0703 01:44:19.549170 6104 raft_consensus.cc:2730] T c352f70a51154a85a5a5b0f3d14d405f P aa8b9c58b466436faa5086e85b5224a7: Snoozing failure detection for 2.182s (starting election) I0703 01:44:19.549561 6104 raft_consensus.cc:479] T c352f70a51154a85a5a5b0f3d14d405f P aa8b9c58b466436faa5086e85b5224a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 377 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false } } I0703 01:44:19.550822 3816 raft_consensus.cc:2243] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6932f1cd716a48fe8d6c5573acfc48bf in term 2. I0703 01:44:19.550745 6104 leader_election.cc:243] T c352f70a51154a85a5a5b0f3d14d405f P aa8b9c58b466436faa5086e85b5224a7 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064) I0703 01:44:19.551450 5031 leader_election.cc:379] T c352f70a51154a85a5a5b0f3d14d405f P 6932f1cd716a48fe8d6c5573acfc48bf [CANDIDATE]: Term 2 election: Vote granted by peer 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064) I0703 01:44:19.551537 5031 leader_election.cc:270] T c352f70a51154a85a5a5b0f3d14d405f P 6932f1cd716a48fe8d6c5573acfc48bf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0703 01:44:19.551959 6101 raft_consensus.cc:2730] T c352f70a51154a85a5a5b0f3d14d405f P 6932f1cd716a48fe8d6c5573acfc48bf: Snoozing failure detection for 1.704s (election complete) I0703 01:44:19.552034 6101 raft_consensus.cc:2537] T c352f70a51154a85a5a5b0f3d14d405f P 6932f1cd716a48fe8d6c5573acfc48bf [term 2 FOLLOWER]: Leader election won for term 2 I0703 01:44:19.552429 6101 raft_consensus.cc:593] T c352f70a51154a85a5a5b0f3d14d405f P 6932f1cd716a48fe8d6c5573acfc48bf [term 2 LEADER]: Becoming Leader. State: Replica: 6932f1cd716a48fe8d6c5573acfc48bf, State: Running, Role: LEADER I0703 01:44:19.552857 6101 consensus_queue.cc:204] T c352f70a51154a85a5a5b0f3d14d405f P 6932f1cd716a48fe8d6c5573acfc48bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 385, Committed index: 385, Last appended: 1.385, Last appended by leader: 385, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 377 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false } } I0703 01:44:19.557590 3448 catalog_manager.cc:3798] T c352f70a51154a85a5a5b0f3d14d405f P 6932f1cd716a48fe8d6c5573acfc48bf reported cstate change: term changed from 1 to 2, leader changed from 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195) to 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198). New cstate: current_term: 2 leader_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" committed_config { opid_index: 377 OBSOLETE_local: false peers { permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I0703 01:44:19.560104 3816 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "c352f70a51154a85a5a5b0f3d14d405f" candidate_uuid: "aa8b9c58b466436faa5086e85b5224a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 385 } } ignore_live_leader: false dest_uuid: "8c1e8943f5654f71b030ec7c0b733742" is_pre_election: true I0703 01:44:19.560330 3816 raft_consensus.cc:2173] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate aa8b9c58b466436faa5086e85b5224a7 in current term 2: Already voted for candidate 6932f1cd716a48fe8d6c5573acfc48bf in this term. I0703 01:44:19.561089 4897 leader_election.cc:393] T c352f70a51154a85a5a5b0f3d14d405f P aa8b9c58b466436faa5086e85b5224a7 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064). Message: Invalid argument: T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate aa8b9c58b466436faa5086e85b5224a7 in current term 2: Already voted for candidate 6932f1cd716a48fe8d6c5573acfc48bf in this term. I0703 01:44:19.565157 4963 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "c352f70a51154a85a5a5b0f3d14d405f" candidate_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 385 } } ignore_live_leader: false dest_uuid: "aa8b9c58b466436faa5086e85b5224a7" is_pre_election: true I0703 01:44:19.565372 4963 raft_consensus.cc:2730] T c352f70a51154a85a5a5b0f3d14d405f P aa8b9c58b466436faa5086e85b5224a7: Snoozing failure detection for 2.064s (vote granted) I0703 01:44:19.565454 4963 raft_consensus.cc:2243] T c352f70a51154a85a5a5b0f3d14d405f P aa8b9c58b466436faa5086e85b5224a7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6932f1cd716a48fe8d6c5573acfc48bf in term 1. I0703 01:44:19.565965 4965 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "c352f70a51154a85a5a5b0f3d14d405f" candidate_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 385 } } ignore_live_leader: false dest_uuid: "aa8b9c58b466436faa5086e85b5224a7" I0703 01:44:19.566186 4965 raft_consensus.cc:2782] T c352f70a51154a85a5a5b0f3d14d405f P aa8b9c58b466436faa5086e85b5224a7 [term 1 FOLLOWER]: Advancing to term 2 I0703 01:44:19.566273 4965 raft_consensus.cc:2730] T c352f70a51154a85a5a5b0f3d14d405f P aa8b9c58b466436faa5086e85b5224a7: Snoozing failure detection for 1.982s (vote granted) I0703 01:44:19.568850 5028 leader_election.cc:379] T c352f70a51154a85a5a5b0f3d14d405f P 6932f1cd716a48fe8d6c5573acfc48bf [CANDIDATE]: Term 2 pre-election: Vote granted by peer aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197:51329) I0703 01:44:19.568857 4965 raft_consensus.cc:2243] T c352f70a51154a85a5a5b0f3d14d405f P aa8b9c58b466436faa5086e85b5224a7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6932f1cd716a48fe8d6c5573acfc48bf in term 2. I0703 01:44:19.569527 6104 leader_election.cc:243] T c352f70a51154a85a5a5b0f3d14d405f P aa8b9c58b466436faa5086e85b5224a7 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198:51033) I0703 01:44:19.576691 5028 leader_election.cc:379] T c352f70a51154a85a5a5b0f3d14d405f P 6932f1cd716a48fe8d6c5573acfc48bf [CANDIDATE]: Term 2 election: Vote granted by peer aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197:51329) I0703 01:44:19.581702 4961 raft_consensus.cc:1081] T c352f70a51154a85a5a5b0f3d14d405f P aa8b9c58b466436faa5086e85b5224a7 [term 2 FOLLOWER]: Refusing update from remote peer 6932f1cd716a48fe8d6c5573acfc48bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 385. Preceding OpId from leader: term: 2 index: 387. (index mismatch) I0703 01:44:19.582352 6105 consensus_queue.cc:962] T c352f70a51154a85a5a5b0f3d14d405f P 6932f1cd716a48fe8d6c5573acfc48bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 386, Last known committed idx: 385, Time since last communication: 0.000s I0703 01:44:19.583256 3816 raft_consensus.cc:1081] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 [term 2 FOLLOWER]: Refusing update from remote peer 6932f1cd716a48fe8d6c5573acfc48bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 385. Preceding OpId from leader: term: 2 index: 387. (index mismatch) I0703 01:44:19.585978 6105 consensus_queue.cc:962] T c352f70a51154a85a5a5b0f3d14d405f P 6932f1cd716a48fe8d6c5573acfc48bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c1e8943f5654f71b030ec7c0b733742" member_type: VOTER last_known_addr { host: "127.5.240.195" port: 51064 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 386, Last known committed idx: 385, Time since last communication: 0.000s I0703 01:44:19.586856 5098 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "c352f70a51154a85a5a5b0f3d14d405f" candidate_uuid: "aa8b9c58b466436faa5086e85b5224a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 385 } } ignore_live_leader: false dest_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" is_pre_election: true I0703 01:44:19.587321 5098 raft_consensus.cc:2204] T c352f70a51154a85a5a5b0f3d14d405f P 6932f1cd716a48fe8d6c5573acfc48bf [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate aa8b9c58b466436faa5086e85b5224a7 for term 2 because replica is either leader or believes a valid leader to be alive. I0703 01:44:19.587996 4897 leader_election.cc:393] T c352f70a51154a85a5a5b0f3d14d405f P aa8b9c58b466436faa5086e85b5224a7 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198:51033). Message: Invalid argument: T c352f70a51154a85a5a5b0f3d14d405f P 6932f1cd716a48fe8d6c5573acfc48bf [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate aa8b9c58b466436faa5086e85b5224a7 for term 2 because replica is either leader or believes a valid leader to be alive. I0703 01:44:19.588129 4897 leader_election.cc:270] T c352f70a51154a85a5a5b0f3d14d405f P aa8b9c58b466436faa5086e85b5224a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0703 01:44:19.590152 6104 raft_consensus.cc:2730] T c352f70a51154a85a5a5b0f3d14d405f P aa8b9c58b466436faa5086e85b5224a7: Snoozing failure detection for 2.211s (election complete) I0703 01:44:19.593156 6104 raft_consensus.cc:2485] T c352f70a51154a85a5a5b0f3d14d405f P aa8b9c58b466436faa5086e85b5224a7 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0703 01:44:19.616502 3446 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet c352f70a51154a85a5a5b0f3d14d405f (attempt 1) I0703 01:44:19.646719 5098 consensus_peers.cc:455] T c352f70a51154a85a5a5b0f3d14d405f P 6932f1cd716a48fe8d6c5573acfc48bf -> Peer aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197:51329): Closing peer: aa8b9c58b466436faa5086e85b5224a7 I0703 01:44:19.646808 5098 consensus_peers.cc:455] T c352f70a51154a85a5a5b0f3d14d405f P 6932f1cd716a48fe8d6c5573acfc48bf -> Peer 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064): Closing peer: 8c1e8943f5654f71b030ec7c0b733742 I0703 01:44:19.647112 5098 consensus_queue.cc:204] T c352f70a51154a85a5a5b0f3d14d405f P 6932f1cd716a48fe8d6c5573acfc48bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 387, Committed index: 387, Last appended: 2.387, Last appended by leader: 385, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 388 OBSOLETE_local: false peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false } } I0703 01:44:19.649147 4965 raft_consensus.cc:1081] T c352f70a51154a85a5a5b0f3d14d405f P aa8b9c58b466436faa5086e85b5224a7 [term 2 FOLLOWER]: Refusing update from remote peer 6932f1cd716a48fe8d6c5573acfc48bf: Log matching property violated. Preceding OpId in replica: term: 2 index: 387. Preceding OpId from leader: term: 2 index: 388. (index mismatch) I0703 01:44:19.649828 6105 consensus_queue.cc:962] T c352f70a51154a85a5a5b0f3d14d405f P 6932f1cd716a48fe8d6c5573acfc48bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 388, Last known committed idx: 387, Time since last communication: 0.000s I0703 01:44:19.652125 6101 raft_consensus.cc:2684] T c352f70a51154a85a5a5b0f3d14d405f P 6932f1cd716a48fe8d6c5573acfc48bf [term 2 LEADER]: Committing config change with OpId 2.388: config changed from index 377 to 388, VOTER 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195) evicted. New config: { opid_index: 388 OBSOLETE_local: false peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false } } } I0703 01:44:19.653221 4965 raft_consensus.cc:2684] T c352f70a51154a85a5a5b0f3d14d405f P aa8b9c58b466436faa5086e85b5224a7 [term 2 FOLLOWER]: Committing config change with OpId 2.388: config changed from index 377 to 388, VOTER 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195) evicted. New config: { opid_index: 388 OBSOLETE_local: false peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false } } } I0703 01:44:19.655351 3437 catalog_manager.cc:3352] ChangeConfig:REMOVE_PEER RPC for tablet c352f70a51154a85a5a5b0f3d14d405f with cas_config_opid_index 377: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0703 01:44:19.656128 3446 catalog_manager.cc:3798] T c352f70a51154a85a5a5b0f3d14d405f P 6932f1cd716a48fe8d6c5573acfc48bf reported cstate change: config changed from index 377 to 388, VOTER 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195) evicted. New cstate: current_term: 2 leader_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" committed_config { opid_index: 388 OBSOLETE_local: false peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I0703 01:44:19.658577 3446 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet c352f70a51154a85a5a5b0f3d14d405f on 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064) (TS 8c1e8943f5654f71b030ec7c0b733742 not found in new config with opid_index 388) I0703 01:44:19.659168 3796 tablet_service.cc:800] Processing DeleteTablet for tablet c352f70a51154a85a5a5b0f3d14d405f with delete_type TABLET_DATA_TOMBSTONED (TS 8c1e8943f5654f71b030ec7c0b733742 not found in new config with opid_index 388) from {username='jenkins'} at 127.5.240.195:39619 I0703 01:44:19.662533 6133 tablet_replica.cc:261] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742: stopping tablet replica I0703 01:44:19.662758 6133 maintenance_manager.cc:237] P 8c1e8943f5654f71b030ec7c0b733742: Unregistered op CompactRowSetsOp(c352f70a51154a85a5a5b0f3d14d405f) I0703 01:44:19.662889 6133 maintenance_manager.cc:237] P 8c1e8943f5654f71b030ec7c0b733742: Unregistered op MinorDeltaCompactionOp(c352f70a51154a85a5a5b0f3d14d405f) I0703 01:44:19.663002 6133 maintenance_manager.cc:237] P 8c1e8943f5654f71b030ec7c0b733742: Unregistered op MajorDeltaCompactionOp(c352f70a51154a85a5a5b0f3d14d405f) I0703 01:44:19.663120 6133 maintenance_manager.cc:237] P 8c1e8943f5654f71b030ec7c0b733742: Unregistered op UndoDeltaBlockGCOp(c352f70a51154a85a5a5b0f3d14d405f) I0703 01:44:19.663233 6133 maintenance_manager.cc:237] P 8c1e8943f5654f71b030ec7c0b733742: Unregistered op FlushMRSOp(c352f70a51154a85a5a5b0f3d14d405f) I0703 01:44:19.663338 6133 maintenance_manager.cc:237] P 8c1e8943f5654f71b030ec7c0b733742: Unregistered op FlushDeltaMemStoresOp(c352f70a51154a85a5a5b0f3d14d405f) I0703 01:44:19.663401 6133 maintenance_manager.cc:237] P 8c1e8943f5654f71b030ec7c0b733742: Unregistered op LogGCOp(c352f70a51154a85a5a5b0f3d14d405f) I0703 01:44:19.663475 6133 raft_consensus.cc:2040] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 [term 2 FOLLOWER]: Raft consensus shutting down. I0703 01:44:19.663622 6133 raft_consensus.cc:2067] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742 [term 2 FOLLOWER]: Raft consensus is shut down! I0703 01:44:19.664428 6133 ts_tablet_manager.cc:1369] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:44:19.676656 6133 ts_tablet_manager.cc:1382] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.387 I0703 01:44:19.676839 6133 log.cc:986] T c352f70a51154a85a5a5b0f3d14d405f P 8c1e8943f5654f71b030ec7c0b733742: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-2/wal/wals/c352f70a51154a85a5a5b0f3d14d405f I0703 01:44:19.681572 3437 catalog_manager.cc:3103] TS 8c1e8943f5654f71b030ec7c0b733742 (127.5.240.195:51064): tablet c352f70a51154a85a5a5b0f3d14d405f (table rebalance_test_table_1 [id=29732119416840ec89c7a3633f79c5fd]) successfully deleted I0703 01:44:19.950497 6104 raft_consensus.cc:457] T 049bd2ef267f47fca634858420ac35b9 P aa8b9c58b466436faa5086e85b5224a7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 57bea6dc182c41a78304417a3cbfc4a7) I0703 01:44:19.950610 6104 raft_consensus.cc:2730] T 049bd2ef267f47fca634858420ac35b9 P aa8b9c58b466436faa5086e85b5224a7: Snoozing failure detection for 1.594s (starting election) I0703 01:44:19.950695 6104 raft_consensus.cc:479] T 049bd2ef267f47fca634858420ac35b9 P aa8b9c58b466436faa5086e85b5224a7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 381 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: false } } I0703 01:44:19.951205 6104 leader_election.cc:243] T 049bd2ef267f47fca634858420ac35b9 P aa8b9c58b466436faa5086e85b5224a7 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746) I0703 01:44:19.951345 6104 leader_election.cc:243] T 049bd2ef267f47fca634858420ac35b9 P aa8b9c58b466436faa5086e85b5224a7 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377) I0703 01:44:19.955801 3692 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "049bd2ef267f47fca634858420ac35b9" candidate_uuid: "aa8b9c58b466436faa5086e85b5224a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 393 } } ignore_live_leader: false dest_uuid: "57bea6dc182c41a78304417a3cbfc4a7" is_pre_election: true I0703 01:44:19.955991 3692 raft_consensus.cc:2730] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7: Snoozing failure detection for 1.724s (vote granted) I0703 01:44:19.956041 3692 raft_consensus.cc:2243] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa8b9c58b466436faa5086e85b5224a7 in term 1. I0703 01:44:19.959292 4894 leader_election.cc:379] T 049bd2ef267f47fca634858420ac35b9 P aa8b9c58b466436faa5086e85b5224a7 [CANDIDATE]: Term 2 pre-election: Vote granted by peer 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746) I0703 01:44:19.959384 4894 leader_election.cc:270] T 049bd2ef267f47fca634858420ac35b9 P aa8b9c58b466436faa5086e85b5224a7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. I0703 01:44:19.959722 6104 raft_consensus.cc:2730] T 049bd2ef267f47fca634858420ac35b9 P aa8b9c58b466436faa5086e85b5224a7: Snoozing failure detection for 1.853s (election complete) I0703 01:44:19.959789 6104 raft_consensus.cc:2537] T 049bd2ef267f47fca634858420ac35b9 P aa8b9c58b466436faa5086e85b5224a7 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0703 01:44:19.959846 6104 raft_consensus.cc:457] T 049bd2ef267f47fca634858420ac35b9 P aa8b9c58b466436faa5086e85b5224a7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 57bea6dc182c41a78304417a3cbfc4a7) I0703 01:44:19.959899 6104 raft_consensus.cc:2730] T 049bd2ef267f47fca634858420ac35b9 P aa8b9c58b466436faa5086e85b5224a7: Snoozing failure detection for 1.599s (starting election) I0703 01:44:19.959964 6104 raft_consensus.cc:2782] T 049bd2ef267f47fca634858420ac35b9 P aa8b9c58b466436faa5086e85b5224a7 [term 1 FOLLOWER]: Advancing to term 2 I0703 01:44:19.962054 6104 raft_consensus.cc:479] T 049bd2ef267f47fca634858420ac35b9 P aa8b9c58b466436faa5086e85b5224a7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 381 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: false } } I0703 01:44:19.962630 6104 leader_election.cc:243] T 049bd2ef267f47fca634858420ac35b9 P aa8b9c58b466436faa5086e85b5224a7 [CANDIDATE]: Term 2 election: Requesting vote from peer 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746) I0703 01:44:19.962815 6104 leader_election.cc:243] T 049bd2ef267f47fca634858420ac35b9 P aa8b9c58b466436faa5086e85b5224a7 [CANDIDATE]: Term 2 election: Requesting vote from peer 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377) I0703 01:44:19.966298 3692 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "049bd2ef267f47fca634858420ac35b9" candidate_uuid: "aa8b9c58b466436faa5086e85b5224a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 393 } } ignore_live_leader: false dest_uuid: "57bea6dc182c41a78304417a3cbfc4a7" I0703 01:44:19.966411 3692 raft_consensus.cc:2782] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 [term 1 FOLLOWER]: Advancing to term 2 I0703 01:44:19.966508 3692 raft_consensus.cc:2730] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7: Snoozing failure detection for 1.655s (vote granted) I0703 01:44:19.968652 3692 raft_consensus.cc:2243] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa8b9c58b466436faa5086e85b5224a7 in term 2. I0703 01:44:19.969233 4894 leader_election.cc:379] T 049bd2ef267f47fca634858420ac35b9 P aa8b9c58b466436faa5086e85b5224a7 [CANDIDATE]: Term 2 election: Vote granted by peer 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746) I0703 01:44:19.969301 4894 leader_election.cc:270] T 049bd2ef267f47fca634858420ac35b9 P aa8b9c58b466436faa5086e85b5224a7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0703 01:44:19.969488 6104 raft_consensus.cc:2730] T 049bd2ef267f47fca634858420ac35b9 P aa8b9c58b466436faa5086e85b5224a7: Snoozing failure detection for 2.149s (election complete) I0703 01:44:19.969600 6104 raft_consensus.cc:2537] T 049bd2ef267f47fca634858420ac35b9 P aa8b9c58b466436faa5086e85b5224a7 [term 2 FOLLOWER]: Leader election won for term 2 I0703 01:44:19.975939 6153 raft_consensus.cc:457] T 049bd2ef267f47fca634858420ac35b9 P 1157c91cc6f54a68a1d99017ac8444aa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 57bea6dc182c41a78304417a3cbfc4a7) I0703 01:44:19.976096 6153 raft_consensus.cc:2730] T 049bd2ef267f47fca634858420ac35b9 P 1157c91cc6f54a68a1d99017ac8444aa: Snoozing failure detection for 1.870s (starting election) I0703 01:44:19.976207 6153 raft_consensus.cc:479] T 049bd2ef267f47fca634858420ac35b9 P 1157c91cc6f54a68a1d99017ac8444aa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 381 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: false } } I0703 01:44:19.976848 6153 leader_election.cc:243] T 049bd2ef267f47fca634858420ac35b9 P 1157c91cc6f54a68a1d99017ac8444aa [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746) I0703 01:44:19.977660 3692 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "049bd2ef267f47fca634858420ac35b9" candidate_uuid: "1157c91cc6f54a68a1d99017ac8444aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 393 } } ignore_live_leader: false dest_uuid: "57bea6dc182c41a78304417a3cbfc4a7" is_pre_election: true I0703 01:44:19.977829 3692 raft_consensus.cc:2173] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1157c91cc6f54a68a1d99017ac8444aa in current term 2: Already voted for candidate aa8b9c58b466436faa5086e85b5224a7 in this term. I0703 01:44:19.979480 5161 leader_election.cc:393] T 049bd2ef267f47fca634858420ac35b9 P 1157c91cc6f54a68a1d99017ac8444aa [CANDIDATE]: Term 2 pre-election: Vote denied by peer 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746). Message: Invalid argument: T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1157c91cc6f54a68a1d99017ac8444aa in current term 2: Already voted for candidate aa8b9c58b466436faa5086e85b5224a7 in this term. I0703 01:44:19.979694 6153 leader_election.cc:243] T 049bd2ef267f47fca634858420ac35b9 P 1157c91cc6f54a68a1d99017ac8444aa [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197:51329) I0703 01:44:19.984876 6104 raft_consensus.cc:593] T 049bd2ef267f47fca634858420ac35b9 P aa8b9c58b466436faa5086e85b5224a7 [term 2 LEADER]: Becoming Leader. State: Replica: aa8b9c58b466436faa5086e85b5224a7, State: Running, Role: LEADER I0703 01:44:19.985306 6104 consensus_queue.cc:204] T 049bd2ef267f47fca634858420ac35b9 P aa8b9c58b466436faa5086e85b5224a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 392, Committed index: 392, Last appended: 1.393, Last appended by leader: 393, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 381 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: false } } I0703 01:44:19.994082 3448 catalog_manager.cc:3798] T 049bd2ef267f47fca634858420ac35b9 P aa8b9c58b466436faa5086e85b5224a7 reported cstate change: term changed from 1 to 2, leader changed from 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194) to aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197). New cstate: current_term: 2 leader_uuid: "aa8b9c58b466436faa5086e85b5224a7" committed_config { opid_index: 381 OBSOLETE_local: false peers { permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: false } health_report { overall_health: UNKNOWN } } } I0703 01:44:20.000805 4961 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "049bd2ef267f47fca634858420ac35b9" candidate_uuid: "1157c91cc6f54a68a1d99017ac8444aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 393 } } ignore_live_leader: false dest_uuid: "aa8b9c58b466436faa5086e85b5224a7" is_pre_election: true I0703 01:44:20.001065 4961 raft_consensus.cc:2204] T 049bd2ef267f47fca634858420ac35b9 P aa8b9c58b466436faa5086e85b5224a7 [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 1157c91cc6f54a68a1d99017ac8444aa for term 2 because replica is either leader or believes a valid leader to be alive. I0703 01:44:20.002005 5161 leader_election.cc:393] T 049bd2ef267f47fca634858420ac35b9 P 1157c91cc6f54a68a1d99017ac8444aa [CANDIDATE]: Term 2 pre-election: Vote denied by peer aa8b9c58b466436faa5086e85b5224a7 (127.5.240.197:51329). Message: Invalid argument: T 049bd2ef267f47fca634858420ac35b9 P aa8b9c58b466436faa5086e85b5224a7 [term 2 LEADER]: Leader pre-election vote request: Denying vote to candidate 1157c91cc6f54a68a1d99017ac8444aa for term 2 because replica is either leader or believes a valid leader to be alive. I0703 01:44:20.002110 5161 leader_election.cc:270] T 049bd2ef267f47fca634858420ac35b9 P 1157c91cc6f54a68a1d99017ac8444aa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0703 01:44:20.002342 6153 raft_consensus.cc:2730] T 049bd2ef267f47fca634858420ac35b9 P 1157c91cc6f54a68a1d99017ac8444aa: Snoozing failure detection for 1.656s (election complete) I0703 01:44:20.002451 6153 raft_consensus.cc:2782] T 049bd2ef267f47fca634858420ac35b9 P 1157c91cc6f54a68a1d99017ac8444aa [term 1 FOLLOWER]: Advancing to term 2 I0703 01:44:20.003116 5234 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "049bd2ef267f47fca634858420ac35b9" candidate_uuid: "aa8b9c58b466436faa5086e85b5224a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 393 } } ignore_live_leader: false dest_uuid: "1157c91cc6f54a68a1d99017ac8444aa" is_pre_election: true I0703 01:44:20.004804 6153 raft_consensus.cc:2485] T 049bd2ef267f47fca634858420ac35b9 P 1157c91cc6f54a68a1d99017ac8444aa [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0703 01:44:20.004983 5233 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "049bd2ef267f47fca634858420ac35b9" candidate_uuid: "aa8b9c58b466436faa5086e85b5224a7" candidate_term: 2 candidate_status { last_received { term: 1 index: 393 } } ignore_live_leader: false dest_uuid: "1157c91cc6f54a68a1d99017ac8444aa" I0703 01:44:20.005131 5233 raft_consensus.cc:2218] T 049bd2ef267f47fca634858420ac35b9 P 1157c91cc6f54a68a1d99017ac8444aa [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate aa8b9c58b466436faa5086e85b5224a7 for term 2 because replica is already servicing an update from a current leader or another vote. I0703 01:44:20.005738 5234 raft_consensus.cc:2730] T 049bd2ef267f47fca634858420ac35b9 P 1157c91cc6f54a68a1d99017ac8444aa: Snoozing failure detection for 2.473s (vote granted) I0703 01:44:20.005810 5234 raft_consensus.cc:2243] T 049bd2ef267f47fca634858420ac35b9 P 1157c91cc6f54a68a1d99017ac8444aa [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa8b9c58b466436faa5086e85b5224a7 in term 2. I0703 01:44:20.005831 4894 leader_election.cc:393] T 049bd2ef267f47fca634858420ac35b9 P aa8b9c58b466436faa5086e85b5224a7 [CANDIDATE]: Term 2 election: Vote denied by peer 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377). Message: Service unavailable: T 049bd2ef267f47fca634858420ac35b9 P 1157c91cc6f54a68a1d99017ac8444aa [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate aa8b9c58b466436faa5086e85b5224a7 for term 2 because replica is already servicing an update from a current leader or another vote. I0703 01:44:20.006413 4894 leader_election.cc:379] T 049bd2ef267f47fca634858420ac35b9 P aa8b9c58b466436faa5086e85b5224a7 [CANDIDATE]: Term 2 pre-election: Vote granted by peer 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377) I0703 01:44:20.021670 5233 raft_consensus.cc:1081] T 049bd2ef267f47fca634858420ac35b9 P 1157c91cc6f54a68a1d99017ac8444aa [term 2 FOLLOWER]: Refusing update from remote peer aa8b9c58b466436faa5086e85b5224a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 393. Preceding OpId from leader: term: 2 index: 395. (index mismatch) I0703 01:44:20.022153 3692 raft_consensus.cc:1081] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 [term 2 FOLLOWER]: Refusing update from remote peer aa8b9c58b466436faa5086e85b5224a7: Log matching property violated. Preceding OpId in replica: term: 1 index: 393. Preceding OpId from leader: term: 2 index: 395. (index mismatch) I0703 01:44:20.022327 6154 consensus_queue.cc:962] T 049bd2ef267f47fca634858420ac35b9 P aa8b9c58b466436faa5086e85b5224a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 394, Last known committed idx: 393, Time since last communication: 0.000s I0703 01:44:20.024590 6154 consensus_queue.cc:962] T 049bd2ef267f47fca634858420ac35b9 P aa8b9c58b466436faa5086e85b5224a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57bea6dc182c41a78304417a3cbfc4a7" member_type: VOTER last_known_addr { host: "127.5.240.194" port: 59746 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 394, Last known committed idx: 393, Time since last communication: 0.000s I0703 01:44:20.050878 3446 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 049bd2ef267f47fca634858420ac35b9 (attempt 1) I0703 01:44:20.088981 4961 consensus_peers.cc:455] T 049bd2ef267f47fca634858420ac35b9 P aa8b9c58b466436faa5086e85b5224a7 -> Peer 1157c91cc6f54a68a1d99017ac8444aa (127.5.240.199:55377): Closing peer: 1157c91cc6f54a68a1d99017ac8444aa I0703 01:44:20.089386 4961 consensus_peers.cc:455] T 049bd2ef267f47fca634858420ac35b9 P aa8b9c58b466436faa5086e85b5224a7 -> Peer 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746): Closing peer: 57bea6dc182c41a78304417a3cbfc4a7 I0703 01:44:20.089807 4961 consensus_queue.cc:204] T 049bd2ef267f47fca634858420ac35b9 P aa8b9c58b466436faa5086e85b5224a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 395, Committed index: 395, Last appended: 2.395, Last appended by leader: 393, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 396 OBSOLETE_local: false peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: false } } I0703 01:44:20.094554 5233 raft_consensus.cc:1081] T 049bd2ef267f47fca634858420ac35b9 P 1157c91cc6f54a68a1d99017ac8444aa [term 2 FOLLOWER]: Refusing update from remote peer aa8b9c58b466436faa5086e85b5224a7: Log matching property violated. Preceding OpId in replica: term: 2 index: 395. Preceding OpId from leader: term: 2 index: 396. (index mismatch) I0703 01:44:20.095703 6167 consensus_queue.cc:962] T 049bd2ef267f47fca634858420ac35b9 P aa8b9c58b466436faa5086e85b5224a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 396, Last known committed idx: 395, Time since last communication: 0.000s I0703 01:44:20.098474 6104 raft_consensus.cc:2684] T 049bd2ef267f47fca634858420ac35b9 P aa8b9c58b466436faa5086e85b5224a7 [term 2 LEADER]: Committing config change with OpId 2.396: config changed from index 381 to 396, VOTER 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194) evicted. New config: { opid_index: 396 OBSOLETE_local: false peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: false } } } I0703 01:44:20.102167 3434 catalog_manager.cc:3352] ChangeConfig:REMOVE_PEER RPC for tablet 049bd2ef267f47fca634858420ac35b9 with cas_config_opid_index 381: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0703 01:44:20.102778 5233 raft_consensus.cc:2684] T 049bd2ef267f47fca634858420ac35b9 P 1157c91cc6f54a68a1d99017ac8444aa [term 2 FOLLOWER]: Committing config change with OpId 2.396: config changed from index 381 to 396, VOTER 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194) evicted. New config: { opid_index: 396 OBSOLETE_local: false peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: false } } } I0703 01:44:20.103322 3446 catalog_manager.cc:3798] T 049bd2ef267f47fca634858420ac35b9 P aa8b9c58b466436faa5086e85b5224a7 reported cstate change: config changed from index 381 to 396, VOTER 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194) evicted. New cstate: current_term: 2 leader_uuid: "aa8b9c58b466436faa5086e85b5224a7" committed_config { opid_index: 396 OBSOLETE_local: false peers { permanent_uuid: "aa8b9c58b466436faa5086e85b5224a7" member_type: VOTER last_known_addr { host: "127.5.240.197" port: 51329 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1157c91cc6f54a68a1d99017ac8444aa" member_type: VOTER last_known_addr { host: "127.5.240.199" port: 55377 } attrs { promote: false } health_report { overall_health: HEALTHY } } } I0703 01:44:20.106935 3446 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 049bd2ef267f47fca634858420ac35b9 on 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746) (TS 57bea6dc182c41a78304417a3cbfc4a7 not found in new config with opid_index 396) I0703 01:44:20.107849 3676 tablet_service.cc:800] Processing DeleteTablet for tablet 049bd2ef267f47fca634858420ac35b9 with delete_type TABLET_DATA_TOMBSTONED (TS 57bea6dc182c41a78304417a3cbfc4a7 not found in new config with opid_index 396) from {username='jenkins'} at 127.5.240.194:39884 I0703 01:44:20.121439 6173 tablet_replica.cc:261] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7: stopping tablet replica I0703 01:44:20.121567 6173 maintenance_manager.cc:237] P 57bea6dc182c41a78304417a3cbfc4a7: Unregistered op CompactRowSetsOp(049bd2ef267f47fca634858420ac35b9) I0703 01:44:20.121651 6173 maintenance_manager.cc:237] P 57bea6dc182c41a78304417a3cbfc4a7: Unregistered op MinorDeltaCompactionOp(049bd2ef267f47fca634858420ac35b9) I0703 01:44:20.121731 6173 maintenance_manager.cc:237] P 57bea6dc182c41a78304417a3cbfc4a7: Unregistered op MajorDeltaCompactionOp(049bd2ef267f47fca634858420ac35b9) I0703 01:44:20.128134 6173 maintenance_manager.cc:237] P 57bea6dc182c41a78304417a3cbfc4a7: Unregistered op UndoDeltaBlockGCOp(049bd2ef267f47fca634858420ac35b9) I0703 01:44:20.128221 6173 maintenance_manager.cc:237] P 57bea6dc182c41a78304417a3cbfc4a7: Unregistered op FlushMRSOp(049bd2ef267f47fca634858420ac35b9) I0703 01:44:20.128290 6173 maintenance_manager.cc:237] P 57bea6dc182c41a78304417a3cbfc4a7: Unregistered op FlushDeltaMemStoresOp(049bd2ef267f47fca634858420ac35b9) I0703 01:44:20.128355 6173 maintenance_manager.cc:237] P 57bea6dc182c41a78304417a3cbfc4a7: Unregistered op LogGCOp(049bd2ef267f47fca634858420ac35b9) I0703 01:44:20.128419 6173 raft_consensus.cc:2040] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 [term 2 FOLLOWER]: Raft consensus shutting down. I0703 01:44:20.128600 6173 raft_consensus.cc:2067] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7 [term 2 FOLLOWER]: Raft consensus is shut down! I0703 01:44:20.131479 6173 ts_tablet_manager.cc:1369] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:44:20.145150 6173 ts_tablet_manager.cc:1382] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.395 I0703 01:44:20.145272 6173 log.cc:986] T 049bd2ef267f47fca634858420ac35b9 P 57bea6dc182c41a78304417a3cbfc4a7: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-1/wal/wals/049bd2ef267f47fca634858420ac35b9 I0703 01:44:20.148659 3434 catalog_manager.cc:3103] TS 57bea6dc182c41a78304417a3cbfc4a7 (127.5.240.194:59746): tablet 049bd2ef267f47fca634858420ac35b9 (table rebalance_test_table_0 [id=5bc38732cb8746a7ad1c2b9fede5e7d9]) successfully deleted I0703 01:44:20.262053 6101 raft_consensus.cc:457] T 60d3228294e945c5b343ae62e76dc2f0 P 6932f1cd716a48fe8d6c5573acfc48bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 55223945fe3d4dd8ac03ba9dab346f54) I0703 01:44:20.262145 6101 raft_consensus.cc:2730] T 60d3228294e945c5b343ae62e76dc2f0 P 6932f1cd716a48fe8d6c5573acfc48bf: Snoozing failure detection for 1.948s (starting election) I0703 01:44:20.262301 6101 raft_consensus.cc:479] T 60d3228294e945c5b343ae62e76dc2f0 P 6932f1cd716a48fe8d6c5573acfc48bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 500 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false } } I0703 01:44:20.262929 6101 leader_election.cc:243] T 60d3228294e945c5b343ae62e76dc2f0 P 6932f1cd716a48fe8d6c5573acfc48bf [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193:54919) I0703 01:44:20.263098 6101 leader_election.cc:243] T 60d3228294e945c5b343ae62e76dc2f0 P 6932f1cd716a48fe8d6c5573acfc48bf [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 213596374e4142dca6ecfc701ba1c566 (127.5.240.196:51288) I0703 01:44:20.268564 3573 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "60d3228294e945c5b343ae62e76dc2f0" candidate_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 503 } } ignore_live_leader: false dest_uuid: "55223945fe3d4dd8ac03ba9dab346f54" is_pre_election: true I0703 01:44:20.268828 3573 raft_consensus.cc:2190] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6932f1cd716a48fe8d6c5573acfc48bf for term 2 because replica has last-logged OpId of term: 1 index: 504, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 503. I0703 01:44:20.269927 5031 leader_election.cc:393] T 60d3228294e945c5b343ae62e76dc2f0 P 6932f1cd716a48fe8d6c5573acfc48bf [CANDIDATE]: Term 2 pre-election: Vote denied by peer 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193:54919). Message: Invalid argument: T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6932f1cd716a48fe8d6c5573acfc48bf for term 2 because replica has last-logged OpId of term: 1 index: 504, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 503. I0703 01:44:20.285812 6178 raft_consensus.cc:457] T 60d3228294e945c5b343ae62e76dc2f0 P 213596374e4142dca6ecfc701ba1c566 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 55223945fe3d4dd8ac03ba9dab346f54) I0703 01:44:20.286782 6178 raft_consensus.cc:2730] T 60d3228294e945c5b343ae62e76dc2f0 P 213596374e4142dca6ecfc701ba1c566: Snoozing failure detection for 1.543s (starting election) I0703 01:44:20.286972 6178 raft_consensus.cc:479] T 60d3228294e945c5b343ae62e76dc2f0 P 213596374e4142dca6ecfc701ba1c566 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 500 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false } } I0703 01:44:20.287392 4835 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "60d3228294e945c5b343ae62e76dc2f0" candidate_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 503 } } ignore_live_leader: false dest_uuid: "213596374e4142dca6ecfc701ba1c566" is_pre_election: true I0703 01:44:20.289023 6178 leader_election.cc:243] T 60d3228294e945c5b343ae62e76dc2f0 P 213596374e4142dca6ecfc701ba1c566 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193:54919) I0703 01:44:20.289507 4835 raft_consensus.cc:2190] T 60d3228294e945c5b343ae62e76dc2f0 P 213596374e4142dca6ecfc701ba1c566 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6932f1cd716a48fe8d6c5573acfc48bf for term 2 because replica has last-logged OpId of term: 1 index: 504, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 503. I0703 01:44:20.290335 5031 leader_election.cc:393] T 60d3228294e945c5b343ae62e76dc2f0 P 6932f1cd716a48fe8d6c5573acfc48bf [CANDIDATE]: Term 2 pre-election: Vote denied by peer 213596374e4142dca6ecfc701ba1c566 (127.5.240.196:51288). Message: Invalid argument: T 60d3228294e945c5b343ae62e76dc2f0 P 213596374e4142dca6ecfc701ba1c566 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6932f1cd716a48fe8d6c5573acfc48bf for term 2 because replica has last-logged OpId of term: 1 index: 504, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 503. I0703 01:44:20.290489 5031 leader_election.cc:270] T 60d3228294e945c5b343ae62e76dc2f0 P 6932f1cd716a48fe8d6c5573acfc48bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0703 01:44:20.290541 3573 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "60d3228294e945c5b343ae62e76dc2f0" candidate_uuid: "213596374e4142dca6ecfc701ba1c566" candidate_term: 2 candidate_status { last_received { term: 1 index: 504 } } ignore_live_leader: false dest_uuid: "55223945fe3d4dd8ac03ba9dab346f54" is_pre_election: true I0703 01:44:20.290678 3573 raft_consensus.cc:2730] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54: Snoozing failure detection for 1.649s (vote granted) I0703 01:44:20.290750 3573 raft_consensus.cc:2243] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 213596374e4142dca6ecfc701ba1c566 in term 1. I0703 01:44:20.290802 6105 raft_consensus.cc:2730] T 60d3228294e945c5b343ae62e76dc2f0 P 6932f1cd716a48fe8d6c5573acfc48bf: Snoozing failure detection for 1.678s (election complete) I0703 01:44:20.290900 6105 raft_consensus.cc:2485] T 60d3228294e945c5b343ae62e76dc2f0 P 6932f1cd716a48fe8d6c5573acfc48bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0703 01:44:20.291291 4765 leader_election.cc:379] T 60d3228294e945c5b343ae62e76dc2f0 P 213596374e4142dca6ecfc701ba1c566 [CANDIDATE]: Term 2 pre-election: Vote granted by peer 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193:54919) I0703 01:44:20.291353 4765 leader_election.cc:270] T 60d3228294e945c5b343ae62e76dc2f0 P 213596374e4142dca6ecfc701ba1c566 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. I0703 01:44:20.294356 6178 leader_election.cc:243] T 60d3228294e945c5b343ae62e76dc2f0 P 213596374e4142dca6ecfc701ba1c566 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198:51033) I0703 01:44:20.294607 6178 raft_consensus.cc:2730] T 60d3228294e945c5b343ae62e76dc2f0 P 213596374e4142dca6ecfc701ba1c566: Snoozing failure detection for 1.989s (election complete) I0703 01:44:20.294741 6178 raft_consensus.cc:2537] T 60d3228294e945c5b343ae62e76dc2f0 P 213596374e4142dca6ecfc701ba1c566 [term 1 FOLLOWER]: Leader pre-election won for term 2 I0703 01:44:20.294855 6178 raft_consensus.cc:457] T 60d3228294e945c5b343ae62e76dc2f0 P 213596374e4142dca6ecfc701ba1c566 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 55223945fe3d4dd8ac03ba9dab346f54) I0703 01:44:20.295027 6178 raft_consensus.cc:2730] T 60d3228294e945c5b343ae62e76dc2f0 P 213596374e4142dca6ecfc701ba1c566: Snoozing failure detection for 1.638s (starting election) I0703 01:44:20.295284 6178 raft_consensus.cc:2782] T 60d3228294e945c5b343ae62e76dc2f0 P 213596374e4142dca6ecfc701ba1c566 [term 1 FOLLOWER]: Advancing to term 2 I0703 01:44:20.297998 6178 raft_consensus.cc:479] T 60d3228294e945c5b343ae62e76dc2f0 P 213596374e4142dca6ecfc701ba1c566 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 500 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false } } I0703 01:44:20.302572 6178 leader_election.cc:243] T 60d3228294e945c5b343ae62e76dc2f0 P 213596374e4142dca6ecfc701ba1c566 [CANDIDATE]: Term 2 election: Requesting vote from peer 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193:54919) I0703 01:44:20.302985 6178 leader_election.cc:243] T 60d3228294e945c5b343ae62e76dc2f0 P 213596374e4142dca6ecfc701ba1c566 [CANDIDATE]: Term 2 election: Requesting vote from peer 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198:51033) I0703 01:44:20.311787 3573 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "60d3228294e945c5b343ae62e76dc2f0" candidate_uuid: "213596374e4142dca6ecfc701ba1c566" candidate_term: 2 candidate_status { last_received { term: 1 index: 504 } } ignore_live_leader: false dest_uuid: "55223945fe3d4dd8ac03ba9dab346f54" I0703 01:44:20.311933 3573 raft_consensus.cc:2782] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 [term 1 FOLLOWER]: Advancing to term 2 I0703 01:44:20.312024 3573 raft_consensus.cc:2730] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54: Snoozing failure detection for 1.903s (vote granted) I0703 01:44:20.315016 3573 raft_consensus.cc:2243] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 213596374e4142dca6ecfc701ba1c566 in term 2. I0703 01:44:20.316159 4765 leader_election.cc:379] T 60d3228294e945c5b343ae62e76dc2f0 P 213596374e4142dca6ecfc701ba1c566 [CANDIDATE]: Term 2 election: Vote granted by peer 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193:54919) I0703 01:44:20.316220 4765 leader_election.cc:270] T 60d3228294e945c5b343ae62e76dc2f0 P 213596374e4142dca6ecfc701ba1c566 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. I0703 01:44:20.316411 6182 raft_consensus.cc:2730] T 60d3228294e945c5b343ae62e76dc2f0 P 213596374e4142dca6ecfc701ba1c566: Snoozing failure detection for 1.736s (election complete) I0703 01:44:20.316504 6182 raft_consensus.cc:2537] T 60d3228294e945c5b343ae62e76dc2f0 P 213596374e4142dca6ecfc701ba1c566 [term 2 FOLLOWER]: Leader election won for term 2 I0703 01:44:20.316650 6182 raft_consensus.cc:593] T 60d3228294e945c5b343ae62e76dc2f0 P 213596374e4142dca6ecfc701ba1c566 [term 2 LEADER]: Becoming Leader. State: Replica: 213596374e4142dca6ecfc701ba1c566, State: Running, Role: LEADER I0703 01:44:20.317018 6182 consensus_queue.cc:204] T 60d3228294e945c5b343ae62e76dc2f0 P 213596374e4142dca6ecfc701ba1c566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 503, Committed index: 503, Last appended: 1.504, Last appended by leader: 504, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 500 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false } } I0703 01:44:20.324051 3448 catalog_manager.cc:3798] T 60d3228294e945c5b343ae62e76dc2f0 P 213596374e4142dca6ecfc701ba1c566 reported cstate change: term changed from 1 to 2, leader changed from 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193) to 213596374e4142dca6ecfc701ba1c566 (127.5.240.196). New cstate: current_term: 2 leader_uuid: "213596374e4142dca6ecfc701ba1c566" committed_config { opid_index: 500 OBSOLETE_local: false peers { permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false } health_report { overall_health: UNKNOWN } } } I0703 01:44:20.336731 5095 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "60d3228294e945c5b343ae62e76dc2f0" candidate_uuid: "213596374e4142dca6ecfc701ba1c566" candidate_term: 2 candidate_status { last_received { term: 1 index: 504 } } ignore_live_leader: false dest_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" is_pre_election: true I0703 01:44:20.336894 5095 raft_consensus.cc:2730] T 60d3228294e945c5b343ae62e76dc2f0 P 6932f1cd716a48fe8d6c5573acfc48bf: Snoozing failure detection for 2.684s (vote granted) I0703 01:44:20.336964 5095 raft_consensus.cc:2243] T 60d3228294e945c5b343ae62e76dc2f0 P 6932f1cd716a48fe8d6c5573acfc48bf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 213596374e4142dca6ecfc701ba1c566 in term 1. I0703 01:44:20.337381 5098 tablet_service.cc:995] Received RequestConsensusVote() RPC: tablet_id: "60d3228294e945c5b343ae62e76dc2f0" candidate_uuid: "213596374e4142dca6ecfc701ba1c566" candidate_term: 2 candidate_status { last_received { term: 1 index: 504 } } ignore_live_leader: false dest_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" I0703 01:44:20.337544 5098 raft_consensus.cc:2782] T 60d3228294e945c5b343ae62e76dc2f0 P 6932f1cd716a48fe8d6c5573acfc48bf [term 1 FOLLOWER]: Advancing to term 2 I0703 01:44:20.337602 4765 leader_election.cc:379] T 60d3228294e945c5b343ae62e76dc2f0 P 213596374e4142dca6ecfc701ba1c566 [CANDIDATE]: Term 2 pre-election: Vote granted by peer 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198:51033) I0703 01:44:20.337610 5098 raft_consensus.cc:2730] T 60d3228294e945c5b343ae62e76dc2f0 P 6932f1cd716a48fe8d6c5573acfc48bf: Snoozing failure detection for 3.189s (vote granted) I0703 01:44:20.340466 5098 raft_consensus.cc:2243] T 60d3228294e945c5b343ae62e76dc2f0 P 6932f1cd716a48fe8d6c5573acfc48bf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 213596374e4142dca6ecfc701ba1c566 in term 2. I0703 01:44:20.341146 4765 leader_election.cc:379] T 60d3228294e945c5b343ae62e76dc2f0 P 213596374e4142dca6ecfc701ba1c566 [CANDIDATE]: Term 2 election: Vote granted by peer 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198:51033) I0703 01:44:20.769626 3573 raft_consensus.cc:1081] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 [term 2 FOLLOWER]: Refusing update from remote peer 213596374e4142dca6ecfc701ba1c566: Log matching property violated. Preceding OpId in replica: term: 1 index: 504. Preceding OpId from leader: term: 2 index: 505. (index mismatch) I0703 01:44:20.770416 6182 consensus_queue.cc:962] T 60d3228294e945c5b343ae62e76dc2f0 P 213596374e4142dca6ecfc701ba1c566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55223945fe3d4dd8ac03ba9dab346f54" member_type: VOTER last_known_addr { host: "127.5.240.193" port: 54919 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 505, Last known committed idx: 503, Time since last communication: 0.000s I0703 01:44:20.790797 5097 raft_consensus.cc:1081] T 60d3228294e945c5b343ae62e76dc2f0 P 6932f1cd716a48fe8d6c5573acfc48bf [term 2 FOLLOWER]: Refusing update from remote peer 213596374e4142dca6ecfc701ba1c566: Log matching property violated. Preceding OpId in replica: term: 1 index: 503. Preceding OpId from leader: term: 2 index: 505. (index mismatch) I0703 01:44:20.793946 6178 consensus_queue.cc:962] T 60d3228294e945c5b343ae62e76dc2f0 P 213596374e4142dca6ecfc701ba1c566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 505, Last known committed idx: 503, Time since last communication: 0.000s I0703 01:44:20.796799 6199 rpcz_store.cc:265] Call kudu.tserver.TabletServerService.Write from 127.5.240.193:52720 (ReqId={client: 6134728413ca43e89ac48ee688c1fec2, seq_no=1459, attempt_no=0}) took 2036ms. Request Metrics: {"child_traces":[["txn",{"apply.queue_time_us":4195,"num_ops":1,"prepare.queue_time_us":49,"prepare.run_cpu_time_us":479,"prepare.run_wall_time_us":480,"replication_time_us":2030757}]]} I0703 01:44:20.925437 3448 catalog_manager.cc:3549] Sending ChangeConfig:REMOVE_PEER on tablet 60d3228294e945c5b343ae62e76dc2f0 (attempt 1) I0703 01:44:20.944983 4835 consensus_peers.cc:455] T 60d3228294e945c5b343ae62e76dc2f0 P 213596374e4142dca6ecfc701ba1c566 -> Peer 6932f1cd716a48fe8d6c5573acfc48bf (127.5.240.198:51033): Closing peer: 6932f1cd716a48fe8d6c5573acfc48bf I0703 01:44:20.945067 4835 consensus_peers.cc:455] T 60d3228294e945c5b343ae62e76dc2f0 P 213596374e4142dca6ecfc701ba1c566 -> Peer 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193:54919): Closing peer: 55223945fe3d4dd8ac03ba9dab346f54 I0703 01:44:20.945361 4835 consensus_queue.cc:204] T 60d3228294e945c5b343ae62e76dc2f0 P 213596374e4142dca6ecfc701ba1c566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 506, Committed index: 506, Last appended: 2.506, Last appended by leader: 504, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 507 OBSOLETE_local: false peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false } } I0703 01:44:20.947280 5095 raft_consensus.cc:1081] T 60d3228294e945c5b343ae62e76dc2f0 P 6932f1cd716a48fe8d6c5573acfc48bf [term 2 FOLLOWER]: Refusing update from remote peer 213596374e4142dca6ecfc701ba1c566: Log matching property violated. Preceding OpId in replica: term: 2 index: 506. Preceding OpId from leader: term: 2 index: 507. (index mismatch) I0703 01:44:20.947958 6216 consensus_queue.cc:962] T 60d3228294e945c5b343ae62e76dc2f0 P 213596374e4142dca6ecfc701ba1c566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 507, Last known committed idx: 506, Time since last communication: 0.000s I0703 01:44:20.955643 6216 raft_consensus.cc:2684] T 60d3228294e945c5b343ae62e76dc2f0 P 213596374e4142dca6ecfc701ba1c566 [term 2 LEADER]: Committing config change with OpId 2.507: config changed from index 500 to 507, VOTER 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193) evicted. New config: { opid_index: 507 OBSOLETE_local: false peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false } } } I0703 01:44:20.958657 5095 raft_consensus.cc:2684] T 60d3228294e945c5b343ae62e76dc2f0 P 6932f1cd716a48fe8d6c5573acfc48bf [term 2 FOLLOWER]: Committing config change with OpId 2.507: config changed from index 500 to 507, VOTER 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193) evicted. New config: { opid_index: 507 OBSOLETE_local: false peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false } } } I0703 01:44:20.959208 3437 catalog_manager.cc:3352] ChangeConfig:REMOVE_PEER RPC for tablet 60d3228294e945c5b343ae62e76dc2f0 with cas_config_opid_index 500: ChangeConfig:REMOVE_PEER succeeded (attempt 1) I0703 01:44:20.963146 3448 catalog_manager.cc:3798] T 60d3228294e945c5b343ae62e76dc2f0 P 6932f1cd716a48fe8d6c5573acfc48bf reported cstate change: config changed from index 500 to 507, VOTER 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193) evicted. New cstate: current_term: 2 leader_uuid: "213596374e4142dca6ecfc701ba1c566" committed_config { opid_index: 507 OBSOLETE_local: false peers { permanent_uuid: "213596374e4142dca6ecfc701ba1c566" member_type: VOTER last_known_addr { host: "127.5.240.196" port: 51288 } attrs { promote: false } } peers { permanent_uuid: "6932f1cd716a48fe8d6c5573acfc48bf" member_type: VOTER last_known_addr { host: "127.5.240.198" port: 51033 } attrs { promote: false } } } I0703 01:44:20.994468 3448 catalog_manager.cc:3125] Sending DeleteTablet(TABLET_DATA_TOMBSTONED) for tablet 60d3228294e945c5b343ae62e76dc2f0 on 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193:54919) (TS 55223945fe3d4dd8ac03ba9dab346f54 not found in new config with opid_index 507) I0703 01:44:20.995663 3553 tablet_service.cc:800] Processing DeleteTablet for tablet 60d3228294e945c5b343ae62e76dc2f0 with delete_type TABLET_DATA_TOMBSTONED (TS 55223945fe3d4dd8ac03ba9dab346f54 not found in new config with opid_index 507) from {username='jenkins'} at 127.5.240.193:52692 I0703 01:44:20.998097 6223 tablet_replica.cc:261] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54: stopping tablet replica I0703 01:44:20.998203 6223 maintenance_manager.cc:237] P 55223945fe3d4dd8ac03ba9dab346f54: Unregistered op CompactRowSetsOp(60d3228294e945c5b343ae62e76dc2f0) I0703 01:44:20.998291 6223 maintenance_manager.cc:237] P 55223945fe3d4dd8ac03ba9dab346f54: Unregistered op MinorDeltaCompactionOp(60d3228294e945c5b343ae62e76dc2f0) I0703 01:44:20.998374 6223 maintenance_manager.cc:237] P 55223945fe3d4dd8ac03ba9dab346f54: Unregistered op MajorDeltaCompactionOp(60d3228294e945c5b343ae62e76dc2f0) I0703 01:44:21.001292 6223 maintenance_manager.cc:237] P 55223945fe3d4dd8ac03ba9dab346f54: Unregistered op UndoDeltaBlockGCOp(60d3228294e945c5b343ae62e76dc2f0) I0703 01:44:21.001390 6223 maintenance_manager.cc:237] P 55223945fe3d4dd8ac03ba9dab346f54: Unregistered op FlushMRSOp(60d3228294e945c5b343ae62e76dc2f0) I0703 01:44:21.001461 6223 maintenance_manager.cc:237] P 55223945fe3d4dd8ac03ba9dab346f54: Unregistered op FlushDeltaMemStoresOp(60d3228294e945c5b343ae62e76dc2f0) I0703 01:44:21.001525 6223 maintenance_manager.cc:237] P 55223945fe3d4dd8ac03ba9dab346f54: Unregistered op LogGCOp(60d3228294e945c5b343ae62e76dc2f0) I0703 01:44:21.001617 6223 raft_consensus.cc:2040] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 [term 2 FOLLOWER]: Raft consensus shutting down. I0703 01:44:21.001731 6223 pending_rounds.cc:63] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54: Trying to abort 1 pending transactions. I0703 01:44:21.001785 6223 pending_rounds.cc:68] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54: Aborting transaction as it isn't in flight: id { term: 2 index: 506 } timestamp: 6269368159832784896 op_type: WRITE_OP write_request { tablet_id: "60d3228294e945c5b343ae62e76dc2f0" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true } } row_operations { rows: "\001\007\000\177\255Pu&\211\332\277\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "16fbf457eecd50bc" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 6269368159436787712 } request_id { client_id: "6134728413ca43e89ac48ee688c1fec2" seq_no: 1461 first_incomplete_seq_no: 1461 attempt_no: 2 } I0703 01:44:21.002811 6223 raft_consensus.cc:2067] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54 [term 2 FOLLOWER]: Raft consensus is shut down! I0703 01:44:21.003759 6223 ts_tablet_manager.cc:1369] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED I0703 01:44:21.013224 6223 ts_tablet_manager.cc:1382] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.506 I0703 01:44:21.013322 6223 log.cc:986] T 60d3228294e945c5b343ae62e76dc2f0 P 55223945fe3d4dd8ac03ba9dab346f54: Deleting WAL directory at /data/somelongdirectorytoavoidrpathissues/src/kudutest/kudu-admin-test.0.RebalanceParamTest.Rebalance_3.1530607157582131-6083/raft_consensus-itest-cluster/ts-0/wal/wals/60d3228294e945c5b343ae62e76dc2f0 I0703 01:44:21.014294 3437 catalog_manager.cc:3103] TS 55223945fe3d4dd8ac03ba9dab346f54 (127.5.240.193:54919): tablet 60d3228294e945c5b343ae62e76dc2f0 (table rebalance_test_table_3 [id=2d96fe2971a440c3b073db7c0073362b]) successfully deleted W0703 01:44:23.332222 4764 connection.cc:512] client connection to 127.5.240.254:32866 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.254:32866): Connection reset by peer (error 104) W0703 01:44:23.338141 5030 connection.cc:512] client connection to 127.5.240.254:32866 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.254:32866): Connection reset by peer (error 104) W0703 01:44:23.338452 4896 connection.cc:512] client connection to 127.5.240.254:32866 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.254:32866): Connection reset by peer (error 104) W0703 01:44:23.338899 5163 connection.cc:512] client connection to 127.5.240.254:32866 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.254:32866): Connection reset by peer (error 104) W0703 01:44:23.377414 5164 connection.cc:512] client connection to 127.5.240.196:51288 recv error: Network error: failed to read from TLS socket (remote: 127.5.240.196:51288): Connection reset by peer (error 104)